2 Lecture outline

• General aims and structure
• Expectations
• Pointers for coding
• Topics / refreshers
  • Papers: structures and processes
  • DAGs, variables, identification strategies
  • Designs and design diagnosis

2.1 Aims

• Write a paper
• Follow open science principles

3 Topics 1: Papers

3.1 Classic paper structure

https://macartan.github.io/teaching/how-to-write

Classic structure

1. Motivation
2. Theory
3. Strategy (perhaps descriptives here)
4. Main results
5. Disucission / deepening
6. Conclusion

4 Models: Arguments as DAGs

4.1 An argument:

Here is a complete, albeit barebones (and possibly incorrect), argument:

• Good institutions (I) cause economic growth (G), except in countries with large stocks of natural resources (N)
• The reason is that institutions encourage people to invest (V) which spurs growth (this effect does not kick in in natural resource rich countries as people just live off rent)
  
• Growth also makes it easier to maintain good institutions, which creates a virtuous cycle
• Being an ally (A) of the US helps growth, but it can corrupt domestic institutions.
• Historically, places with climates (C) suitable for colonizers to settle had better institutions. These climatic conditions are otherwise irrelevant for contemporary economic growth.

4.2 Some counterarguments:

• Places with climates suitable for colonizers benefited from better access to international markets which led to growth.
• Good soil is also important for growth!
• Good institutions also make sure that investments yield greater returns and that’s what causes growth

4.3 Questions on Nodes

• What are the dependent variables?
• What are the independent variables?
• What are the mediating variables?
• What are the conditioning variables?
• What are the confounding variables?
• What are the instrumental variables?
• Graph the relations between the variables.

4.4 Questions on Inference

• Say I and G are positively correlated. Does this mean that I causes G?

• Say I and G are negatively correlated. Does this mean that I does not cause G?

• How might you estimate the effect of I on G?

• How does C help establish the link between I and G?

• Where is the theory? Is in equivalent to the graph or is it something else that generates the graph?

• How might you check if the proposed theory is correct?

• Which of the counterarguments are strong and why?

4.5 A graph

4.6 Exercises: Dissect these arguments

Four arguments. For each one you should identify the:

• type of argument (effect of \(X\), cause of \(Y\), effect of \(X\) on \(Y\))
• unit of analysis
• dependent variable(s)
• independent variable(s)
• mediator(s)
• possible conditioning variable(s)
• possible confounder(s)
• possible identification strategy
• relevant key agent(s) (actor(s))
• measurement strategy

4.7 A. Natural resources and conflict

In developing countries that discover natural resources, such as oil, the ruling elite can extract wealth without needing to tax citizens and develop the state apparatus. Because the state does not rely on taxation for government revenue, it does not need to set up accountability structures or extend its reach and citizens do not feel that they have ownership over the state. The state therefore becomes both less democratic and weaker than if it had not discovered the resources.

4.8 B. Democracy and growth

Rich countries are more likely to be democratic for the simple reason that when people become wealthier they refuse to be dictated to by others and they demand a role in government. The marginal effects of income increases are greater for poorer countries because the impacts on eduction are greatest at these levels. You can test this proposition by exploiting natural variation in commodity prices which provide shocks to national income, especially for countries dependent on primary commodity exports.

4.9 C. Factor Endowments and Coalitions

When countries increase trade (imports and exports), the returns to economic factors (such as labor, land and capital) are affected differently. Specifically, the returns to factors that are the most abundant are positive, while the returns to factors that are the most scarce are negative. Therefore, the relative factor endowments of a country will predict what sort of political coalitions will form (eg Land versus Labor + Capital) and which groups will favor free trade policies.

4.10 D. Democratic peace

In democratic states, leaders are accountable for any losses incurred as a result of the wars that they enter into. Two states with democratic leaders are also more likely to share a common set of norms, and to engage in trade with one another. Therefore, two democracies are far less likely to enter into war with one another than a democracy and a non-democracy, or two non-democracies.

5 Design declaration and diagnosis

book: https://book.declaredesign.org/

5.1 The MIDA Framework

Four elements of any research design:

• Model: set of models of what causes what and how
• Inquiry: a question stated in terms of the model
• Data strategy: the set of procedures we use to gather information from the world (sampling, assignment, measurement)
• Answer strategy: how we summarize the data produced by the data strategy

5.2 Four elements of any research design

5.3 Examples of MIDA elements

• M: DAGs, game theoretic models
• I: ATEs, CATEs, COEs, models
• D: Sampling schemes, assignment schemes, text analysis, interview
• A: Experiment, observational, quantitative, qualitative:
  • Conditioning on observables
  • Difference in differences
  • RDD
  • Instrumental variables

5.4 Declaration, Diagnosis, Redesign cycle

• Declaration: Telling the computer what M, I, D, and A are.

• Diagnosis: Estimating “diagnosands” like power, bias, rmse, error rates, ethical harm, amount learned.

• Redesign : Fine-tuning features of the data and answer strategies to understand how they change the diagnosands

• Different sample sizes

• Different randomization procedures

• Different estimation strategies

• Implementation: effort into compliance versus more effort into sample size

5.7 Cheat sheet

https://raw.githubusercontent.com/rstudio/cheatsheets/master/declaredesign.pdf

5.8 A simple design

N <- 100
b <- .5

design <- 
  declare_model(N = N, U = rnorm(N), 
                potential_outcomes(Y ~ b * Z + U)) + 
  declare_assignment(Z = simple_ra(N), Y = reveal_outcomes(Y ~ Z)) + 
  declare_inquiry(ate = mean(Y_Z_1 - Y_Z_0)) + 
  declare_estimator(Y ~ Z, inquiry = "ate", .method = lm_robust)

You now have a two arm design object in memory!

If you just type design it will run the design—a good check to make sure the design has been declared properly.

5.9 Make data from the design

data <- draw_data(design)

data |> head () |> kable()

5.10 Draw estimands

draw_estimands(design) |>
  kable(digits = 2)

5.11 Draw estimates

draw_estimates(design) |> 
  kable(digits = 2) 

5.12 Get estimates

get_estimates(design, data) |>
  kable(digits = 2)

5.13 Simulate design

simulate_design(design, sims = 3) |>
  kable(digits = 2)

5.14 Diagnose design

design |> 
  diagnose_design(sims = 100) 

5.15 Redesign

new_design <-
  
  design |> redesign(b = 0)

• Modify any arguments that are explicitly called on by design steps.
• Or add, remove, or replace steps

6 Topics 3: Workflow

6.2 Folders and files.

• Have all files: writing, files, data files, additional analysis files or images, etc in a single directory with relative references

  • Number your folders.
  • Have few files in each folder.
  • I often label files with date: 20201005_paper.Rmd
  • Compile regularly and have a readable compiled file beside your work file
  • Keep your main document clean
  • Have an archive folder 0_archive and backup old copies regularly (not so important if you have good versioning)

6.3 Examples

See examples in sample_project

6.4 Tasks

• Keep a to do list
• If you use github it is great to use “issues” to keep track of to dos
• Try to complete well defined tasks in one sitting. Work in chunk.
• If you have a repetitive task there is probably a way to automate it: ask for help
• If you have a conceptually hard task that you are not making progress on, stop, move away, and try it from a whole new angle
• Order your tasks: figure out whether you work better linearly or doing parallel work.
• Cross off tasks when done
• Don’t be afraid to discard work
• Be ambitious but don’t let the best be the enemy of the good.

7 Pointers for coding

You should generate your replication files at the same time as your analysis

7.1 How to: Replication files

• Best in self-contained documents for easy third party viewing. e.g. .html via .qmd or .Rmd

Some examples:

• https://macartan.github.io/projects/replications/
• https://wzb-ipi.github.io/vaccine_freedoms/

7.2 Good coding rules

• https://bookdown.org/content/d1e53ac9-28ce-472f-bc2c-f499f18264a3/code.html
• https://www.r-bloggers.com/2018/09/r-code-best-practices/

7.3 Good coding rules

• Metadata first
• Call packages at the beginning: use pacman
• Put options at the top
• Call all data files once, at the top. Best to call directly from a public archive, when possible.
  
• Use functions and define them at the top: comment them; useful sometimes to illustrate what they do
• Replicate first, re-analyze second. Use sections.
• Have subsections named after specific tables, figures or analyses

7.4 Aim

Nothing local, everything relative: so please do not include hardcoded paths to your computer

• First best: if someone has access to your .Rmd/.qmd file they can hit render or compile and the whole thing reproduces first time.

• But: often you need ancillary files for data and code. That’s OK but aims should still be that with a self contained folder someone can open a master.Rmd file, hit compile and get everything. I usually have an input and an output subfolder.

Resources and ideas from the institute for replication https://i4replication.org/reproducibility.html