Aims

• Engage deeply with fresh cutting edge work (as well as some older canonical work)
• Develop skills in empirical methods
• Exposure to open science practices

Key idea: you (often) can’t fully evaluate a piece of work until you have looked at the data and done the analysis yourself.

Syllabus

is here: https://macartan.github.io/teaching/ds-hu-2023

see also the replication repo: https://macartan.github.io/ds_hu_2003_reps/

The readings: 1 Macro processes

The readings: 2 Group politics

The readings: 3 Institutions and accountability

The readings: 4 Aid and interventions

Expectations

• 5 tasks
• Work in four “rep teams”: 1 team per session \(\times 4\)
• Prepare a research design or short paper, perhaps building on existing work. Typically this contains: (i) a theoretical argument or motivation, (ii) a proposed empirical test of that argument (iii) a formal design object and (iv) a discussion of policy prescriptions that might result from the argument. We will discuss these in the final days of class. Focus on something that might turn into a research project.
• Plus general reading and participation. Only four papers per session: read them all carefully!

Rep team job

• Engage with both methods and substance of paper
• Access data
• Run basic replication of main results
• Draft a pre-re-analysis plan
• Implement new analyses
• Share clean re-analysis files with class via Moodle
• Prepare a “replicable presentation” and present to the class (ca 45 mins)

How to: Replication files

• Best in self-contained documents for easy third party viewing. e.g. .html via .qmd or .Rmd

Some examples:

• https://macartan.github.io/projects/replications/
• https://wzb-ipi.github.io/vaccine_freedoms/

Good coding rules

• https://bookdown.org/content/d1e53ac9-28ce-472f-bc2c-f499f18264a3/code.html
• https://www.r-bloggers.com/2018/09/r-code-best-practices/

Good coding rules

• Metadata first
• Call packages at the beginning: use pacman
• Put options at the top
• Call all data files once, at the top. Best to call directly from a public archive, when possible.
  
• Use functions and define them at the top: comment them; useful sometimes to illustrate what they do
• Replicate first, re-analyze second. Use sections.
• Have subsections named after specific tables, figures or analyses

Aim

Nothing local, everything relative: so please do not include hardcoded paths to your computer

• First best: if someone has access to your .Rmd/.qmd file they can hit render or compile and the whole thing reproduces first time.

• But: often you need ancillary files for data and code. That’s OK but aims should still be that with a self contained folder someone can open a master.Rmd file, hit compile and get everything. I usually have an input and an output subfolder.

Resources and ideas from the institute for replication https://i4replication.org/reproducibility.html

Longer term goal: replication package to make it easier to access and share replications like these

Collaborative coding / writing

• Do not get in the business of passing attachments around
• Share self contained folders; folders contain a small set of live documents plus an archive. Old versions of documents are in archive. Only one version of the most recent document is in a main folder.
• Data is self contained folder (in) and is never edited directly

How to: Contents

Sample TOC for presentations:

• Motivation
• Theory: e.g. a DAG plus motivation for it
• Design (e.g. ex ante perspective): A design diagnosis
• Analysis:
  • descriptives: show data!! Make sure you and we all understand the type of data we are facing
  • replication
  • re-analysis: for robustness, for strengthening
• Interpretation (explanation, implications, generalizability)

Don’t skip the big picture.

Contents

See: How to critique: https://macartan.github.io/teaching/how-to-critique

Biggest message: be probing but be sympathetic

How to: Theory

• Strong recommend you try to initially present all arguments as a DAG

make_model(
   "Inequality -> Mobilization -> Democratization <- Threat <- Inequality;
   Pressure -> Democratization")  |> 
  plot()

How to: Theory

…and then use the DAG to describe new analysis, e.g. questions regarding:

• identification
• mechanisms
• plausible or implausible assumptions
• possible confounds

Straight replication

• most cases code will be provided
• encourage you not to rely on this; try from scratch

If you fail to replicate:

• this might be your fault!
• check your code against their code; compare Ns; break it down to estimate single quantities
• if it’s a small discrepancy don’t worry
• if it’s large, check with me; if it’s important check with authors (politely) to help make sense of things; always be generous

Re-analysis

Two distinct overarching goals:

1. Check robustness
2. Deepen the analysis

Robustness: Things you might do:

• Plot data to understand where effects are coming from and checking no weirdness
• Swap out datasets (sometimes possible with observational data)
• Check robustness to specification / assumptions
• Check missingness patterns
• Assess plausible external validity
• Check for multiple comparisons
• Check against pre-analysis plans

Deepening: Things you might do

• Principled exploration of heterogeneity (e.g. using causal forests)
• Principled incorporation of controls (e.g. Lasso approaches)
• Search for additional implications of theory: if the theory is right what should we see? If the theory is right where should effects be strongest or weakest?

Reanalysis: Avoiding fishing

• There’s a risk of engaging in a type of fishing when you do replication or re-analysis.
• Say you decide to control for something and a result disappears: is this a threat to the analysis?
• Not necessarily
• In our Research Design book we recommend using design declaration to justify re-analysis decisions

Reanalysis: Avoiding fishing

e.g. 

Home ground dominance. Holding the original M constant (i.e., the home ground of the original study), if you can show that a new answer strategy A’ yields better diagnosands than the original A, then A’ can be justified by home ground dominance.

Robustness to alternative models. A second justification for a change in answer strategy is that you can show that a new answer strategy is robust to both the original model M and a new, also plausible, M’.

The simplest growth model: Solow

Can someone walk us through the Solow model?

The simplest growth model: The economy

Production function.

\[Y = F(K,L)\] Per capita:

\[y = f(k)\] e.g.

\[y = Ak^\alpha\]

The simplest growth model: What changes

Savings are constant:

\[\text{savings} = s y_t\]

Depreciation constant share of capital: \[\text{depreciation}=\delta k\]

Law of motion of capital:

\[k_{t+1} = sk_t^\alpha - \delta k\]

Steady state has:

\[k_t = k_{t+1} = k_t^*\leftrightarrow sAk_t^{*\alpha} = \delta k^{*} \leftrightarrow k^* = \left(\frac{sA}{\delta}\right)^{\frac1{1-\alpha}}\]

The simplest growth model: What drivers?

• So what causes long term growth?
• What causes variation in income?
• Where is the politics?
• What growth models do you have in your head?

An argument:

Here is a complete, albeit barebones (and possibly incorrect), argument:

• Good institutions (I) cause economic growth (G), except in countries with large stocks of natural resources (N)
• The reason is that institutions encourage people to invest (V) which spurs growth (this effect does not kick in in natural resource rich countries as people just live off rent)
  
• Growth also makes it easier to maintain good institutions, which creates a virtuous cycle
• Being an ally (A) of the US helps growth, but it can corrupt domestic institutions.
• Historically, places with climates (C) suitable for colonizers to settle had better institutions. These climatic conditions are otherwise irrelevant for contemporary economic growth.

Some counterarguments:

• Places with climates suitable for colonizers benefited from better access to international markets which led to growth.
• Good soil is also important for growth!
• Good institutions also make sure that investments yield greater returns and that’s what causes growth

Questions on Nodes

• What are the dependent variables?
• What are the independent variables?
• What are the mediating variables?
• What are the conditioning variables?
• What are the confounding variables?
• What are the instrumental variables?
• Graph the relations between the variables.

Questions on Inference

• Say I and G are positively correlated. Does this mean that I causes G?

• Say I and G are negatively correlated. Does this mean that I does not cause G?

• How might you estimate the effect of I on G?

• How does C help establish the link between I and G?

• Where is the theory? Is in equivalent to the graph or is it something else that generates the graph?

• How might you check if the proposed theory is correct?

• Which of the counterarguments are strong and why?

A graph

Exercises: Dissect these arguments

Four arguments. For each one you should identify the:

• type of argument (effect of \(X\), cause of \(Y\), effect of \(X\) on \(Y\))
• unit of analysis
• dependent variable(s)
• independent variable(s)
• mediator(s)
• possible conditioning variable(s)
• possible confounder(s)
• possible identification strategy
• relevant key agent(s) (actor(s))
• measurement strategy

A. Natural resources and conflict

In developing countries that discover natural resources, such as oil, the ruling elite can extract wealth without needing to tax citizens and develop the state apparatus. Because the state does not rely on taxation for government revenue, it does not need to set up accountability structures or extend its reach and citizens do not feel that they have ownership over the state. The state therefore becomes both less democratic and weaker than if it had not discovered the resources.

B. Democracy and growth

Rich countries are more likely to be democratic for the simple reason that when people become wealthier they refuse to be dictated to by others and they demand a role in government. The marginal effects of income increases are greater for poorer countries because the impacts on eduction are greatest at these levels. You can test this proposition by exploiting natural variation in commodity prices which provide shocks to national income, especially for countries dependent on primary commodity exports.

C. Factor Endowments and Coalitions

When countries increase trade (imports and exports), the returns to economic factors (such as labor, land and capital) are affected differently. Specifically, the returns to factors that are the most abundant are positive, while the returns to factors that are the most scarce are negative. Therefore, the relative factor endowments of a country will predict what sort of political coalitions will form (eg Land versus Labor + Capital) and which groups will favor free trade policies.

D. Democratic peace

In democratic states, leaders are accountable for any losses incurred as a result of the wars that they enter into. Two states with democratic leaders are also more likely to share a common set of norms, and to engage in trade with one another. Therefore, two democracies are far less likely to enter into war with one another than a democracy and a non-democracy, or two non-democracies.

The MIDA Framework

Q: Is my research design good?

A: Well let’s simulate it to see how it performs.

Q: What should I put in the simulation?

A: All elements of a research design.

Q: What are the elements of a research design?

A: M! I! D! A!

Four elements of any research design

• Model: set of models of what causes what and how
• Inquiry: a question stated in terms of the model
• Data strategy: the set of procedures we use to gather information from the world (sampling, assignment, measurement)
• Answer strategy: how we summarize the data produced by the data strategy

Four elements of any research design

In code: Key commands for making a design

• declare_model()
• declare_inquiry()
• declare_assignment()
• declare_measurement()
• declare_inquiry
• declare_estimator()

and there are more declare_ functions!

In code: Key commands for using a design

• draw_data(design)
• draw_estimands(design)
• draw_estimates(design)
• get_estimates(design, data)
• run_design(design), simulate_design(design)
• diagnose_design(design)
• redesign(design, N = 200)
• design |> redesign(N = c(200, 400)) |> diagnose_designs()
• compare_designs(), compare_diagnoses()

Cheat sheet

https://raw.githubusercontent.com/rstudio/cheatsheets/master/declaredesign.pdf

A simple design

N <- 100
b <- .5

design <- 
  declare_model(N = N, U = rnorm(N), 
                potential_outcomes(Y ~ b * Z + U)) + 
  declare_assignment(Z = simple_ra(N), Y = reveal_outcomes(Y ~ Z)) + 
  declare_inquiry(ate = mean(Y_Z_1 - Y_Z_0)) + 
  declare_estimator(Y ~ Z, inquiry = "ate", .method = lm_robust)

You now have a two arm design object in memory!

If you just type design it will run the design—a good check to make sure the design has been declared properly.

Make data from the design

data <- draw_data(design)

data |> head () |> kable()

Draw estimands

draw_estimands(design) |>
  kable(digits = 2)

Draw estimates

draw_estimates(design) |> 
  kable(digits = 2) 

Get estimates

get_estimates(design, data) |>
  kable(digits = 2)

Simulate design

simulate_design(design, sims = 3) |>
  kable(digits = 2)

Diagnose design

design |> 
  diagnose_design(sims = 100) 

Redesign

new_design <-
  
  design |> redesign(b = 0)

• Modify any arguments that are explicitly called on by design steps.
• Or add, remove, or replace steps

Compare designs

redesign(design, N = 50) %>%
  
  compare_diagnoses(design) 

Error in eval(expr, envir, enclos): object 'run' not found