Causal Inference

2.2 The problem of unidentified mediators

Which effects are identified by the random assignment of \(X\)?

2.3 The problem of unidentified mediators

An obvious approach is to first examine the (average) effect of X on M1 and then use another manipulation to examine the (average) effect of M1 on Y.

• But both of these average effects may be positive (for example) even if there is no effect of X on Y through M1.

2.4 The problem of unidentified mediators

An obvious approach is to first examine the (average) effect of X on M1 and then use another manipulation to examine the (average) effect of M1 on Y.

• Similarly both of these average effects may be zero even if X affects on Y through M1 for every unit!

2.5 The problem of unidentified mediators

Both instances of unobserved confounding between \(M\) and \(Y\):

2.6 The problem of unidentified mediators

Both instances of unobserved confounding between \(M\) and \(Y\):

2.7 The problem of unidentified mediators

• Another somewhat obvious approach is to see how the effect of \(X\) on \(Y\) in a regression is reduced when you control for \(M\).

• If the effect of \(X\) on \(Y\) passes through \(M\) then surely there should be no effect of \(X\) on \(Y\) after you control for \(M\).

• This common strategy associated with Baron and Kenny (1986) is also not guaranteed to produce reliable results. See for instance Green, Ha, and Bullock (2010)

2.8 Baron Kenny issues

df <- fabricate(N = 1000, 
                U = rbinom(N, 1, .5),     X = rbinom(N, 1, .5),
                M = ifelse(U==1, X, 1-X), Y = ifelse(U==1, M, 1-M)) 
            
list(lm(Y ~ X, data = df), 
     lm(Y ~ X + M, data = df)) |> texreg::htmlreg() 

2.9 The problem of unidentified mediators

• See Imai on better ways to think about this problem and designs to address it.

2.10 The problem of unidentified mediators: Quantities

• Using potential outcomeswe can describe a mediation effect as (see Imai et al): \[\delta_i(t) = Y_i(t, M_i(1)) - Y_i(t, M_i(0)) \textbf{ for } t = 0,1\]
• The direct effect is: \[\psi_i(t) = Y_i(1, M_i(t)) - Y_i(0, M_i(t)) \textbf{ for } t = 0,1\]
• This is a decomposition, since: \[Y_i(1, M_i(1)) - Y_1(0, M_i(0)) = \frac{1}{2}(\delta_i(1) + \delta_i(0) + \psi_i(1) + \psi_i(0)) \]
• If there are no interaction effects—ie \(\delta_i(1) = \delta_i(0), \psi_i(1) = \psi_i(0)\), then \[Y_i(1, M_i(1)) - Y_1(0, M_i(0)) = \delta_i + \psi_i\]

2.11 The problem of unidentified mediators: Solutions?

The bad news is that although a single experiment might identify the total effect, it can not identify these elements of the direct effect.

So:

• Check formal requirement for identification under single experiment design (“sequential ignorability”—that, conditional on actual treatment, it is as if the value of the mediation variable is randomly assigned relative to potential outcomes). But this is strong (and in fact unverifiable) and if it does not hold, bounds on effects always include zero (Imai et al)

• Consider sensitivity analyses

2.12 Implicit mediation

You can use interactions with covariates if you are willing to make assumptions on no heterogeneity of direct treatment effects over covariates.

eg you think that money makes people get to work faster because they can buy better cars; you look at the marginal effect of more money on time to work for people with and without cars and find it higher for the latter.

This might imply mediation through transport but only if there is no direct effect heterogeneity (eg people with cars are less motivated by money).

2.16 In CausalQueries

model |> update_model(data = truth |> make_data(n = 1000)) |>
  query_distribution(queries = queries, using = "posteriors") 

Error in if (parent_nodes == "") {: argument is of length zero

Why such poor behavior? Why isn’t weight going onto indirect effects?

Turns out the data is consistent with direct effects only: specifically that whenever \(M\) is responsive to \(X\), \(Y\) is responsive to \(X\).

2.17 In CausalQueries

Error in if (parent_nodes == "") {: argument is of length zero

3.4 The list experiment: Strategy

• Respondents are given a short list and a long list.

• The long list differs from the short list in having one extra item—the sensitive item

• We ask how many items in each list does a respondent agree with:

  • \(Y_i(0)\) is the number of elements on a short list that a respondent agrees with
  • \(Y_i(1)\) is the number of elements on a long list that a respondent agrees with
  • \(Y_i(1) - Y_i(0)\) is an indicator for whether an individual agrees with the sensitive item
  • \(\mathbb{E}[Y_i(1) - Y_i(0)]\) is the share of people agreeing with sensitive item

3.5 The list experiment: Simplified example

How many of these do you agree with:

[Note: this is obviously not a good list. Why not?]

3.14 Individual or group effects?

• This is typically used to estimate average levels

• However you can use it in the obvious way to get average levels for groups: this is equivalent to calculating group level heterogeneous effects

• Extending the idea you can even get individual level estimates: for instance you might use causal forests

• You can also use this to estimate the effect of an experimental treatment on an item that’s measured using a list, without requiring individual level estimates:

\[Y_i = \beta_0 + \beta_1Z_i + \beta_2Long_i + \beta_3Z_iLong_i\]

4.5 Innovative designs

• Randomization of where police are stationed (India)
• Randomization of how government tax collectors get paid (do they get a share?) (Pakistan)
• Randomization of the voting rules for determining how decisions get made (Afghanistan)
• Random assignment of populations to peacekeepers (Liberia)
• Random assignment of ex-combatants out of their networks (Indonesia)
• Randomization of students to ethnically homogeneous or ethnically diverse schools (anywhere?)

5.1 Constraint: Is it ethical to manipulate subjects for research purposes?

• There is no foundationless answer to this question. So let’s take some foundations from the Belmont report and seek to ensure:

  1. Respect for persons
  2. Beneficence
  3. Justice

• Unfortunately, operationalizing these requires further ethical theories. Let’s assume that (1) is operationalized by informed consent (a very liberal idea). We are a bit at sea for (2) and (3) (the Belmont report suggests something like a utilitarian solution).

• The major focus on (1) by IRBs might follow from the view that if subjects consent, then they endorse the ethical calculations made for 2 and 3 — they think that it is good and fair.

• This is a little tricky, though, since the study may not be good or fair because of implications for non-subjects.

5.2 Is it ethical to manipulate subjects for research purposes?

• The problem is that many (many) field experiments have nothing like informed consent.

• For example, whether the government builds a school in your village, whether an ad appears on your favorite radio show, and so on.

• Consider three cases:

  1. You work with a nonprofit to post (true?) posters about the crimes of politicians on billboards to see effects on voters
  2. You hire confederates to offer bribes to police officers to see if they are more likely to bend the law for coethnics
  3. The British government asks you to work on figuring out how the use of water cannons helps stop rioters rioting

5.3 Is it ethical to manipulate subjects for research purposes?

• Consider three cases:

  • You work with a nonprofit to post (true?) posters about the crimes of politicians on billboards to see effects on voters
  • You hire confederates to offer bribes to police officers to see if they are more likely to bend the law for coethnics
  • The British government asks you to work on figuring out how the use of water cannons helps stop rioters rioting

• In all cases, there is no consent given by subjects.

• In 2 and 3, the treatment is possibly harmful for subjects, and the results might also be harmful. But even in case 1, there could be major unintended harmful consequences.

• In cases 1 and 3, however, the “intervention” is within the sphere of normal activities for the implementer.

5.4 Constraint: Is it ethical to manipulate subjects for research purposes?

• Sometimes it is possible to use this point of difference to make a “spheres of ethics” argument for “embedded experimentation.”

• Spheres of Ethics Argument: Experimental research that involves manipulations that are not normally appropriate for researchers may nevertheless be ethical if:

  • Researchers and implementers agree on a division of responsibility where implementers take on responsibility for actions
  • Implementers have legitimacy to make these decisions within the sphere of the intervention
  • Implementers are indeed materially independent of researchers (no swapping hats)

5.5 Constraint: Is it ethical to manipulate subjects for research purposes?

• Difficulty with this argument:
  • Question begging: How to determine the legitimacy of the implementer? (Can we rule out Nazi doctors?)

Otherwise keep focus on consent and desist if this is not possible

6.2 Contentious Issues

Experimental researchers are deeply engaged in the movement towards more transparency social science research.

Contentious issues (mostly):

• Data. How soon should you make your data available? My view: as soon as possibe. Along with working papers and before publication. Before it affects policy in any case. Own the ideas not the data.

  • Hard core: no citation without (analytic) replication. Perhaps. Non-replicable results should not be influencing policy.

• Where should you make your data available? Dataverse is focal for political science. Not personal website (mea culpa)

• What data should you make available? Disagreement is over how raw your data should be. My view: as raw as you can but at least post cleaning and pre-manipulation.

6.3 Open science checklist

Experimental researchers are deeply engaged in the movement towards more transparency social science research.

• Should you register?: Hard to find reasons against. But case strongest in testing phase rather than exploratory phase.

• Registration: When should you register? My view: Before treatment assignment. (Not just before analysis, mea culpa)

• Registration: Should you deviate from a preanalysis plan if you change your mind about optimal estimation strategies. My view: Yes, but make the case and describe both sets of results.

8.1 Myth: Concerns about fishing presuppose researcher dishonesty

• Fishing can happen in very subtle ways, and may seem natural and justifiable.

• Example:

  • I am interested in whether more democratic institutions result in better educational outcomes.
  • I examine the relationship between institutions and literacy and between institutions and school attendance.
  • The attendance measure is significant and the literacy one is not. Puzzled, I look more carefully at the literacy measure and see various outliers and indications of measurement error. As I think more I realize too that literacy is a slow moving variable and may not be the best measure anyhow. I move forward and start to analyze the attendance measure only, perhaps conducting new tests, albeit with the same data.

8.5 Myth: Registration only makes sense for experimental studies, not for observational studies

• The key distinction is between prospective and retrospective studies.

• Not between experimental and observational studies.

• A reason (from the medical literature) why registration is especially important for experiments: because you owe it to subjects

• A reason why registration is less important for experiments: because it is more likely that the intended analysis is implied by the design in an experimental study. Researcher degrees of freedom may be greatest for observational qualitative analyses.