1.2.1 Local Average Treatment Effects

Sometimes you give a medicine but only a nonrandom sample of people actually try to use it. Can you still estimate the medicine’s effect?

Say that people are one of 3 types:

1. \(n_a\) “always takers” have \(X=1\) no matter what and have average outcome \(\overline{y}_a\)
2. \(n_n\) “never takers” have \(X=0\) no matter what with outcome \(\overline{y}_n\)
3. \(n_c\) “compliers have” \(X=Z\) and average outcomes \(\overline{y}^1_c\) if treated and \(\overline{y}^0_c\) if not.

1.2.2 Local Average Treatment Effects

Sometimes you give a medicine but only a non random sample of people actually try to use it. Can you still estimate the medicine’s effect?

We can figure something about types:

1.2.3 Local Average Treatment Effects

You give a medicine to 50% but only a non random sample of people actually try to use it. Can you still estimate the medicine’s effect?

Key insight: the contributions of the \(a\)s and \(n\)s are the same in the \(Z=0\) and \(Z=1\) groups so if you difference you are left with the changes in the contributions of the \(c\)s.

1.2.4 Local Average Treatment Effects

Average in \(Z=0\) group: \(\frac{{n_c} \overline{y}^0_{c}+ \left(n_{n}\overline{y}_{n} +{n_a} \overline{y}_a\right)}{n_a+n_c+n_n}\)

Average in \(Z=1\) group: \(\frac{{n_c} \overline{y}^1_{c} + \left(n_{n}\overline{y}_{n} +{n_a} \overline{y}_a \right)}{n_a+n_c+n_n}\)

So, the difference is the ITT: \(({\overline{y}^1_c-\overline{y}^0_c})\frac{n_c}{n}\)

Last step:

\[ITT = ({\overline{y}^1_c-\overline{y}^0_c})\frac{n_c}{n}\]

\[\leftrightarrow\]

\[LATE = \frac{ITT}{\frac{n_c}{n}}= \frac{\text{Intent to treat effect}}{\text{First stage effect}}\]

1.2.5 The good and the bad of LATE

• You get a well-defined estimate even when there is non-random take-up
• May sometimes be used to assess mediation or knock-on effects
• But:
  • You need assumptions (monotonicity and the exclusion restriction – where were these used above?)
  • Your estimate is only for a subpopulation
  • The subpopulation is not chosen by you and is unknown
  • Different encouragements may yield different estimates since they may encourage different subgroups

1.3.1 Logic

We have group \(A\) that enters treatment at some point and group \(B\) that never does

The estimate:

\[\hat\tau = (\mathbb{E}[Y^A | post] - \mathbb{E}[Y^A | pre]) -(\mathbb{E}[Y^B | post] - \mathbb{E}[Y^B | pre])\] (how different is the change in \(A\) compared to the change in \(B\)?)

can be written:

\[\hat\tau = (\mathbb{E}[Y_1^A | post] - \mathbb{E}[Y_0^A | pre]) -(\mathbb{E}[Y_0^B | post] - \mathbb{E}[Y_0^B | pre])\]

1.3.2 Logic

Cleaning up

\[\hat\tau = (\mathbb{E}[Y_1^A | post] - \mathbb{E}[Y_0^A | pre]) -(\mathbb{E}[Y_0^B | post] - \mathbb{E}[Y_0^B | pre])\]

\[\hat\tau = (\mathbb{E}[Y_1^A | post] - \mathbb{E}[Y_0^A | post]) + ((\mathbb{E}[Y_0^A | post] - \mathbb{E}[Y_0^A | pre]) -(\mathbb{E}[Y_0^B | post] - \mathbb{E}[Y_0^B | pre]))\]

\[\hat\tau_{ATT} = \tau_{ATT} + \text{Difference in trends}\]

1.3.4 Diagnosis

Here only the two way fixed effects is unbiased and only for the ATT.

The ATT here is averaging over effects for treated units (later periods only). We know nothing about the size of effects in earlier periods when all units are in control!

design |> diagnose_design() 

1.3.6 Extends to multiple units easily

design |> redesign(n_units = 10) |> diagnose_design() 

1.3.9 Heterogeneity

• Things get much more complicated when there is (a) heterogeneous timing in treatment take up and (b) heterogeneous effects

• It’s only recently been appreciated how tricky things can get

• But we already have an intuition from our analysis of trials with heterogeneous assignment and heterogeneous effects:

• in such cases fixed effects analysis weights stratum level treatment effects by the variance in assignment to treatment

• something similar here

1.3.10 Staggared assignments

Just two units assigned at different times:

trend = 0

design <- 
  declare_model(
    unit = add_level(N = 2, ui = rnorm(N), I = 1:N),
    time = add_level(N = 6, ut = rnorm(N), T = 1:N, nest = FALSE),
    obs = cross_levels(by = join_using(unit, time))) +
  declare_model(
    potential_outcomes(Y ~ trend*T + (1+Z)*(I == 2))) +
  declare_assignment(Z = 1*((I == 1) * (T>3) + (I == 2) * (T>5))) +
  declare_measurement(Y = reveal_outcomes(Y~Z), 
                      I_c = I - mean(I)) +
  declare_inquiry(mean(Y_Z_1 - Y_Z_0)) +
  declare_estimator(Y ~ Z, label = "1. naive") + 
  declare_estimator(Y ~ Z + I, label = "2. FE1") + 
  declare_estimator(Y ~ Z + as.factor(T), label = "3. FE2") + 
  declare_estimator(Y ~ Z + I + as.factor(T), label = "4. FE3") + 
  declare_estimator(Y ~ Z*I_c + as.factor(T), label = "5. Sat")  

1.3.11 Staggared assignments diagnosis

1.3.14 Where do these numbers come?

• The estimate when we control for time is -1: this comes from weighting the time-stratum level effects according to the variance of assignment to each stratum
• it weights periods 4 and 5 only and equally, yielding the difference in outcomes for unit 1 in treatment (0) and group 2 in control (1)

design |> draw_data()  |> group_by(time) |> summarize(var = mean(Z)*(1-mean(Z))) |>
  mutate(weight = var/sum(var)) |> kable(digits = 2)

1.3.24 Chaisemartin and d’Haultfoeuille (2020)

Note the inquiry

run_design(design) |> kable(digits = 2)

1.3.26 Triple differencing

You are interested in the effects of influx of refugees on right wing voting

• You have (say more conservative) states with no refugees at any period
• You have (say more liberal) states with refugees post 2016 only

You want to do differences in differences comparing these states before and after

• However you worry that things change differntially in the conservative and liberal states: no parallel trends

• but you can identify areas within states that are more or less likely to be exposed and compare differnces in differences in the exposed and unexposed groups.

1.3.27 Triple differencing

So:

• Two types of states: \(L \in \{0,1\}\), only \(L=1\) types get refugee influx
• Two time periods: \(Post \in \{0,1\}\), refugee influx occurs in period \(Post = 1\)
• Two groups: \(B \in \{0,1\}\), \(B=1\) types affected by refugee influx

\[Y = \beta_0 + \beta_1 L + \beta_2 B + \beta_3 Post + \beta_4 LB + \beta_5 L Post + \beta_6 B Post + \beta_7L B Post + \epsilon\]

1.4.2 Intuition

• Kids born on 31 August start school a year younger than kids born on 1 September: does starting younger help or hurt?

• Kids born on 12 September 1983 are more likely to register Republican than kids born on 10 September 1983: can this identify the effects of registration on long term voting?

• A district in which Republicans got 50.1% of the vote get a Republican representative while districts in which Republicans got 49.9% of the vote do not: does having a Republican representative make a difference for these districts?

1.4.3 Argument for identification

Setting:

• Typically the decision is based on a value on a “running variable”, \(X\). e.g. Treatment if \(X > 0\)
• The estimand is \(\mathbb{E}[Y(1) - Y(0)|X=0]\)

Two arguments:

1. Continuity: \(\mathbb{E}[Y(1)|X=x]\) and \(\mathbb{E}[Y(0)|X=x]\) are continuous (at \(x=0\)) in \(x\): so \(\lim_{\hat x \rightarrow 0}\mathbb{E}[Y(0)|X=\hat x] = \mathbb{E}[Y(0)|X=\hat 0]\)

2. Local randomization: tiny things that determine exact values of \(x\) are as if random and so we can think of a local experiment around \(X=0\).

1.4.4 Argument for identification

Note:

• continuity argument requires continuous \(x\): granularity
• also builds off a conditional expectation function defined at \(X=0\)

Exclusion restriction is implicit in continuity: If something else happens at the threshold then the conditional expectation functions jump at the thresholds

Implicit: \(X\) is exogenous in the sense that units cannot adjust \(X\) in order to be on one or the other side of the threshold

1.4.11 Bandwidth tradeoff

rdd_design |> 
  redesign(bandwidth = seq(from = 0.05, to = 0.5, by = 0.05)) |> 
  diagnose_designs()

• As we increase the bandwidth, the lm bias gets worse, but slowly, while the error falls.
• The best bandwidth is relatively wide.
• This is more true for the optimal estimator.

1.4.12 Geographic RDs

Are popular in political science:

• Put a lot of pressure on assumption of no alternative treatment—including “random” country level shocks!
• Put a lot of pressure on no sorting assumptions (why was the border put where it was; why did units settle here or there?)
• Put a lot of pressure on SUTVA: people on one side are literally proximate to people on another

See Keele and Titiunik (2015)

1.5.2 SUTVA violations

Table: Potential outcomes for four units for different treatment profiles. \(D_i\) is an allocation and \(y_j(D_i)\) is the potential outcome for (row) unit \(j\) given (column) \(D_i\).

• The key is to think through the structure of spillovers.
• Here immediate neighbors are exposed
• In this case we can define a direct treatment (being exposed) and an indirect treatment (having a neighbor exposed) and we can work out the propensity for each unit of receiving each type of treatment
• These may be non uniform (here central types are more likely to have teated neighbors); but we can still use the randomization to assess effects

1.5.3 SUTVA violations

1.6.2 The problem of unidentified mediators

Which effects are identified by the random assignment of \(X\)?

1.6.3 The problem of unidentified mediators

An obvious approach is to first examine the (average) effect of X on M1 and then use another manipulation to examine the (average) effect of M1 on Y.

• But both of these average effects may be positive (for example) even if there is no effect of X on Y through M1.

1.6.4 The problem of unidentified mediators

An obvious approach is to first examine the (average) effect of X on M1 and then use another manipulation to examine the (average) effect of M1 on Y.

• Similarly both of these average effects may be zero even if X affects on Y through M1 for every unit!

1.6.5 The problem of unidentified mediators

Both instances of unobserved confounding between \(M\) and \(Y\):

1.6.6 The problem of unidentified mediators

Both instances of unobserved confounding between \(M\) and \(Y\):

1.6.7 The problem of unidentified mediators

• Another somewhat obvious approach is to see how the effect of \(X\) on \(Y\) in a regression is reduced when you control for \(M\).

• If the effect of \(X\) on \(Y\) passes through \(M\) then surely there should be no effect of \(X\) on \(Y\) after you control for \(M\).

• This common strategy associated with Baron and Kenny (1986) is also not guaranteed to produce reliable results. See for instance Green, Ha, and Bullock (2010)

1.6.8 Baron Kenny issues

df <- fabricate(N = 1000, 
                U = rbinom(N, 1, .5),     X = rbinom(N, 1, .5),
                M = ifelse(U==1, X, 1-X), Y = ifelse(U==1, M, 1-M)) 
            
list(lm(Y ~ X, data = df), 
     lm(Y ~ X + M, data = df)) |> texreg::htmlreg() 

1.6.9 The problem of unidentified mediators

• See Imai on better ways to think about this problem and designs to address it.

1.6.10 The problem of unidentified mediators: Quantities

• Using potential outcomeswe can describe a mediation effect as (see Imai et al): \[\delta_i(t) = Y_i(t, M_i(1)) - Y_i(t, M_i(0)) \textbf{ for } t = 0,1\]
• The direct effect is: \[\psi_i(t) = Y_i(1, M_i(t)) - Y_i(0, M_i(t)) \textbf{ for } t = 0,1\]
• This is a decomposition, since: \[Y_i(1, M_i(1)) - Y_1(0, M_i(0)) = \frac{1}{2}(\delta_i(1) + \delta_i(0) + \psi_i(1) + \psi_i(0)) \]
• If there are no interaction effects—ie \(\delta_i(1) = \delta_i(0), \psi_i(1) = \psi_i(0)\), then \[Y_i(1, M_i(1)) - Y_1(0, M_i(0)) = \delta_i + \psi_i\]

1.6.11 The problem of unidentified mediators: Solutions?

The bad news is that although a single experiment might identify the total effect, it can not identify these elements of the direct effect.

So:

• Check formal requirement for identification under single experiment design (“sequential ignorability”—that, conditional on actual treatment, it is as if the value of the mediation variable is randomly assigned relative to potential outcomes). But this is strong (and in fact unverifiable) and if it does not hold, bounds on effects always include zero (Imai et al)

• Consider sensitivity analyses

1.6.12 Implicit mediation

You can use interactions with covariates if you are willing to make assumptions on no heterogeneity of direct treatment effects over covariates.

eg you think that money makes people get to work faster because they can buy better cars; you look at the marginal effect of more money on time to work for people with and without cars and find it higher for the latter.

This might imply mediation through transport but only if there is no direct effect heterogeneity (eg people with cars are less motivated by money).

1.6.16 In CausalQueries

model |> update_model(data = truth |> make_data(n = 1000)) |>
  query_distribution(queries = queries, using = "posteriors") 

Why such poor behavior? Why isn’t weight going onto indirect effects?

Turns out the data is consistent with direct effects only: specifically that whenever \(M\) is responsive to \(X\), \(Y\) is responsive to \(X\).

1.6.17 In CausalQueries

1.7.4 The list experiment: Strategy

• Respondents are given a short list and a long list.

• The long list differs from the short list in having one extra item—the sensitive item

• We ask how many items in each list does a respondent agree with:

  • \(Y_i(0)\) is the number of elements on a short list that a respondent agrees with
  • \(Y_i(1)\) is the number of elements on a long list that a respondent agrees with
  • \(Y_i(1) - Y_i(0)\) is an indicator for whether an individual agrees with the sensitive item
  • \(\mathbb{E}[Y_i(1) - Y_i(0)]\) is the share of people agreeing with sensitive item

1.7.5 The list experiment: Simplified example

How many of these do you agree with:

[Note: this is obviously not a good list. Why not?]

1.7.14 Individual or group effects?

• This is typically used to estimate average levels

• However you can use it in the obvious way to get average levels for groups: this is equivalent to calculating group level heterogeneous effects

• Extending the idea you can even get individual level estimates: for instance you might use causal forests

• You can also use this to estimate the effect of an experimental treatment on an item that’s measured using a list, without requiring individual level estimates:

\[Y_i = \beta_0 + \beta_1Z_i + \beta_2Long_i + \beta_3Z_iLong_i\]