2.2.2 The topics

2.3.2 Optional

• Prepare a research design or short paper, perhaps building on existing work. Typically this contains:
  • a problem statement
  • a description of an experimental strategy to address the problem
  • empirical (ideally) or simulation based results (ok)
  • a discussion of implications
  • A passing paper will illustrate identify a causal effect of substantive interest credibly; a good paper will innovate on method or topic.

2.6.1 Task

We are collectively going to try to replicate an experimental design that was implemented in Germany.

We are free to modify this design as will. Our goal is to:

• Modify the design (or not)
• Generate a pre-analysis plan, including power estimates, if we can
• File an ethics application: HARD DEADLINE January 19
• Actually implement it!
• Run analysis
• Short write up

We are going to make mistakes and learn to work as a team.

2.6.3 Starter pack I

Why do native Europeans discriminate against Muslim immigrants? Can shared ideas between natives and immigrants reduce discrimination? We hypothesize that natives’ bias against Muslim immigrants is shaped by the belief that Muslims hold conservative attitudes about women’s rights and this ideational basis for discrimination is more pronounced among native women. We test this hypothesis in a large-scale field experiment conducted in 25 cities across Germany, during which 3,797 unknowing bystanders were exposed to brief social encounters with confederates who revealed their ideas regarding gender roles. We find significant discrimination against Muslim women, but this discrimination is eliminated when Muslim women signal that they hold progressive gender attitudes. Through an implicit association test and a follow-up survey among German adults, we further confirm the centrality of ideational stereotypes in structuring opposition to Muslims. Our findings have important implications for reducing conflict between native–immigrant communities in an era of increased cross-border migration.

2.6.6 Starter Pack II : A protest study

Team

2.6.7 Starter Pack II : A protest study

Despite decades of scholarship on protest effects, we know little about how bystanders—citizens who observe protests without participating—are affected by them. Understanding the impact of protest on bystanders is crucial as they constitute a growing audience whose latent support, normative beliefs, and concrete actions can make or break a movement’s broader societal impact. To credibly assess the effects of protests on observers, we design and implement a field experiment in Berlin in which we randomly route pedestrians past (treatment) or away from (control) three large-scale Fridays for Future (FFF) climate strikes. Using data gathered on protest days as well as through a one-month follow-up survey, we find evidence for a substantial increase in immediate donations to climate causes but no detectable impact on climate attitudes, vote intentions, or norm perceptions. Our findings challenge the prevailing assumption in both scholarship and public discourse that protest effects operate via impacts on public opinion and call for renewed theorizing that centers on observers’ immediate behavioral activation.

2.6.9 Starter Pack II : A protest study

Results

2.8.2 Good coding rules

• Metadata first
• Call packages at the beginning: use pacman
• Put options at the top
• Call all data files once, at the top. Best to call directly from a public archive, when possible.
  
• Use functions and define them at the top: comment them; useful sometimes to illustrate what they do
• Replicate first, re-analyze second. Use sections.
• (For replications) Have subsections named after specific tables, figures or analyses

2.8.4 Collaborative coding / writing

• Do not get in the business of passing attachments around
• Documents in some cloud: git, osf, Dropbox, Drive, Nextcloud
• General rule: only post non sensitive, non proprietary material
• Share self contained folders; folders contain a small set of live documents plus an archive. Old versions of documents are in archive. Only one version of the most recent document is in a main folder.
• Data is self contained folder (in) and is never edited directly
• Update to github frequently

3.5.3 Innovative designs

Randomization of

• encouragements to gain citizenship (New York)
• where police are stationed (India)
• how government tax collectors get paid (do they get a share?) (Pakistan)
• the voting rules for determining how decisions get made (Afghanistan)
• of populations to peacekeepers (Liberia)
• of ex-combatants out of their networks (Indonesia)
• students to ethnically homogeneous or ethnically diverse schools (Ethiopia)

3.6.1 Constraint: Is it ethical to manipulate subjects for research purposes?

• There is no foundationless answer to this question.

• Belmont principles commonly used for guidance:

  1. Respect for persons
  2. Beneficence
  3. Justice

• Unfortunately, operationalizing these requires further ethical theories. (1) is often operationalized by informed consent (a very liberal idea). (2) and (3) sometimes by more utiliarian principles

• The major focus on (1) by IRBs might follow from the view that if subjects consent, then they endorse the ethical calculations made for 2 and 3 — they think that it is good and fair.

• Trickiness: can a study be good or fair because of implications for non-subjects?

3.6.2 Is it ethical to manipulate subjects for research purposes?

• Many (many) field experiments have nothing like informed consent.

• For example, whether the government builds a school in your village, whether an ad appears on your favorite radio show, and so on.

• Consider three cases:

  1. You work with a nonprofit to post (true?) posters about the crimes of politicians on billboards to see effects on voters
  2. You hire confederates to offer bribes to police officers to see if they are more likely to bend the law for coethnics
  3. The British government asks you to work on figuring out how the use of water cannons helps stop rioters rioting

3.6.3 Is it ethical to manipulate subjects for research purposes?

• Consider three cases:

  • You work with a nonprofit to post (true?) posters about the crimes of politicians on billboards to see effects on voters
  • You hire confederates to offer bribes to police officers to see if they are more likely to bend the law for coethnics
  • The British government asks you to work on figuring out how the use of water cannons helps stop rioters rioting

• In all cases, there is no consent given by subjects.

• In 2 and 3, the treatment is possibly harmful for subjects, and the results might also be harmful. But even in case 1, there could be major unintended harmful consequences.

• In cases 1 and 3, however, the “intervention” is within the sphere of normal activities for the implementer.

3.6.4 Constraint: Is it ethical to manipulate subjects for research purposes?

• Sometimes it is possible to use this point of difference to make a “spheres of ethics” argument for “embedded experimentation.”

• Spheres of Ethics Argument: Experimental research that involves manipulations that are not normally appropriate for researchers may nevertheless be ethical if:

  • Researchers and implementers agree on a division of responsibility where implementers take on responsibility for actions
  • Implementers have legitimacy to make these decisions within the sphere of the intervention
  • Implementers are indeed materially independent of researchers (no swapping hats)

3.6.5 Constraint: Is it ethical to manipulate subjects for research purposes?

• Difficulty with this argument:
  • Question begging: How to determine the legitimacy of the implementer? (Can we rule out Nazi doctors?)

Otherwise keep focus on consent and desist if this is not possible

3.6.7 APSA Ethics: General [Abbr]

1. Political science researchers should respect autonomy, consider the wellbeing of participants and other people affected by their research, and be open about the ethical issues they face.

2. Political science researchers have an individual responsibility to consider the ethics of their research related activities and cannot outsource ethical reflection to review boards, other institutional bodies, or regulatory agencies.

3. These principles describe the standards of conduct and reflexive openness that are expected of political science researchers. … [In cases of reasonable deviations], researchers should acknowledge and justify deviations in scholarly publications and presentations of their work.

3.6.8 APSA Ethics: Power

4. When designing and conducting research, political scientists should be aware of power differentials between researcher and researched, and the ways in which such power differentials can affect the voluntariness of consent and the evaluation of risk and benefit.

1. especially with low-power or vulnerable participants
2. covert or deceptive research with more than minimal harm may sometimes be ethically permissible in research with powerful parties

3.6.9 APSA Ethics: Consent

5. Political science researchers should generally seek informed consent from individuals who are directly engaged by the research process, especially if research involves more than minimal risk of harm or if it is plausible to expect that engaged individuals would withhold consent if consent were sought.

1. informed and voluntary
2. continuing consent
3. observation of public behavior does not usually invoke this principle of consent.
4. forgoing perhaps if minimal risk, or seeking consent increases the risks for participants
5. discuss consent in publications and presentations

3.6.11 APSA Ethics: Harm and Trauma

7. Political science researchers should consider the harms associated with their research.

• Researchers should generally avoid harm when possible, minimize harm when avoidance is not possible, and not conduct research when harm is excessive.

• do not limit concern to physical and psychological risks to the participant.

8. Political science researchers should anticipate and protect individual participants from trauma stemming from participation in research.

3.6.12 APSA Ethics: Confidentiality

9. Political science researchers should generally keep the identities of research participants confidential; when circumstances require, researchers should adopt the higher standard of ensuring anonymity.

1. Researchers should clearly communicate assurances of confidentiality / anonymity
2. If confidentiality bit provided, communicate this and justify c./d. consider risks at all stages
3. Researchers who determine that it would be unethical to share materials derived from human subjects should be prepared to justify their decision to journal editors, to reviewers, etc

3.6.13 APSA Ethics: Impact

10. Political science researchers conducting studies on political processes should consider the broader social impacts of the research process as well as the impact on the experience of individuals directly engaged by the research. In general, political science researchers should not compromise the integrity of political processes for research purposes without the consent of individuals that are directly engaged by the research process.

1. cases in which research that produces impacts on political processes without consent of individuals directly engaged by the research might be appropriate. [examples]

2. Studies of interventions by third parties do not usually invoke this principle on impact. [details]

3. This principle is not intended to discourage any form of political engagement by political scientists in their non-research activities or private lives.

4. researchers should report likely impacts

3.6.15 APSA Ethics: Shared Responsibility

12. The responsibility to promote ethical research goes beyond the individual researcher or research team.

1. Mentors, advisors, dissertation committee members, and instructors

2. Graduate programs in political science should include ethics instruction in their formal and informal graduate curricula;

3. Editors and reviewers should encourage researchers to be open about the ethical decisions …

4. Journals, departments, and associations should incorporate ethical commitments into their mission, bylaws, instruction, practices, and procedures.

3.8.2 Contentious Issues

Experimental researchers are deeply engaged in the movement towards more transparency social science research.

Contentious issues (mostly):

• Data. How soon should you make your data available? My view: as soon as possibe. Along with working papers and before publication. Before it affects policy in any case. Own the ideas not the data.

  • Hard core: no citation without (analytic) replication. Perhaps. Non-replicable results should not be influencing policy.

• Where should you make your data available? Dataverse is focal for political science. Not personal website (mea culpa)

• What data should you make available? Disagreement is over how raw your data should be. My view: as raw as you can but at least post cleaning and pre-manipulation.

3.8.3 Open science checklist

Experimental researchers are deeply engaged in the movement towards more transparency social science research.

• Should you register?: Hard to find reasons against. But case strongest in testing phase rather than exploratory phase.

• Registration: When should you register? My view: Before treatment assignment. (Not just before analysis, mea culpa)

• Registration: Should you deviate from a preanalysis plan if you change your mind about optimal estimation strategies. My view: Yes, but make the case and describe both sets of results.

3.8.18 Myth: Concerns about fishing presuppose researcher dishonesty

• Fishing can happen in very subtle ways, and may seem natural and justifiable.

• Example:

  • I am interested in whether more democratic institutions result in better educational outcomes.
  • I examine the relationship between institutions and literacy and between institutions and school attendance.
  • The attendance measure is significant and the literacy one is not. Puzzled, I look more carefully at the literacy measure and see various outliers and indications of measurement error. As I think more I realize too that literacy is a slow moving variable and may not be the best measure anyhow. I move forward and start to analyze the attendance measure only, perhaps conducting new tests, albeit with the same data.

3.8.22 Myth: Registration only makes sense for experimental studies, not for observational studies

• The key distinction is between prospective and retrospective studies.

• Not between experimental and observational studies.

• A reason (from the medical literature) why registration is especially important for experiments: because you owe it to subjects

• A reason why registration is less important for experiments: because it is more likely that the intended analysis is implied by the design in an experimental study. Researcher degrees of freedom may be greatest for observational qualitative analyses.

4.1.1 Motivation

The intervention based motivation for understanding causal effects:

• We want to know if a particular intervention (like aid) caused a particular outcome (like reduced corruption).
• We need to know:
  1. What happened?
  2. What would the outcome have been if there were no intervention?
• The problem:
  1. … this is hard
  2. … this is impossible

The problem in 2 is that you need to know what would have happened if things were different. You need information on a counterfactual.

4.1.2 Potential Outcomes

• For each unit, we assume that there are two post-treatment outcomes: \(Y_i(1)\) and \(Y_i(0)\).
  • \(Y(1)\) is the outcome that would obtain if the unit received the treatment.
  • \(Y(0)\) is the outcome that would obtain if it did not.
• The causal effect of Treatment (relative to Control) is: \(\tau_i = Y_i(1) - Y_i(0)\)
• Note:
  • The causal effect is defined at the individual level.
  • There is no “data generating process” or functional form.
  • The causal effect is defined relative to something else, so a counterfactual must be conceivable (did Germany cause the second world war?).
  • Are there any substantive assumptions made here so far?

4.1.10 Causal claims: Contribution or attribution?

The counterfactual model is about attribution in a very conditional sense.

• This is problem for research programs that define “explanation” in terms of figuring out the things that cause \(Y\)

• Real difficulties conceptualizing what it means to say one cause is more important than another cause. What does that mean?

4.1.19 Causal claims: The rub and the solution

• Now for some magic. We really want to estimate: \[ \tau_i = Y_i(1) - Y_i(0)\]

• BUT: We never can observe both \(Y_i(1)\) and \(Y_i(0)\)

• Say we lower our sights and try to estimate an average treatment effect: \[ \tau = \mathbb{E} [Y(1)-Y(0)]\]

• Now make use of the fact that \[\mathbb E[Y(1)-Y(0)] = \mathbb E[Y(1)]- \mathbb E [Y(0)] \]

• In words: The average of differences is equal to the difference of averages.

• The magic is that while we can’t hope to measure the differences; we are good at measuring averages.

4.1.20 Causal claims: The rub and the solution

• So we want to estimate \(\mathbb{E} [Y(1)]\) and \(\mathbb{E} [Y(0)]\).
• We know that we can estimate averages of a quantity by taking the average value from a random sample of units
• To do this here we need to select a random sample of the \(Y(1)\) values and a random sample of the \(Y(0)\) values, in other words, we randomly assign subjects to treatment and control conditions.
• When we do that we can in fact estimate: \[ \mathbb {E}_N[Y_i(1) | Z_i = 1) - \mathbb {E}_N(Y_i(0) | Z_i = 0]\] which in expectation equals: \[ \mathbb{E} [Y_i(1) | Z_i = 1 \text{ or } Z_i = 0] - \mathbb{E} [Y_i(0) | Z_i = 1 \text{ or } Z_i = 0]\]

4.1.21 Causal claims: The rub and the solution

• This highlights a deep connection between random assignment and random sampling: when we do random assignment we are in fact randomly sampling from different possible worlds.

4.1.28 Exercise your potential outcomes 1

Consider the following potential outcomes table:

Questions for us: What are the unit level treatment effects? What is the average treatment effect?

4.1.29 Exercise your potential outcomes 2

Consider the following potential outcomes table:

Questions for us: Fill in the blanks.

• Assuming a constant treatment effect of \(+1\)
• Assuming a constant treatment effect of \(-1\)
• Assuming an average treatment effect of \(0\)

What is the actual treatment effect?

4.3.3 Heterogeneous Effects with Endogeneous Categories

• V(t): Violence(Treatment)
• R(t, v): Reporting(Treatment, Violence)

• Expected reporting given violence in control = Pr(Type 1) (explanation: both types see violence but only Type 1 reports)

• Expected reporting given violence in treatment = 100% (explanation: only Type 1 sees violence and this type also reports)

So you might infer a large effect on violence reporting.

Question: What is the actual effect of treatment on the propensity to report violence?

4.3.9 Missing data can create an endogeneous subgroup problem

• It is well known that missing data can undo the magic of random assignment.
• One seemingly promising approach is to match into pairs ex ante and drop pairs together ex post.
• Say potential outcomes looked like this (2 pairs of 2 units):

4.3.10 Missing data

• Say though that treated cases are likely to drop out of the sample if things go badly (e.g. they get a negative score or die)
• Then you might see no attrition if those would-be attritors are not treated.
• You might assume you have no problem (after all, no attrition).
• No missing data when the normal cases happens to be selected

4.3.11 Missing data

• But in cases in which you have attrition, dropping the pair doesn’t necessarily help.
• The problem is potential missingness still depends on potential outcomes
• The kicker is that the method can produce bias even if (in fact) there is no attrition!
• But missing data when the vulnerable cases happens to be selected

4.3.14 Multistage games

Question: Does visibility alter the extent to which subjects follow norms to punish antisocial behavior (and reward prosocial behavior)? Consider a trust game in which we are interested in how information on receivers affects their actions

What do we think? Does visibility make people react more to investments?

4.3.18 Multistage games

What to do?

Solutions?

1. Analysis could focus on the effect of treatment on respondent behavior, directly.
   • This would get the correct answer but to a different question (Does information affect the share of contributions returned by subjects on average?)
2. Strategy method can sometimes help address the problem, but note that that is (a) changing the question and (b) putting demands on respondent imagination and honesty
3. First mover action could be directly manipulated, but unless deception is used that is also changing the question
4. First movers could be selected because they act in predictable ways (bordering on deception?)

Take away: Proceed with extreme caution when estimating effects beyond the first stage.

4.5.1 Key insight

The most powerful results from the study of DAGs give procedures for figuring out when conditioning aids or hinders causal identification.

• You can read off a confounding variable from a DAG.
  • You figure out what to condition on for causal identification.
• You can read off “colliders” from a DAG
  • Sometimes you have to avoid conditioning on these
• Sometimes a variable might be both, so
  • you have to condition on it
  • you have to avoid conditioning on it
  • Ouch.

4.7.3 Conditional independence from graphs

\(A\) and \(B\) are conditionally independent, given \(C\) if on every path between \(A\) and \(B\):

• there is some chain (\(\bullet\rightarrow \bullet\rightarrow\bullet\) or \(\bullet\leftarrow \bullet\leftarrow\bullet\)) or fork (\(\bullet\leftarrow \bullet\rightarrow\bullet\)) with the central element in \(C\),

or

• there is an inverted fork (\(\bullet\rightarrow \bullet\leftarrow\bullet\)) with the central element (and its descendants) not in \(C\)

Notes:

• In this case we say that \(A\) and \(B\) are d-separated by \(C\).
• \(A\), \(B\), and \(C\) can all be sets
• Note that a path can involve arrows pointing any direction \(\bullet\rightarrow \bullet\rightarrow \bullet\leftarrow \bullet\rightarrow\bullet\)

4.7.4 Test yourself

Are A and D unconditionally independent:

• if you do not condition on anything?
• if you condition on B?
• if you condition on C?
• if you condition on B and C?

4.7.5 Back to this example

• \(Z = f_1(U_1, U_2)\)
• \(X = f_2(U_2)\)
• \(Y = f_3(X, U_1)\)

1. Let’s graph this
2. Now: say we removed the arrow from \(X\) to \(Y\)
   • Would you expect to see a correlation between \(X\) and \(Y\) if you did not control for \(Z\)
   • Would you expect to see a correlation between \(X\) and \(Y\) if you did control for \(Z\)

4.7.6 Back to this example

Now: say we removed the arrow from \(X\) to \(Y\)

• Would you expect to see a correlation between \(X\) and \(Y\) if you did not control for \(Z\)?
• Would you expect to see a correlation between \(X\) and \(Y\) if you did control for \(Z\)?

4.7.7 Conditional distributions given do operations

We nor formalize things a little more:

• The probability of outcome \(x\) can always be written in the form \[P(X_1 = x_1)P(X_2 = x_2|X_1=x_1)(X_3 = x_3|X_1=x_1, X_2 = x_2)\dots\]

• This can be done with any ordering of variables.

• We want to describe the distribution in a simpler way that takes account of parent-child relations and that can be used to capture interventions.

4.7.8 Conditional distributions given do operations

• Given an ordering of variables, the Markovian parents of variable \(X_j\) are the minimal set of variables such that when you condition on these, \(X_j\) is independent of all other prior variables in the ordering

\[P: P(x_1,x_2,\dots x_n) = \prod_{}P(x_j|pa_j)\]

• A DAG is “causal Bayesian network” or “Causal DAG” if (and only if) the probability distribution resulting from setting some set \(X_i\) to \(\hat{x}'_i\) (i.e. do(X=x')) is:

\[P_{\hat{x}_i}: P(x_1,x_2,\dots x_n|\hat{x}_i) = \mathbb{I}(x_i = x_i')\prod_{-i}P(x_j|pa_j)\]

where the parents \(pa_j\) are the parents on the graph (parents of \(x_i\) are the nodes with arrows pointing into \(x_i\).)

4.7.9 Conditional distributions given do operations

So:

\[P_{\hat{x}_i}: P(x_1,x_2,\dots x_n|\hat{x}_i) = \mathbb{I}(x_i = x_i')\prod_{-i}P(x_j|pa_j)\]

• This means that there is only probability mass on vectors in which \(x_i = x_i'\) (reflecting the success of control) and all other variables are determined by their parents, given the values that have been set for \(x_i\).

• Such expressions will be critical later when we want to consider identificatation.

• They let us assess whether the probability of an outcome \(y\), say, depends on the value of some other node, given some other node, or given interventions on some other node.

4.7.10 Conditional distributions given do operations

Illustration, say we have binary \(X\) causes binary \(M\) which cases binary \(Y\); say we intervene and set \(M=1\). Then what is the distribution of \((x,m,y)\)?

It is:

\[\Pr(x,m,y) = \Pr(x)\mathbb I(M = 1)\Pr(y|m)\]

4.8.1 From graphs to Causal Models

A “causal model” is:

1.Variables

• An ordered list of \(n\) endogenous nodes, \(\mathcal{V}= (V^1, V^2,\dots, V^n)\), with a specification of a range for each of them
• A list of \(n\) exogenous nodes, \(\Theta = (\theta^1, \theta^2,\dots , \theta^n)\)

2. A list of \(n\) functions \(\mathcal{F}= (f^1, f^2,\dots, f^n)\), one for each element of \(\mathcal{V}\) such that each \(f^i\) takes as arguments \(\theta^i\) as well as elements of \(\mathcal{V}\) that are prior to \(V^i\) in the ordering

3. A probability distribution over \(\Theta\)

4.8.4 Effects on a DAG

For instance the simplest model consistent with \(X \rightarrow Y\):

• Endogenous Nodes = \(\{X, Y\}\), both with range \(\{0,1\}\)

• Exogenous Nodes = \(\{\theta^X, \theta^Y\}\), with ranges \(\{\theta^X_0, \theta^X_1\}\) and \(\{\theta^Y_{00}\theta^Y_{01}, \theta^Y_{10}, \theta^Y_{11}\}\)

• Functional equations:

  • \(f_Y\): \(\theta^Y =\theta^Y_{ij} \rightarrow \{Y = i \text{ if } X=0; Y = j \text{ if } X=1\}\)
  • \(f_X\): \(\theta^X =\theta^X_{i} \rightarrow \{X = i\}\)

• Distributions on \(\Theta\): \(\Pr(\theta^i = \theta^i_k) = \lambda^i_k\)

4.8.6 Recap: Things you need to know about causal inference

1. A causal claim is a statement about what didn’t happen.
2. If you know that \(A\) causes \(B\) and that \(B\) causes \(C\), this does not mean that you know that \(A\) causes \(C\).
3. There is no causation without manipulation.
4. There is a fundamental problem of causal inference.
5. You can estimate average causal effects even if you cannot observe any individual causal effects.
6. Estimating average causal effects via differences in means does not require that treatment and control groups are identical.
7. Estimating average causal effects via differences in means is fraught when you condition on post treatment variables or on colliders.

4.11.1 Estimands: ATE, ATT, ATC, S-, P-

• ATE is Average Treatment Effect (all units)
• ATT is Average Treatment Effect on the Treated
• ATC is Average Treatment Effect on the Controls

4.11.2 Estimands: ATE, ATT, ATC, S-, P-

Say that units are randomly assigned to treatment in different strata (maybe just one); with fixed, though possibly different, shares assigned in each stratum. Then the key estimands and estimators are:

where \(x\) indexes strata, \(p_x\) is the share of units in each stratum that is treated, and \(w_x\) is the size of a stratum.

4.11.4 Broader classes of estimands: LATE/CATE

The CATEs are conditional average treatment effects, for example the effect for men or for women. These are straightfoward.

However we might also imagine conditioning on unobservable or counterfactual features.

• The LATE (or CACE: “complier average causal effect”) asks about the effect of a treatment (\(X\)) on an outcome (\(Y\)) for people that are responsive to an encouragement (\(Z\))

\[LATE = \frac{1}{|C|}\sum_{j\in C}(Y_j(X=1) - Y_j(X=0))\] \[C:=\{j:X_j(Z=1) > X_j(Z=0) \}\]

We will return to these in the study of instrumental variables.

4.11.7 Distribution estimands

Many inquiries are averages of individual effects, even if the groups are not known,

But they do not have to be:

• Inquiries might relate to distributional quantities such as:

  • The effect of treatment on the variance in outcomes: \(var(Y(1)) - var(Y(0))\)
  • The variance of treatment effects: \(var(Y(1) - Y(0))\)
  • Other inequality measures (e.g. Ginis; (Imbens and Rubin (2015) 20.3.2))

You might even be interested in \(\min(Y_i(1) - Y_i(0))\).

4.11.8 Spillover estimands

There are lots of interesting “spillover” estimands.

Imagine there are three individuals and each person’s outcomes depends on the assignments of all others. For instance \(Y_1(Z_1, Z_2, Z_3\), or more generally, \(Y_i(Z_i, Z_{i+1 (\text{mod }3)}, Z_{i+2 (\text{mod }3)})\).

Then three estimands might be:

• \(\frac13\left(\sum_{i}{Y_i(1,0,0) - Y_i(0,0,0)}\right)\)
• \(\frac13\left(\sum_{i}{Y_i(1,1,1) - Y_i(0,0,0)}\right)\)
• \(\frac13\left(\sum_{i}{Y_i(0,1,1) - Y_i(0,0,0)}\right)\)

Interpret these. What others might be of interest?

4.11.9 Differences in CATEs and interaction estimands

A difference in CATEs is a well defined estimand that might involve interventions on one node only:

• \(\mathbb{E}_{\{W=1\}}[Y(X=1) - Y(X=0)] - \mathbb{E}_{\{W=0\}}[Y(X=1) - Y(X=0)]\)

It captures differences in effects.

An interaction is an effect on an effect:

• \(\mathbb{E}[Y(X=1, W=1) - Y(X=0, W=1)] - \mathbb{E}[Y(X=1, W=0) - Y(X=0, W=0)]\)

Note in the latter the expectation is taken over the whole population.

4.11.10 Mediation estimands and complex counterfactuals

Say \(X\) can affect \(Y\) directly, or indirectly through \(M\). then we can write potential outcomes as:

• \(Y(X=x, M=m)\)
• \(M(X=x)\)

We can then imagine inquiries of the form:

• \(Y(X=1, M=M(X=1)) - Y(X=0, M=M(X=0))\)
• \(Y(X=1, M=1) - Y(X=0, M=1)\)
• \(Y(X=1, M=M(X=1)) - Y(X=1, M=M(X=0))\)

Interpret these. What others might be of interest?

4.13.2 Identification (Definition)

• Identifiability Let \(Q(M)\) be a query defined over a class of models \(\mathcal M\), then \(Q\) is identifiable if \(P(M_1) = P(M_2) \rightarrow Q(M_1) = Q(M_1)\).

• Identifiability with constrained data Let \(Q(M)\) be a query defined over a class of models \(\mathcal M\), then \(Q\) is identifiable from features \(F(M)\) if \(F(M_1) = F(M_2) \rightarrow Q(M_1) = Q(M_1)\).

Based on Defn 3.2.3 in Pearl.

• Essentially: Each underlying value produces a unique data distribution. When you see that distribution you recover the parameter.

4.13.3 Example without identification 1

Informally a quantity is “identified” if it can be “recovered” once you have enough data.

• Say for example average wage is \(x^m\) for men and \(x^w\) for women (in some very large population).
• If I gather lots and lots of data on the wages of (male and female) couples, e.g. \(x^c_i = x^m_i + x^w_i\) then, although this will be informative, it will never be sufficient to recover \(x^m\) for men and \(x^w\).
• I can recover \(x^c\), but there are too many combinations of possible values of \(x^m\) and \(x^w\) consistent with the observed data.

4.13.4 Example without identification 2

• share \(a\) of units have negative effects
• \(b\) have positive effects
• \(c\) that always have \(Y=0\) and
• \(d\) always have \(Y=1\).

Then with very large (experimental) data we observe:

What quantities are identified?

• \(b-a\)?
• \(d-c\)?
• \(b\)?

4.13.5 Example without identification 2

What if we :

• knew that \(a=0\)
• observed that \(\alpha_01=0\)

What quantities are now identified?

• \(b-a\)?
• \(d-c\)?
• \(b\)?

4.13.6 Identification : Goal

Our goal in causal inference is to estimate quantities such as:

\[\Pr(Y|\hat{x})\]

where \(\hat{x}\) is interpreted as \(X\) set to \(x\) by “external” control. Equivalently: \(do(X=x)\) or sometimes \(X \leftarrow x\).

• If this quantity is identifiable then we can recover it with infinite data.

• If it is not identifiable, then, even in the best case, we are not guaranteed to get the right answer.

Are there general rules for determining whether this quantity can be identified? Yes.

4.14.1 When to condition? What to condition on?

The key idea is that you want to find a set of variables such that when you condition on these you get what you would get if you used a do operation.

Intuition:

• You could imagine creating a “mutilated” graph by removing all the arrows leading out of X
• Then select a set of variables, \(Z\), such that \(X\) and \(Y\) are d-separated by \(Z\) on the the mutilated graph
• When you condition on these you are making sure that any covariation between \(X\) and \(Y\) is covariation that is due to the effects of \(X\)

4.14.7 Backdoor Criterion: (Pearl 1995)

The backdoor criterion is satisfied by \(Z\) (relative to \(X\), \(Y\)) if:

1. No node in \(Z\) is a descendant of \(X\)
2. \(Z\) blocks every backdoor path from \(X\) to \(Y\) (i.e. every path that contains an arrow into \(X\))

In that case you can identify the effect of \(X\) on \(Y\) by conditioning on \(Z\):

\[P(Y=y | \hat{x}) = \sum_z P(Y=y| X = x, Z=z)P(z)\] (This is eqn 3.19 in Pearl (2000))

4.14.9 Backdoor Proof

Following Pearl (2009), Chapter 11. Let \(T\) denote the set of parents of \(X\): \(T := pa(X)\), with (possibly vector valued) realizations \(t\). These might not all be observed.

If the backdoor criterion is satisfied, we have:

1. \(Y\) is independent of \(T\), given \(X\) and observed data, \(Z\) (since \(Z\) blocks backdoor paths)
2. \(X\) is independent of \(Z\) given \(T\). (Since \(Z\) includes only nondescendents)

• Key idea: The intervention level relates to the observational level as follows: \[p(y|\hat{x}) = \sum_{t\in T} p(t)p(y|x, t)\]

• Think of this as fully accounting for the (possibly unobserved) causes of \(X\), \(T\)

4.14.10 Backdoor Proof

We want to get to:

\[p(y|\hat{x}) = \sum_{t\in T} p(t)p(y|x, t)\]

• But of course we do not observe \(T\), rather we observe \(Z\). So we now need to write everything in terms of \(Z\) rather than \(T\).

We bring \(Z\) into the picture by writing:

\[p(y|\hat{x}) = \sum_{t\in T} p(t) \sum_z p(y|x, t, z)p(z|x, t)\]

now we want to get rid of \(T\)…

4.14.11 Backdoor Proof

now we want to get rid of \(T\)…

• Using the two conditions from the backdoor definition above:

  1. replace \(p(y|x, t, z)\) with \(p(y | x, z)\)
  2. replace \(p(z|x, t)\) with \(p(z|t)\)

This gives: \[p(y|\hat x) = \sum_{t \in T} p(t) \sum_z p(y|x, z)p(z|t)\]

Cleaning up, we can get rid of \(T\):

\[p(y|\hat{x}) = \sum_z p(y|x, z)\sum_{t\in T} p(z|t)p(t) = \sum_z p(y| x, z)p(z)\]

4.14.12 Backdoor proof figure

For intuition:

We would be happy if we could condition on the parent \(T\), but \(T\) is not observed. However we can use \(Z\) instead making use of the fact that:

1. \(p(y|x, t, z) = p(y | x, z)\) (since \(Z\) blocks)
2. \(p(z|x, t) = p(z|t)\) (since \(Z\) is upstream and blocked by parents, \(T\))

4.14.13 Adjustment criterion

See Shpitser, VanderWeele, and Robins (2012)

The adjustment criterion is satisfied by \(Z\) (relative to \(X\), \(Y\)) if:

1. no element of \(Z\) is a descendant in the mutilated graph of any variable \(W\not\in X\) which lies on a proper causal path from \(X\) to \(Y\)
2. \(Z\) blocks all noncausal paths from \(X\) to \(Y\)

Note:

• mutilated graph: remove arrows pointing into \(X\)
• proper pathway: A proper causal pathway from \(X\) to \(Y\) only intersects \(X\) at the endpoint

4.14.15 Frontdoor criterion (Pearl)

Consider this DAG:

• The relationship between \(X\) and \(Y\) is confounded by \(U\).
• However the \(X\rightarrow Y\) effect is the product of the \(X\rightarrow M\) effect and the \(M\rightarrow Y\) effect

Why?

4.14.16 Identification through the front door

If:

• \(M\) (possibly a set) blocks all directed paths from \(X\) to \(Y\)
• there is no backdoor path \(X\) to \(M\)
• \(X\) blocks all backdoor paths from \(M\) to \(Y\) and
• all (\(m,z\)) combinations arise with positive probability

Then \(\Pr(y| \hat x)\) is identifiable and given by:

\[\Pr(y| \hat x) = \sum_m\Pr(m|x)\sum_{x'}\left(\Pr(y|m,x')\Pr(x')\right)\]

4.14.17 Frontdoor criterion (Proof)

We want to get \(\Pr(y | \hat x)\)

From the graph the joint distribution of variables is:

\[\Pr(x,m,y,u) = \Pr(u)\Pr(x|u)\Pr(m|x)\Pr(y|m,u)\] If we intervened on \(X\) we would have (\(\Pr(X = x |u)=1\)):

\[\Pr(m,y,u | \hat x) = \Pr(u)\Pr(m|x)\Pr(y|m,u)\] If we sum up over \(u\) and \(m\) we get:

\[\Pr(m,y| \hat x) = \Pr(m|x)\sum_u\left(\Pr(y|m,u)\Pr(u)\right)\] \[\Pr(y| \hat x) = \sum_m\Pr(m|x)\sum_u\left(\Pr(y|m,u)\Pr(u)\right)\]

The first part is fine; the second part however involves \(u\) which is unobserved. So we need to get the \(u\) out of \(\sum_u\left(\Pr(y|m,u)\Pr(u)\right)\).

4.14.18 Frontdoor criterion

Now, from the graph:

1. \(M\) is d-separated from \(U\) by \(X\):

\[\Pr(u|m, x) = \Pr(u|x)\] 2. \(X\) is d-separated from \(Y\) by \(M\), \(U\)

\[\Pr(y|x, m, u) = \Pr(y|m,u)\] That’s enough to get \(u\) out of \(\sum_u\left(\Pr(y|m,u)\Pr(u)\right)\)

4.14.19 Frontdoor criterion

\[\sum_u\left(\Pr(y|m,u)\Pr(u)\right) = \sum_x\sum_u\left(\Pr(y|m,u)\Pr(u|x)\Pr(x)\right)\]

Using the 2 equalities we got from the graph:

\[\sum_u\left(\Pr(y|m,u)\Pr(u)\right) = \sum_x\sum_u\left(\Pr(y|x,m,u)\Pr(u|x,m)\Pr(x)\right)\]

So:

\[\sum_u\left(\Pr(y|m,u)\Pr(u)\right) = \sum_x\left(\Pr(y|m,x)\Pr(x)\right)\]

Intuitively: \(X\) blocks the back door between \(Z\) and \(Y\) just as well as \(U\) does

4.14.20 Frontdoor criterion

Substituting we are left with:

\[\Pr(y| \hat x) = \sum_m\Pr(m|x)\sum_{x'}\left(\Pr(y|m,x')\Pr(x')\right)\]

(The \('\) is to distinguish the \(x\) in the summation from the value of \(x\) of interest)

It’s interesting that \(x\) remains in the right hand side in the calculation of the \(m \rightarrow y\) effect, but this is because \(x\) blocks a backdoor from \(m\) to \(y\)

4.14.21 Front foor

Bringing all this together into a claim we have:

If:

• \(M\) (possibly a set) blocks all directed paths from \(X\) to \(Y\)
• there is no backdoor path \(X\) to \(M\)
• \(X\) blocks all backdoor paths from \(M\) to \(Y\) and
• all (\(m,z\)) combinations arise with positive probability

Then \(\Pr(y| \hat x)\) is identifiable and given by:

\[\Pr(y| \hat x) = \sum_m\Pr(m|x)\sum_{x'}\left(\Pr(y|m,x')\Pr(x')\right)\]

4.15.6 Illustration of identification failure from conditioning on a collider

U1 <- rnorm(10000);  U2 <- rnorm(10000)
Z  <- U1+U2
X  <- U2 + rnorm(10000)/2
Y  <- U1*2 + X

lm_robust(Y ~ X) |> tidy() |> kable(digits = 2)

lm_robust(Y ~ X + Z) |> tidy() |> kable(digits = 2)

4.15.9 Let’s look at that in dagitty

g <- dagitty("dag{U1 -> Z  ; U1 -> y ; 
             U2 -> Z ; U2 -> x  -> y; 
             Z -> x}")
adjustmentSets(g, exposure = "x", outcome = "y")

{ U1 }
{ U2, Z }

which means you have to adjust on an unobservable. Here we double check that including or not including “Z” is enough:

isAdjustmentSet(g, "Z", exposure = "x", outcome = "y")

[1] FALSE

isAdjustmentSet(g, NULL, exposure = "x", outcome = "y")

[1] FALSE

5.1.2 ATE: DIM in practice

df <- fabricatr::fabricate(N = 100, Z = rep(0:1, N/2), Y = rnorm(N) + Z)

# by hand
df |>
  summarize(Y1 = mean(Y[Z==1]), 
            Y0 = mean(Y[Z==0]), 
            diff = Y1 - Y0) |> kable(digits = 2)

# with estimatr
estimatr::difference_in_means(Y ~ Z, data = df) |>
  tidy() |> kable(digits = 2)

5.1.3 ATE: DIM in practice

We can also do this with regression:

estimatr::lm_robust(Y ~ Z, data = df) |>
  tidy() |> kable(digits = 2)

See Freedman (2008) on why regression is fine here

5.1.4 ATE: Blocks

Say now different strata or blocks \(\mathcal{S}\) had different assignment probabilities. Then you could estimate:

\[ \widehat{ATE} = \sum_{S\in \mathcal{S}}\frac{n_{S}}{n} \left(\frac{1}{n_{S1}}\sum_{S\cap T}y_i - \frac{1}{n_{S0}}\sum_{S\cap C}y_i \right) \]

Note: you cannot just ignore the blocks because assignment is no longer independent of potential outcomes: you might be sampling units with different potential outcomes with different probabilities.

However, the formula above works fine because selecting is random conditional on blocks.

5.1.5 ATE: Blocks

As a DAG this is just classic confounding:

make_model("Block -> Z ->Y <- Block") |> 
  plot(x_coord = c(2,1,3), y_coord = c(2, 1, 1))

5.1.6 ATE: Blocks in practice

Data with heterogeneous assignments:

df <- fabricatr::fabricate(
  N = 500, X = rep(0:1, N/2), 
  prob = .2 + .3*X,
  Z = rbinom(N, 1, prob),
  ip = 1/(Z*prob + (1-Z)*(1-prob)), # discuss
  Y = rnorm(N) + Z*X)

True effect is 0.5, but:

estimatr::difference_in_means(Y ~ Z, data = df) |>
  tidy() |> kable(digits = 2)

5.1.7 ATE: Blocks in practice

Averaging over effects in blocks

# by hand
estimates <- 
  df |>
  group_by(X) |>
  summarize(Y1 = mean(Y[Z==1]), 
            Y0 = mean(Y[Z==0]), 
            diff = Y1 - Y0,
            W = n())

estimates$diff |> weighted.mean(estimates$W)

[1] 0.7236939

# with estimatr
estimatr::difference_in_means(Y ~ Z, blocks = X, data = df) |>
  tidy() |> kable(digits = 2)

5.1.8 ATE with IPW

This also corresponds to the difference in the weighted average of treatment outcomes (with weights given by the inverse of the probability that each unit is assigned to treatment) and control outcomes (with weights given by the inverse of the probability that each unit is assigned to control).

• The average difference in means estimator is the same as what you would get if you weighted inversely by shares of units in different conditions inside blocks.

5.1.9 ATE with IPW in practice

# by hand
df |>
  summarize(Y1 = weighted.mean(Y[Z==1], ip[Z==1]), 
            Y0 = weighted.mean(Y[Z==0],  ip[Z==0]), # note !
            diff = Y1 - Y0)|> 
  kable(digits = 2)

# with estimatr
estimatr::difference_in_means(Y ~ Z, weights = ip, data = df) |>
  tidy() |> kable(digits = 2)

5.1.10 ATE with IPW

• But inverse propensity weighting is a more general principle, which can be used even if you do not have blocks.

• The intuition for it comes straight from sampling weights — you weight up in order to recover an unbiased estimate of the potential outcomes for all units, whether or not they are assigned to treatment.

• With sampling weights however you can include units even if their weight was 1. Why can you not include these units when doing inverse propensity weighting?

5.1.11 Illustration: Estimating treatment effects with terrible treatment assignments: Fixer

Say you made a mess and used a randomization that was correlated with some variable, \(U\). For example:

• The randomization is done in a way that introduces a correlation between Treatment Assignment and Potential Outcomes
• Then possibly, even though there is no true causal effect, we naively estimate a large one — enormous bias
• However since we know the assignment procedure we can fully correct for the bias

5.1.12 Illustration: Estimating treatment effects with terrible treatment assignments: Fixer

• In the next example, we do this using “inverse propensity score weighting.”
• This is exactly analogous to standard survey weighting — since we selected different units for treatment with different probabilities, we weight them differently to recover the average outcome among treated units (same for control).

5.1.13 Basic randomization: Fixer

Bad assignment, some randomization process you can’t understand (but can replicate) that results in unequal probabilities.

N <- 400
U <- runif(N, .1, .9)

 design <- 
   declare_model(N = N,
                 Y_Z_0 = U + rnorm(N, 0, .1),
                 Y_Z_1 = U + rnorm(N, 0, .1),
                 Z = rbinom(N, 1, U)) +
  declare_measurement(Y = reveal_outcomes(Y ~ Z)) +
  declare_inquiry(ATE = mean(Y_Z_1 - Y_Z_0)) +
  declare_estimator(Y ~ Z, label = "naive") 

5.1.14 Basic randomization: Fixer

Results is a sampling distribution not centered on the true effect (0)

diagnosis <- diagnose_design(design)

diagnosis$simulations_df |>
  ggplot(aes(estimate)) + geom_histogram() + facet_grid(~estimator) +
  geom_vline(xintercept = 0, color = "red")

5.1.15 A fix

To fix you can estimate the assignment probabilities by replicating the assignment many times:

probs <- replicate(1000, design |> draw_data() |> pull(Z)) |> apply(1, mean)

and then use these assignment probabilities in your estimator

design_2 <-
  design +
  declare_measurement(weights = Z/probs + (1-Z)/(1-probs)) +
  declare_estimator(Y ~ Z, weights = weights, label = "smart") 

5.1.16 Basic randomization: Fixer

Implied weights

draw_data(design_2) |> 
  ggplot(aes(probs, weights, color = factor(Z))) + 
  geom_point()

5.1.18 IPW with one unit!

This example is surprising but it helps you see the logic of why inverse weighting gets unbiased estimates (and why that might not guarantee a reasonable answer)

Imagine there is one unit with potential outcomes \(Y(1) = 2, Y(0) = 1\). So the unit level treatment effect is 1.

You toss a coin.

• If you assign to treatment you estimate: \(\hat\tau = \frac{2}{0.5} = 4\)
• If you assign to control you estimate: \(\hat\tau = -\frac{1}{0.5} = -2\)

So your expected estimate is: \[0.5 \times 4 - 0.5 \times (-2) = 1\]

Great on average but always lousy

5.1.19 Generalization: why IPW works

• Say a given unit is assigned to treatment with probability \(\pi_i\)
• We estimate the average \(Y(1)\) using

\[\hat{\overline{Y_1}} = \frac{1}n\left(\sum_i \frac{Z_iY_i(1)}{\pi_i}\right)\] With independent assignment the expected value of \(\hat{\overline{Y_1}}\) is just:

\[\mathbb{E}[\hat{\overline{Y_1}}] =\frac1n\left( \left(\pi_1 \frac{1\times Y_1(1)}{\pi_1} + (1-\pi_1) \frac{0\times Y_1(1)}{\pi_1}\right) + \left(\pi_2 \frac{1\times Y_2(1)}{\pi_2} + (1-\pi_2) \frac{0\times Y_1(1)}{\pi_2}\right) + \dots\right)\]

\[\mathbb{E}[\hat{\overline{Y_1}}] =\frac1n\left( Y_1(1) + Y_2(1) + \dots\right) = \overline{Y_1}\]

and similarly for \(\mathbb{E}[\hat{\overline{Y_0}}]\) and so using linearity of expectations:

\[\mathbb{E}[\widehat{\overline{Y_1 - Y_0}}] = \overline{Y_1 - Y_0}\]

5.1.20 Generalization: why IPW works

• Note we needed \(\pi_i >0\) and also \(\pi_i <1\) everywhere. Why?
• We used independence here; sampling theory is used to show similar results for e.g. complete randomization
• For blocked randomization this is easy to see

5.2.3 Var(ATE)

• In classical statistics we characterize our uncertainty over an estimate using an estimate of variance of the sampling distribution of the estimator.

• Key idea is we want to be able to say: how likely are we to have gotten such an estimate if the distribution of estimates associated with our design looked a given way.

• More specifically we want to estimate “standard error” or the “standard deviation of the sampling distribution”

(See Woolridge (2023) where the standard error is understood as the “estimate of the standard deviation of the sampling distribution”)

5.2.4 Variance and standard errors

Given:

• \(\hat\tau\) is an estimate for \(\tau\)
• \(\overline{x}\) is the average values of \(x\)

The variance of the estimator of \(n\) repeated ‘runs’ of a design is: \(Var(\hat{\tau}) = \frac{1}n\sum_i(\hat\tau_i - \overline{\hat\tau_i})^2\)

And the standard error is:

\(se(\hat{\tau}) = \sqrt{\frac{1}n\sum_i(\hat\tau_i - \overline{\hat\tau_i})^2}\)

5.2.5 Variance and standard errors

If we have a good measure for the shape of the sampling distribution we can start to make statements of the form:

• What are the chances that an estimate would be this large or larger?

If the sampling distribution is roughly normal, as it may be with large samples, then we can use procedures such as: “there is a 5% probability that an estimate would be more than 1.96 standard errors away from the mean of the sampling distribution”

5.2.6 Var(ATE)

• Key idea: You can estimate variance straight from the data, given knowledge of the assignment process and assuming well defined potential outcomes?

• Recall in general \(Var(x) = \frac{1}n\sum_i(x_i - \overline{x})^2\). here the \(x_i\)s are the treatment effect estimates we might get under different random assignments, the \(n\) is number of different assignments (assumed here all equally likely, but otherwise we can weight) and \(\overline{x}\) is the truth.

• For intuition imagine we have just two units \(A\), \(B\), with potential outcomes \(A_1\), \(A_0\), \(B_1\), \(B_0\).

• When there are two units with outcomes \(x_1, x_2\), the variance simplifies like this:

\[Var(x) = \frac{1}2\left(x_1 - \frac{x_1 + x_2}{2}\right)^2 + \frac{1}2\left(x_2 - \frac{x_1 + x_2}{2}\right)^2 = \left(\frac{x_1 - x_2}{2}\right)^2\]

5.2.7 Var(ATE)

In the two unit case the two possible treatment estimates are: \(\hat{\tau}_1=A_1 - B_0\) and \(\hat{\tau}_2=B_1 - A_0\), depending on what gets put into treatment. So the variance is:

\[Var(\hat{\tau}) = \left(\frac{\hat{\tau}_1 - \hat{\tau}_2}{2}\right)^2 = \left(\frac{(A_1 - B_0) - (B_1 - A_0)}{2}\right)^2 =\left(\frac{(A_1 - B_1) + (A_0 - B_0)}{2}\right)^2 \] which we can re-write as:

\[Var(\hat{\tau}) = \left(\frac{A_1 - B_1}{2}\right)^2 + \left(\frac{A_0 - B_0}{2}\right)^2+ 2\frac{(A_1 - B_1)(A_0-B_0)}{2}\] The first two terms correspond to the variance of \(Y(1)\) and the variance of \(Y(0)\). The last term is a bit pesky though, it corresponds to twice the covariance of \(Y(1)\) and \(Y(0)\).

5.2.8 Var(ATE)

How can we go about estimating this?

\[Var(\hat{\tau}) = \left(\frac{A_1 - B_1}{2}\right)^2 + \left(\frac{A_0 - B_0}{2}\right)^2+ 2\frac{(A_1 - B_1)(A_0-B_0)}{2}\]

In the two unit case it is quite challenging because we do not have an estimate for any of the three terms: we do not have an estimate for the variance in the treatment group or in the control group because we have only one observation in each case; and we do not have an estimate for the covariance because we don’t observe both potential outcomes for any case.

Things do look a bit better however with more units…

5.2.9 Var(ATE): Generalizing

From Freedman Prop 1 / Example 1 (using combinatorics!) we have:

\(V(\widehat{ATE}) = \frac{1}{n-1}\left[\frac{n_C}{n_T}V_1 + \frac{n_T}{n_C}V_0 + 2C_{01}\right]\)

… where \(V_0, V_1\) denote variances and \(C_{01}\) covariance

This is usefully rewritten as:

\[ \begin{split} V(\widehat{ATE}) & = \frac{1}{n-1}\left[\frac{n - n_T}{n_T}V_1 + \frac{n - n_C}{n_C}V_0 + 2C_{01}\right] \\ & = \frac{n}{n-1}\left[\frac{V_1}{n_T} + \frac{V_0}{n_C}\right] - \frac{1}{n-1}\left[V_1 + V_0 - 2C_{01}\right] \end{split} \]

where the final term is positive

5.2.12 Illustration of Neyman Conservative Estimator

An illustration of how conservative the conservative estimator of variance really is (numbers in plot are correlations between \(Y(1)\) and \(Y(0)\).

We confirm that:

1. the estimator is conservative
2. the estimator is more conservative for negative correlations between \(Y(0)\) and \(Y(1)\) — eg if those cases that do particularly badly in control are the ones that do particularly well in treatment, and
3. with \(\tau\) and \(V(Y(0))\) fixed, high positive correlations are associated with highest variance.

5.2.13 Illustration of Neyman Conservative Estimator

Here \(\rho\) is the unobserved correlation between \(Y(1)\) and \(Y(0)\); and \(\Delta\) is the final term in the sample variance equation that we cannot estimate.

5.2.14 Illustration of Neyman Conservative Estimator

5.2.16 Tighter Bounds On Variance Estimate

Example:

• Take a million-observation dataset, with treatment randomly assigned
• Assume \(Y(0)=0\) for everyone and \(Y(1)\) distributed normally with mean 0 and standard deviation of 1000.
• Note here the covariance of \(Y(1)\) and \(Y(0)\) is 0.
• Note the true variance of the estimated sample average treatment effect should be (approx) \(\frac{Var(Y(1))}{{1000000}} + \frac{Var(Y(0))}{{1000000}} = 1+0=1\), for an se of \(1\).
• But using the Neyman estimator (or OLS!) we estimate (approx) \(\frac{Var(Y(1))}{({1000000/2})} + \frac{Var(Y(0))}{({1000000/2})} = 2\), for an se of \(\sqrt{2}\).
• But we can recover the truth knowing the covariance between \(Y(1)\) and \(Y(0)\) is 0.

5.3.1 Calculate a \(p\) value in your head

• Illustrating \(p\) values via “randomization inference”

• Say you randomized assignment to treatment and your data looked like this.

Then:

• Does the treatment improve your health?
• What’s the \(p\) value for the null that treatment had no effect on anybody?

5.3.2 Calculate a \(p\) value in your head

• Illustrating \(p\) values via “randomization inference”
• Say you randomized assignment to treatment and your data looked like this.

Then:

• Does the treatment improve your health?
• What’s the \(p\) value for the null that treatment had no effect on anybody?

5.3.7 Randomization Inference

• Randomization inference can get more complicated when you want to test a null other than the sharp null of no effect.
• Say you wanted to test the null that the effect is 2 for all units. How do you do it?
• Say you wanted to test the null that an interaction effect is zero. How do you do it?
• In both cases by filling in a potential outcomes schedule given the hypothesis in question and then generating a test statistic

5.3.8 ri and CIs

It is possible to use this procedure to generate confidence intervals with a natural interpretation.

• The key idea is that we can use the same procedure to assess the probability of the data given a sharp null of no effect, but also a sharp null of any other **constant* effect.
• We can then see what set of effects we reject and what set we accept
• We are left with a set of values that we cannot reject at the 0.05 level.

5.3.9 ri and CIs in practice

candidates <- seq(-.05, .3, length = 50)

get_p <- function(j)
  (conduct_ri(
      formula = Y ~ Z,
      declaration = assignment,
      sharp_hypothesis = j,
      data = df,
      sims = 5000,
    ) |> summary()
  )$two_tailed_p_value

# Implement
ps <- candidates |> sapply(get_p)

5.3.10 ri and CIs in practice

Warning: calculating confidence intervals this way can be computationally intensive

5.3.11 ri with DeclareDesign

• DeclareDesign can do randomization inference natively
• The trick is to ensure that when calculating the \(p\) values the only stochastic component is the assignment to treatment when calculating the \(p\) values

5.3.12 ri with DeclareDesign (advanced)

Here we get minimal detectable effects by using a design that has two stage simulations so we can estimate the sampling distribution of summaries of the sampling distribution generated from reassignments.

test_stat <- function(data)
  with(data, data.frame(estimate = mean(Y[Z==1]) - mean(Y[Z==0])))

b <- 0

design <- 
  declare_model(N = 100,  Z = complete_ra(N), Y = b*Z + rnorm(N)) +
  declare_estimator(handler = label_estimator(test_stat), label = "actual")+
  declare_measurement(Z = sample(Z))  + # this is the permutation step
  declare_estimator(handler = label_estimator(test_stat), label = "null")

5.3.13 ri with DeclareDesign (advanced)

Simulations data frame from two step simulation. Note computational intensity as number of runs is the product of the sims vector. I speed things up by using a simple estimation function and also using parallelization.

library(future)
options(parallelly.fork.enable = TRUE) 
plan(multicore) 

simulations <- 
  design |> redesign(b = c(0, .25, .5, .75, 1)) |> 
  simulate_design(sims = c(200, 1, 1000, 1)) 

5.3.14 ri with DeclareDesign (advanced)

A snap shot of the simulations dataframe: we have multiple step 3 draws for each design and step 1 draw.

simulations |> head() |> kable(digits = 2)

5.3.15 ri with DeclareDesign (advanced)

Power for each value of b.

simulations |> group_by(b, step_1_draw) |> 
  summarize(p = mean(abs(estimate[estimator == "null"]) >= abs(estimate[estimator == "actual"]))) |>
  group_by(b) |> summarize(power = mean(p <= .05)) |>
  ggplot(aes(b, power)) + geom_line() + geom_hline(yintercept = .8, color = "red")

If you want to figure out more precisely what b gives 80% or 90% power you can narrow down the b range.

5.3.16 ri interactions

Lets now imagine a world with two treatments and we are interested in using ri for assessing the interaction. (Code from Coppock, ri2)

set.seed(1)

N <- 100
declaration <- randomizr::declare_ra(N = N, m = 50)

data <- fabricate(
  N = N,
  Z = conduct_ra(declaration),
  X = rnorm(N),
  Y = .9 * X + .2 * Z + .1 * X * Z + rnorm(N))

5.3.17 ri interactions

The approach is to declare a null model that is nested by the full model. Then \(F\) test statistic from the model comparisons is taken as the test statistic and distribution of this is built up under re-randomizations.

conduct_ri(
    model_1 = Y ~ Z + X,
    model_2 = Y ~ Z + X + Z * X,
    declaration = declaration,
    assignment = "Z",
    sharp_hypothesis = coef(lm(Y ~ Z, data = data))[2],
    data = data, 
    sims = 1000
  )  |> summary() |> kable()

5.3.18 ri interactions with DeclareDesign

Let’s imagine a true model with interactions. We take an estimate. We then ask how likely that estimate is from a null model with constant effects

Note: this is quite a sharp hypothesis

df <- fabricate(N = 1000, Z1 = rep(c(0,1), N/2), Z2 = sample(Z1), Y = Z1 + Z2 - .15*Z1*Z2 + rnorm(N))

my_estimate <- (lm(Y ~ Z1*Z2, data = df) |> coef())[4]

null_model <-  function(df) {
  M0 <- lm(Y ~ Z1 + Z2, data = df) 
  d1 <- coef(M0)[2]
  d2 <- coef(M0)[3]
  df |> mutate(
    Y_Z1_0_Z2_0 = Y - Z1*d1 - Z2*d2,
    Y_Z1_1_Z2_0 = Y + (1-Z1)*d1 - Z2*d2,
    Y_Z1_0_Z2_1 = Y - Z1*d1 + (1-Z2)*d2,
    Y_Z1_1_Z2_1 = Y + (1-Z1)*d1 + (1-Z2)*d2)
  }

5.3.19 ri interactions with DeclareDesign

Let’s imagine a true model with interactions. We take an estimate. We then ask how likely that estimate is from a null model with constant effects

Imputed potential outcomes look like this:

df <- df |> null_model()

df |>  head() |> kable(digits = 2, align = "c")

5.3.20 ri interactions with DeclareDesign

design <- 
  declare_model(data = df) +
  declare_measurement(Z1 = sample(Z1), Z2 = sample(Z2),
                      Y = reveal_outcomes(Y ~ Z1 + Z2)) +
  declare_estimator(Y ~ Z1*Z2, term = "Z1:Z2")

diagnose_design(design, sims = 1000, diagnosands = ri_ps(my_estimate))

5.3.24 ri Applications

5.3.25 ri Applications

If you think hard about assignment you might come up with an allocation like this.

5.3.26 ri Applications

That allocation balances as much as possible. Given the allocation you might randomly assign individuals to different days as well as randomly assigning them to treatments within days. If you then figure out assignment propensities, this is what you would get:

5.3.27 ri Applications

Even under the assumption that the day of measurement does not matter, these assignment probabilities have big implications for analysis.

• Only the type \(A\) subjects could have received any of the three treatments.

• There are no two treatments for which it is possible to compare outcomes for subpopulations \(B\) and \(C\)

• A comparison of \(T1\) versus \(T2\) can only be made for population \(A \cup B\)

• However subpopulation \(A\) is assigned to \(A\) (versus \(B\)) with probability 4/5; while population \(B\) is assigned with probability 3/8

5.3.28 ri Applications

• Implications for design: need to uncluster treatment delivery

• Implications for analysis: need to take account of propensities

Idea: Wacky assignments happen but if you know the propensities you can do the analysis.

5.3.29 ri Applications: Indirect assignments

A particularly interesting application is where a random assignment combines with existing features to determine an assignment to an “indirect” treatment.

• For instance: \(n\) of \(N\) are assigned to a treatment.
• You are interested in whether “having a friend assigned to treatment” makes a difference to a subject. Or maybe “a friend of a friend”
• That means the subject has a complex clustered assignment that depends on how many friends they have
• A bit mind-boggling, but:
  • Rerun your assignment many times and each time figure out whether a subject is assigned to an indirect treatment or not
  • Calculate the implied quantity of interest for each assignment
  • Assess the place of the actual quantity in the sampling distribution

5.4.1 Design-based analysis

Report the analysis that is implied by the design.

This is instantly recognizable from the design and returns all the benefits of the factorial design including all main effects, conditional causal effects, interactions and summary outcomes. It is much clearer and more informative than a regression table.

6.1.1 Experiments

• Experiments are investigations in which an intervention, in all its essential elements, is under the control of the investigator. (Cox & Reid)

• Two major types of control:

1. control over assignment to treatment – this is at the heart of many field experiments
2. control over the treatment itself – this is at the heart of many lab experiments

• Main focus today is on 1 and on the question: how does control over assignment to treatment allow you to make reasonable statements about causal effects?

6.1.3 Basic randomization

• Basic randomization is very simple. For example, say you want to assign 5 of 10 units to treatment. Here is simple code:

sample(0:1, 10, replace = TRUE)

 [1] 0 1 0 1 1 1 1 1 1 0

6.1.5 Basic randomization

Even better is to set it up so that it can reproduce lots of possible draws so that you can check the propensities for each unit.

set.seed(20111112)
P <- replicate(1000, sample(0:1, 10, replace = TRUE)) 
apply(P, 1, mean)

 [1] 0.519 0.496 0.510 0.491 0.524 0.514 0.535 0.497 0.470 0.506

Here the \(P\) matrix gives 1000 possible ways of allocating 5 of 10 units to treatment. We can then confirm that the average propensity is 0.5.

• A huge advantage of this approach is that if you make a mess of the random assignment; you can still generate the P matrix and use that for all analyses!

6.2.2 Cluster Randomization

• Often used if intervention has to function at the cluster level or if outcome defined at the cluster level.

• Disadvantage: loss of statistical power

• However: perfectly possible to assign some treatments at cluster level and then other treatments at the individual level

• Principle: (unless you are worried about spillovers) generally make clusters as small as possible

• Principle: Surprisingly, variability in cluster size makes analysis harder. Try to control assignment so that cluster sizes are similar in treatment and in control.

• Be clear about whether you believe effects are operating at the cluster level or at the individual level. This matters for power calculations.

• Be clear about whether spillover effects operate only within clusters or also across them. If within only you might be able to interpret treatment as the effect of being in a treated cluster…

6.2.3 Cluster Randomization: Block by cluster size

Surprisingly, if clusters are of different sizes the difference in means estimator is not unbiased, even if all units are assigned to treatment with the same probability.

Here’s the intuition.Say there are two clusters each with homogeneous treatment effects:

Then: What is the true average treatment effect? What do you expect to estimate from cluster random assignment?

The solution is to block by cluster size. For more see: http://gking.harvard.edu/files/cluster.pdf

6.3.1 Blocking

There are more or less efficient ways to randomize.

• Randomization helps ensure good balance on all covariates (observed and unobserved) in expectation.
• But balance may not be so great in realization
• Blocking can help ensure balance ex post on observables

6.3.2 Blocking

Consider a case with four units and two strata. There are 6 possible assignments of 2 units to treatment:

Even with a constant treatment effect and everything uniform within blocks, there is variance in the estimation of \(\widehat{\tau}\). This can be eliminated by excluding R1 and R6.

6.3.5 Blocking

• Blocking is a case of restricted randomization. Although each unit is sampled with equal probability, the profiles of possible assignments are not.
• You have to take account of this when doing analysis
• There are many other approaches.
  • Matched Pairs are a particularly fine approach to blocking
  • You could also randomize and then replace the randomization if you do not like the balance. This sounds tricky (and it is) but it is OK as long as you understand the true lottery process you are employing and incorporate that into analysis
  • It is even possible to block on covariates for which you don’t have data ex ante, by using methods in which you allocate treatment over time as a function of features of your sample (also tricky)

6.3.6 Other types of restricted randomization

• Really you can set whatever criterion you want for your set of treated units to have (eg no treated unit beside another treated unit; at least 5 from the north, 10 from the south, guaranteed balance by some continuous variable etc)
• You just have to be sure that you understand the random process that was used and that you can use it in the analysis stage
• But here be dragons
  • The more complex your design, the more complex your analysis.
  • General injunction http://www.ncbi.nlm.nih.gov/pubmed/15580598 ``as ye randomize so shall ye analyze’’)
  • In general you should make sure that a given randomization procedure coupled with a given estimation procedure will produce an unbiased estimate. DeclareDesign can help with this.

6.4.2 Factorial Designs

• Often when you set up an experiment you want to look at more than one treatment.
• Should you do this or not? How should you use your power?

Load up:

Spread out:

6.4.3 Factorial Designs

• Often when you set up an experiment you want to look at more than one treatment.
• Should you do this or not? How should you use your power?

Three arm it?:

Bunch it?:

6.4.4 Factorial Designs

• Surprisingly, adding multiple treatments does not generally eat into your power (unless you are decomposing a complex treatment – then it can. Why?)
• Especially when you use a fully crossed design like the middle one above.
• Fisher: “No aphorism is more frequently repeated in connection with field trials, than that we must ask Nature few questions, or, ideally, one question, at a time. The writer is convinced that this view is wholly mistaken.”
• However – adding multiple treatments does alter the interpretation of your average treatment effects. If T2 is an unusual treatment for example, then half the T1 effect is measured for unusual situations.

This speaks to “spreading out.” Note: the “bunching” example may not pay off and has undesireable feature of introducing a correlation between treatment assignments.

6.4.5 Factorial Designs

Two ways to do favtial assignments in DeclareDesign:

# Block the second assignment
declare_assignment(Z1 = complete_ra(N)) +
declare_assignment(Z2 = block_ra(blocks = Z1)) +
  
# Recode four arms  
declare_assignment(Z = complete_ra(N, num_arms = 4)) +
declare_measurement(Z1 = (Z == "T2" | Z == "T4"),
                      Z2 = (Z == "T3" | Z == "T4"))

6.4.6 Factorial Designs: In practice

• In practice if you have a lot of treatments it can be hard to do full factorial designs – there may be too many combinations.

• In such cases people use fractional factorial designs, like the one below (5 treatments but only 8 units!)

6.4.7 Factorial Designs: In practice

• Then randomly assign units to rows. Note columns might also be blocking covariates.

• In R, look at library(survey)

6.4.8 Factorial Designs: In practice

• But be careful: you have to be comfortable with possibly not having any simple counterfactual unit for any unit (invoke sparsity-of-effects principle).

• In R, look at library(survey)

6.4.9 Controversy?

Muralidharan, Romero, and Wüthrich (2023) write:

Factorial designs are widely used to study multiple treatments in one experiment. While t-tests using a fully-saturated “long” model provide valid inferences, “short” model t-tests (that ignore interactions) yield higher power if interactions are zero, but incorrect inferences otherwise. Of 27 factorial experiments published in top-5 journals (2007–2017), 19 use the short model. After including interactions, over half of their results lose significance. […]

6.5.1 Principle: Address external validity at the design stage

Anything to be done on randomization to address external validity concerns?

• Note 1: There is little or nothing about field experiments that makes the external validity problem greater for these than for other forms of ‘’sample based’’ research
• Note 2: Studies that use up the available universe (cross national studies) actually have a distinct external validity problem
• Two ways to think about external validity issues:
  1. Are things likely to operate in other units like they operate in these units? (even with the same intervention)
  2. Are the processes in operation in this treatment likely to operate in other treatments? (even in this population)

6.5.2 Principle: Address external validity at the design stage

• Two ways to think about external validity issues:
  1. Are things likely to operate in other units like they operate in these units? (even with the same intervention) 2.Are the processes in operation in this treatment likely to operate in other treatments? (even in this population)
• Two approaches for 1.
  • Try to sample cases and estimate population average treatment effects
  • Exploit internal variation: block on features that make the case unusal and assess importance of these (eg is unit poor? assess how effects differ in poor and wealthy components)
• 2 is harder and requires a sharp identification of context free primitives, if there are such things.

6.6.2 Sample data

Here are the first couple of rows and columns of the resulting data frame.

my_data <- draw_data(design)
kable(head(my_data), digits = 2)

6.6.3 Sample data

Here is the distribution between treatment and control:

kable(t(as.matrix(table(my_data$Z))), 
      col.names = c("control", "treatment"))

6.6.4 Complete Random Assignment using the built in function

assignment_complete <-   declare_assignment(Z = complete_ra(N))  

design_complete <- 
  replace_step(design, "assignment", assignment_complete)

6.6.5 Data from complete assignment

We can draw a new set of data and look at the number of subjects in the treatment and control groups.

set.seed(1:5)
data_complete <- draw_data(design_complete)

kable(t(as.matrix(table(data_complete$Z))))

6.6.6 Plotted