Class: https://macartan.github.io/ci/
For each puzzle: explore the issues raised by the puzzle and generate a self contained presentation in .qmd (or .Rmd) that reports on your investigations. Present in the next class session.
DeclareDesignSometimes people worry that with larger samples you are more likely to get a false positive. Is that true?
Assess by generating a simple experimental design from scratch in which we can vary the N and in which there is no true effect of some treatment.
Then:
simulations_df. What shape is it and why?diagnosands_dfHint: the slides contain code for a simple experimental design
Clustering
Now:
Hint For generating hierarchical models use add_level. Also: be sure to have a reasonable large top level shock in order to see differences arising from clustering at the school level. You could also try heterogeneous effects by school.
g <-
declare_model(
L1 = add_level(N = 10, u = rnorm(N)),
L2 = add_level(N = 12, v = rnorm(N)))
g() |> slice(1:3, 13:15) |> kable()| L1 | u | L2 | v |
|---|---|---|---|
| 01 | 0.5917647 | 001 | 0.1260403 |
| 01 | 0.5917647 | 002 | 0.1244666 |
| 01 | 0.5917647 | 003 | -1.4247324 |
| 02 | 0.0470489 | 013 | 1.3728940 |
| 02 | 0.0470489 | 014 | -0.1026590 |
| 02 | 0.0470489 | 015 | -0.9639386 |
Learning about standard errors
The standard error is standard deviation of the sampling distribution of an estimate.
That sounds complicated, but actually the sampling distribution of an estimate lives in the simulations data frame so you can look at its standard deviation and assess whether standard errors estimate it well.
Challenge: Generate a simple experimental design in which there is a correlation (rho) between the two potential outcomes (Y_Z_0 and Y_Z_1).
Show the distribution of the estimates over different values of rho
Assess the performance of the estimates of the standard errors and the coverage as rho goes from -1 to 0 to 1. Describe how coverage changes. (Be sure to be clear on what coverage is!)
Y_Z_0 <- rnorm(1000)
Y_Z_1 <- correlate(rnorm, given = Y_Z_0, rho = .5)
cor(Y_Z_0, Y_Z_1)[1] 0.5254649Declare a design in which:
Discuss results. Do any of these return the right answer?
Hint: You can add three separate declare_estimator steps. They should have distinct labels. The trickiest part is to figure out how to extract the estimate in (c) because you will have both a main term and an interaction term for \(X\).
These do not require coding
Potential outcomes
Consider an outcome \(Y\) that can be affected by two variables \(X_1\) and \(X_2\). All variables are binary. Can you fill in the potential outcomes (rows) for each of the column types?
| X1 is a necessary and sufficient condition for Y | X1 is necessary but not sufficient | X1 is sufficient but not necessary | X1 sometimes causes Y but is neither necessary nor sufficient | |
|---|---|---|---|---|
| Y(0,0) = | ||||
| Y(1,0) = | ||||
| Y(0,1) = | ||||
| Y(1,1) = |
Consider an outcome Y that can be affected by two variables X1 and X2 but say that X2 can itself be affected by X1. Write down possible potential outcomes for Y1 and X2 when: X1 causes X2 and Y, but X1 does not cause Y through X2
| Potential outcome | Value (0/1) |
|---|---|
| Y(0,0) = | |
| Y(1,0) = | |
| Y(0,1) = | |
| Y(1,1) = | |
| X2(0) = | |
| X2(1) = |
Then what is the average effect of X1 on Y? What is the average effect of X2 on Y? Which cause has the biggest effect?
Compare these and discuss.
These next questions use some concepts we have not introduced yet. Don’t worry if your answers are incomplete but do share your thought processes around these.
Hint: The direction of collider bias is related to the ways that \(K\) and \(Y\) interact to produce \(X\). Also: by “conditioning on \(K=1\)” we mean: using only cases for which \(K=1\).
dagitty and check your answer| A | B | C | p |
|---|---|---|---|
| 0 | 0 | 0 | 0.32 |
| 0 | 0 | 1 | 0.08 |
| 0 | 1 | 0 | 0.08 |
| 0 | 1 | 1 | 0.02 |
| 1 | 0 | 0 | 0.08 |
| 1 | 0 | 1 | 0.12 |
| 1 | 1 | 0 | 0.12 |
| 1 | 1 | 1 | 0.18 |
Set up a model in DeclareDesign that has this distribution. Draw a large dataset from it and check if relations of conditional independence implied by your DAG.
Hint:This is relatively tricky. From slides you will see a DAG is a directed acyclic graph. A DAG should represent relations of conditional independence in the sense that any node \(A\) that is separated from another node \(B\) given nodes \(W\) should be conditionally independent given \(W\). You should be able to read from this table which nodes are conditionally independent from each other given other nodes. You should be able to check consistency between the probability distribution and an underlying model by calculating quantities such as \(\Pr(x, m,y) = \Pr(x)\times\Pr(m|x)\times\Pr(y|m,x)\).
Hint: This question is asking about the front door criterion. Check whether the conditions apply for the front door criterion to hold. Note that an effect is not identified if the data pattern it produces is also consistent with a different effect. Is that the case here? The second part of this is more important than the first part. Note you can generate and update models of this form with CausalQueries.
| Block | Z | Y |
|---|---|---|
| 1 | 0 | 0 |
| 1 | 0 | 0 |
| 1 | 1 | 1 |
| 2 | 0 | 0 |
| 2 | 0 | 0 |
| 2 | 0 | 1 |
| 2 | 1 | 0 |
| 2 | 1 | 1 |
| 2 | 1 | 1 |
DeclareDesign to compare the answers you get if you use exactly this data and calculate (1) OLS and controlled for block (2) IPW and (3) the Lin estimatorHint: If you use exactly this data and replicate it there is no stochastic component; you can run with sims = 1; to get the precision of your estimates you can declare the diagnosand mean_se = mean(std.error)
Declare a factorial design with two binary treatments and an interaction between them. For example \(Y = .1*X1 + .3*X2 - .2*X1*X2\)
Have a non-stochastic model declaration so that the only source of randomness is in the assignment procedure.
Calculate a \(p\) values for the null hypothesis of no interaction between treatments using randomization inference.
Hint: You can use the ri2 package. Hint you can keep the top level fixed by declaring shocks outside the design (U <- rnorm(N)) or by using a simulation vector in diagnosis (sims = c(1,1,500, 1, ...))
Replicate Table 1 in Lin (2012); ignore first rows which are theoretical results; include standard errors of estimates (these are not included in Table 1 but are produced by diagnose_design); ‘Tyranny of the minority’ estimator is optional.
Say you want to include a control variable. But you have missingness in the control. Should you proceed and what can you do about it?
Declare a design for an experiment in which a binary covariate X is related to potential outcomes, according to b, and so to treatment effects. Say X is missing with probability p.
Compare answer strategies in which you:
Assess performance (RMSE) over a range of values for p and b. How do you think the comparison of strategies depends upon N?
Bonus: Say instead of a single mediator \(M\) you had a chain: \(M_1, M_2, \dots\). Does lengthening the chain sufficiently allow you to identify causal effects?
Consider the Napkin model: W->Z->X->Y; W <-> X; W <-> Y
make_model("W->Z->X->Y; W <-> X; W <-> Y") |> plot(x_coord = 1:4, y_coord = c(1,1,1,1))
Consider some true parameter vector and generate data from this vector, varying the amount of data from 10 to 100 to 1000 observations. Assume in particular that there is confounding: e.g. that the probability \(X=1, Y=1\) depends on \(W\).
In each case calculate the posterior distribution on the average effect of \(X\) on \(Y\). Assess whether the quantity appears to be identified.
Can you use a formula to calculate an effect directly?
Hint: You can generate a “target” model, generate data from that, and calculate from that a target query.
target_model <- make_model("W->Z->X->Y; W <-> X; W <-> Y") |>
set_parameters(param_name = "Y.11_W.1", parameters = .9) |>
set_parameters(param_name = "X.11_W.1", parameters = .9)Note: this is hard because the W <-> Y confounding implies an X <-> Y confounding. There is no scope for front door adjustment. If you control for \(W\) you open a path from \(X\) to \(Y\) (since \(W\) is a collider) and, more subtly, conditioning on \(Z\) also partly opens a collider path. See this discussion: https://twitter.com/analisereal/status/1273099716956430340
Bayes by hand
Say that we have 50 observations of \(Y_0\) and 50 observations of \(Y_1\) from a random experiment. Assume these are each drawn independently from a normal distribution centered on \(\mu_j\) with sd \(\sigma_j\), \(j\in\{0,1\}\).
Write down a likelihood function that returns the probability of seeing the 100 observations that you see given the four parameters: \(\mu_0, \mu_1, \sigma_0, \sigma_1\).
Use grid <- expand_grid(m0 = ..., m1 = ...) to generate a grid of possible values for the four parameters.
Apply your likelihood function to all the possible parameter values contained in your grid.
Now:
Generate a simple multilevel experimental design (e.g 20 children each in 20 schools). Assume that the treatment effect in each schools is drawn from a normal distribution with a given variance \(\sigma\).
Use design diagnosis to assess the ability of a Bayesian multilevel hierarchical to recover \(\sigma\).
Bonus: Are estimates of treatment effects more or less reliable than what you would get from a frequentist approach that interacts school IDs with treatment?
Randomization
100 students sign up to take part in an experiment. You want to measure the effect of immigration on social trust. Half your subjects are men and half are women and you believe gender is very predictive of social trust.
Your experiment involves varying whether a “native” or an “immigrant” facilitator instructs players in how to play a trust game. You have five native and five immigrant facilitators and you want them each to conduct one session with 10 subjects.
You are free to assign both subjects and facilitators to sessions. Propose an appropriate randomization strategy. Is it blocked? Clustered? Both?
Say now that subjects have already signed up for sessions. You can only assign facilitators to sessions, but you have access to the subject lists before you do so. Describe your optimal randomization strategy. Is it blocked? Clustered? Both?
Heterogeneous propensities
Two of four units are going to be assigned to treatment. A researcher sets up a design in which subjects can decide for themselves the probability with which they receive a treatment. Requested propensities are as below.
Can you :
| Requested propensity | Y0 | Y1 |
|---|---|---|
| 0.2 | 0 | 1 |
| 0.4 | 0 | 2 |
| 0.6 | 1 | 3 |
| 0.8 | 1 | 4 |
You have access to a network of all friendship links in a classroom. This is in the form of an \(n\times n\) adjacency matrix where a 0/1 means the row individual is / is not a friend of the column individual.
You want to provide political information to a set of students and see how much more likely it is that a student that you do not give information to receives the information if a friend is treated compared to the situation in which a friend of a friend is treated. So you want to be sure that some subjects have friends treated and some have only friends of friends treated. How would you assign treatment? How can you work out your treatment assignment probabilities?
Bonus: Generate data and diagnose the estimation strategy available using the interference package
You have 10 units that you want to assign to treatment and control. On each unit you have two covariates, each with a lopsided distribution (for example, log normally distributed) and each strongly associated with the outcome of interest.
You are worried that you will have a good chance of significant imbalance between covariates and are thinking about using a procedure in which you re-randomize in the event in which you have some poor balance (for this you need to define what you mean by poor balance, e.g. you might want that balance is in the lower quartile of possible imbalances).
Compare the following strategies in terms of (a) bias (b) RMSE and (c) coverage:
In case 3 you can make use of your knowledge of the randomization to generate assignment propensities and implement randomization inference.
See Morgan and Rubin
Covariates 1
Compare the power gains from two strategies:
Equation (4.6) in Gerber and Green (2012) suggests that if the sum of two slopes exceeds 1 then there are gains in efficiency from adding a covariate. Show that this is true in practice using a design declaration.
Compare the performance of the Lin estimator an the doubly robust estimator discussed in slides
Covariates 4
Sometimes researchers running an experiment look for imbalance on a covariate and then include the covariate as a control if and only if they see imbalance. Set up a design in which a covariate may or may not affect potential outcomes and assess the performance given different rules
Spillovers
Imagine a study with three subjects. Each subject’s potential outcomes are as follows:
So for example if one unit is in treatment that unit has outcome Y=1, and the others have Y = 0; if all 3 are in treatment they all have Y=3.
Baron Kenny have provided a popular method to implement mediation analysis.
Declare a design with a mediation process and possible violations of sequential ignorability, governed by some parameters k.
Demonstrate under what conditions estimates using the Baron-Kenny procedure are misleading.
DD 16.3 shows poor behavior of a multi period difference in differences design when there is effect heterogeneity.
Imai and Kim (2021) highlights the risk of negative weights in this setting. Can you recover the implied weights and identify which units contribute negatively to estimates?
LATE
Which of these problems could be addressed using instrumental variables? In each case, what kinds of concern might you have about the IV strategy?
No exercises
Say you sampled subject A with probability .6 and subject B with probability .4. Say you assigned each to treatment with probability .6. What weight should you put on A in your analysis if they end up in treatment? What if they end up in control? How about B?
Generate a simple experimental design and estimate treatment effects using a (a) Bayesian regression with rstanarm. Provide informative priors (b) lm_robust
Let the size of the data increase from 10 to 1000 and: plot the estimates from the two approaches as \(N\) increases, plot the average standard error and the posterior variance from the two approaches as \(N\) increases.