1.1.1 The problem of unidentified mediators

• Consider a causal system like the below.
• The effect of X on M1 and M2 can be measured in the usual way.
• But unfortunately, if there are multiple mediators, the effect of M1 (or M2) on Y is not identified.
• The ‘exclusion restriction’ is obviously violated when there are multiple mediators (unless you can account for them all).

1.1.13 The problem of unidentified mediators: Solutions?

Weaker assumptions justify parallel design

• Group A: \(T\) is randomly assigned, \(M\) left free.
• Group B: divided into four groups \(T\times M\) (requires two more assumptions (1) that the manipulation of the mediator only affects outcomes through the mediator (2) no interaction, for each unit, \(Y(1,m)-Y(0,m) = Y(1,m')-Y(0,m')\).)

Takeaway: Understanding mechanisms is harder than you think. Figure out what assumptions fly.

1.1.14 In CausalQueries

Lets imagine that sequential ignorability does not hold. What are our posteriors on mediation quantities when in fact all effects are mediated, effects are strong, and we have lots of data?

model <- make_model("X -> M ->Y <- X; M <-> Y")

plot(model)

1.1.15 In CausalQueries

We imagine a true model and consider estimands:

truth <- make_model("X -> M ->Y") |> 
  set_parameters(c(.5, .5, .1, 0, .8, .1, .1, 0, .8, .1))

queries  <- 
  list(
      indirect = "Y[X = 1, M = M[X=1]] - Y[X = 1, M = M[X=0]]",
      direct = "Y[X = 1, M = M[X=0]] - Y[X = 0, M = M[X=0]]"
      )

truth |> query_model(queries) |> kable()

1.2.1 SUTVA violations (Spillovers)

Spillovers can result in the estimation of weaker effects when effects are actually stronger.

The key problem is that \(Y(1)\) and \(Y(0)\) are not sufficient to describe potential outcomes

1.2.5 Spillovers: direct and indirect treatments

1.2.6 Spillovers: Simulated estimates

1.2.7 Spillovers: Opportunities and Warnings

You can in principle:

• debias estimates
• learn about interesting processes
• optimize design parameters

But to estimate effects you still need some SUTVA like assumption.

1.2.8 Spillovers: Opportunities and Warnings

In this example if one compared the outcome between treated units and all control units that are at least \(n\) positions away from a treated unit you will get the wrong answer unless \(n \geq 7\).

1.3.1 Transparent workflows

Experimental researchers are deeply engaged in the movement towards more transparency social science research.

• Analytic replication. This should be a no brainer. Set everything up so that replication is easy. Use quarto rmarkdown, or similar. Or produce your replication code as a package.

1.3.4 Two distinct rationales for registration

• File drawer bias (Publication bias)

• Analysis bias (Fishing)

1.3.5 File drawer bias

– Say in truth \(X\) affects \(Y\) in 50% of cases.

– Researchers conduct multiple excellent studies. But they only write up the 50% that produce “positive” results.

– Even if each individual study is indisputably correct, the account in the research record – that X affects Y in 100% of cases – will be wrong.

1.3.6 File drawer bias

– Say in truth \(X\) affects \(Y\) in 50% of cases.

– Researchers conduct multiple excellent studies. But they only write up the 50% that produce “positive” results.

– Even if each individual study is indisputably correct, the account in the research record – that X affects Y in 100% of cases – will be wrong.

1.3.7 File drawer bias

Exacerbated by:

– Publication bias – the positive results get published

– Citation bias – the positive results get read and cited

– Chatter bias – the positive results gets blogged, tweeted and TEDed.

1.3.8 Analysis bias (Fishing)

– Say in truth \(X\) affects \(Y\) in 50% of cases.

– But say that researchers enjoy discretion to select measures for \(X\) or \(Y\), or enjoy discretion to select statistical models after seeing \(X\) and \(Y\) in each case.

– Then, with enough discretion, 100% of analyses may report positive effects, even if all studies get published.

1.3.9 Analysis bias (Fishing)

– Say in truth \(X\) affects \(Y\) in 50% of cases.

– But say that researchers enjoy discretion to select measures for \(X\) or \(Y\), or enjoy discretion to select statistical models after seeing \(X\) and \(Y\) in each case.

– Then, with enough discretion, 100% of analyses may report positive effects, even if all studies get published.

1.3.10 Analysis bias (Fishing)

– Try the exact fishy test An Exact Fishy Test (https://macartan.shinyapps.io/fish/)

– What’s the problem with this test?

1.3.11 Evidence-Proofing: Illustration

• When your conclusions do not really depend on the data

• Eg – some evidence will always support your proposition – some interpretation of evidence will always support your proposition

• Knowing the mapping from data to inference in advance gives a handle on the false positive rate.

1.3.12 The scope for fishing

1.3.13 Evidence from political science

Source: Gerber and Malhotra

1.3.14 More evidence from TESS

• Malhotra tracked 221 TESS studies.
• 20% of the null studies were published. 65% not even written up (file drawer or anticipation of publication bias)
• 60% of studies with strong results were published.

Implications are:

• population of results not representative
• (subtler) individual published studies are also more likely to be overestimates

1.3.15 The problem

• Summary: we do not know when we can or cannot trust claims made by researchers.

• [Not a tradition specific claim]

1.3.16 Registration as a possible solution

Simple idea:

• It’s about communication:
• just say what you are planning on doing before you do it
• if you don’t have a plan, say that
• If you do things differently from what you were planning to do, say that

1.3.17 Worries and Myths

Lots of misunderstandings around registration

1.3.19 Structural challenge

Our journal review process is largely organized around advising researchers how to adjust analysis in light of findings in the data.

1.3.20 Myth: Fishing is technique specific

• Frequentists can do it

• Bayesians can do it too.

• Qualitative researchers can also do it.

• You can even do it with descriptive statistics

1.3.21 Myth: Fishing is estimand specific

• You can do it when estimating causal effects
• You can do it when studying mechanisms
• You can do it when estimating counts

1.3.23 Worry: Registration will create administrative burdens for researchers, reviewers, and journals

• Registration will produce some burden but does not require the creation of content that is not needed anyway

• It does shift preparation of analyses forward

• And it also can increase the burden of developing analyses plans even for projects that don’t work. But that is in part, the point.

• Upside is that ultimate analyses may be much easier.

1.3.24 Worry: Registration will force people to implement analyses that they know are wrong

• Most arguments for registration in social science advocate for non-binding registration, where deviations from designs are possible, though they should be described.
• Even if it does not prevent them, a merit of registration is that it makes deviations visible.

1.3.25 Myth: Replication (or other transparency practices) obviates the need for registration

• There are lots of good things to do, including replication.
• Many of these do not substitute for each other. (How to interpret a fished replication of a fished analysis?)
• And they may likely act as complements
• Registration can clarify details of design and analysis and ensure early preparation of material. Indeed material needed for replication may be available even before data collection

1.3.26 Worry: Registration will put researchers at risk of scooping

• But existing registries allow people to protect registered designs for some period
• Registration may let researchers lay claim to a design

1.3.27 Worry: Registration will kill creativity

• This is an empirical question. However, under a nonmandatory system researchers could:
• Register a plan for structured exploratory analysis
• Decide that exploration is at a sufficiently early stage that no substantive registration is possible and proceed without registration.

1.3.28 Implications:

• In neither case would the creation of a registration facility prevent exploration.

• What it might do is make it less credible for someone to claim that they have tested a proposition when in fact the proposition was developed using the data used to test it.

• Registration communicates when researchers are angage in exploration or not. We love exploration and should be proud of it.

1.3.29 Punchline

• Do it!
• But if you have reasons to deviate, deviate transparently
• Don’t implement bad analysis just because you pre-registered
• Instead: reconcile

1.3.30 Reconciliation

Incentives and strategies

1.3.31 Reconciliation