Kudos

• Sharp focus on balance/control in complex vignettes
• Lots of thoughts about processes, complexities, risks
• Nice illustrations and demonstration of effectiveness
• Some very clever diagnoistics

Very hard to argue against this: sensible and scaleable

Causal ordering and entanglements

The thing that I have been wondering most about:

• Does evidence of successful targeting have to be theory based not empirical? Or:
• What implication for current diagnostics?
• What implication for alternative strategies?

Writing

• LLMs do not seem central here?
  • Rather gain of algorithm is that these could be done with LLMs or not
  • Describe core strategy in the abstract and demonstrate with LLMs
• Also useful perhaps if there are different audiences for the formal features and the practical guidelines

So?

1. Theory paper
2. Guidance paper: general strategy plus LLM implementation (package?)

Kudos

• Very thoughtful piece

• Nice conceptualizations

• Multiple implementations

• Constructive approach for a hard problem

• Multiple encouragements to capture both a type of specificity and sensitivity

  • \(\Pr(\text{not activated} | \text{not targeted by } Z)\)
  • \(\Pr(\text{activated} | \text{targeted by } Z)\)

Sensitivity and specificity: Current

Sensitivity and specificity: Possible?

For pairs anyhow…

Vignettes beat Images? Where would the indifference curves be here?

Which of these is going on?

Or truly entangled?

Should we just focus on (real world) Zs? Is this an instruments problem or mediation problem?

Metrics

In what sense are anger and fear on the same metric?

Here we focus on changes normalized by variance But what is to say that is right?

Kudos

the framework constructs theory-aligned alternatives that preserve the persistence and interdependence of political data while excluding the mechanism of interest

Direct shot at model dependency

Assess what you have to believe about the world to draw the inferences you do; in which worlds would you get lost

• love theory focus

• really nicely written

• really nice interpretation and modest handling of different findings

Scope for synthesis?

• Each null world is taken one at a time.

• Can we combine?

• What’s the probability of seeing data like this given no true effect and the following distribution over worlds?

• Or can we nest? When should we nest? When can we not?

• at least within “tractable set”

Ex ante or ex post

• Also connects to one idea in DeclareDesign

Principle 3.2: a good design should work … even when the world is different from what we expect

Null models: An important example is the performance of a research design under a “null model,” where the true effect size is zero.

The DeclareDesign approach would do this ex ante, independent of realized data; you do ex post. What are the merits of each?

Fishing risks

• Implications for replications
• Scope for critics to fish null worlds

Writing

• There is a general deep idea here and there is a specific application

• The application is a little complicated (and you need to bring readers up to speed on the analysis strategy)

• Can you either separate or else build out with abstract examples before this application

Kudos

• Great question

• Love the focus on theory and thinking structurally about this

• Nice examples of when things are easy and when they are hard.

Insight

Key insight:

determining whether \(\tau_{CCA}\) is biased for \(\tau\) reduces to assessing whether including \(M_i\) as a control in the oracle regression biases the treatment estimation for \(\tau\) , i.e. whether \(M_i\) “over-controls”.

Insight

Let \(M\) denote mssingness. Key insight:

Controlling for \(M\) is overcontrolling. It does not open a backdoor path, but it does violate the backdoor criterion, being post-treatment.

• What’s missing? \(X\) or \(Y\)? If \(X\) it is MAR. If \(Y\) it is MNAR. But the data look the same.

Lin, Liu: Illustration

df <- fabricate(N = 10000,
                 D = complete_ra(N),
                 Y = simple_ra(N, prob_unit =.25 + .5 * D),
                 M = simple_ra(N, prob_unit = .8*Y))

Oracle:

Conditioned:

Lin, Liu: Omnibus missingness node

Say you do not control for \(U\)

Moderators and missingness

Say:

• \(W\) moderates the effect of \(X\) on \(Y\). Say \(M=1\) when \(W = 1\), then we only observe the effect of \(X\) on \(Y\) when \(W = 0\).

• But controlling for M is not “overcontrolling”?

• The coefficient changes but it would not with a demeaned treatment. This is a support question: estimand changes.

Moderation

df <- fabricate(N = 10000, D = complete_ra(N),
                W = complete_ra(N), Y = D*W, M = W==1)

Ideal:

CATE (\(M = W =0\)), correctly recovered from missing data analysis: