Causal vs. Descriptive Estimands

Are Preferences Properties or Effects?

Figure 1: Preferences and features combine to determine choices. Manipulating features lets us infer preferences.

Are Preferences Properties or Effects?

But still a kind of formal equivalence:

Figure 2: Preferences and features combine to determine choices. Manipulating features lets us infer preferences.

Clarifies that \(\theta\) is a property in the Pearlean world. Not an event or difference in potential timestamped outcomes as in Rubin.

Are Preferences Properties or Effects?

Figure 3: Preferences and features combine to determine choices. Manipulating features lets us infer preferences.

Three arguments for treating preferences as properties (not causal effects):

1. No real choices: Outcomes are stated intentions, not actual decisions
2. Constitutive features: Changing an attribute creates a different unit — not a modified one
3. Transportability: Properties allow broader inference across contexts

Experiment Type 0: The Stupidest Benchmark

Design

You are interested in the average age of people in a population

You divide the subjects into two groups:

• Group A you ask: in what year were you born
• Group B you ask: what year is it now?

Estimation

You estimate: \(\widehat {\overline {Age}} = \mathbb E_{i \in A}[Y_i(A)] - \mathbb E_{i \in B}[Y_i(B)]\)

• You have a causal estimand and have done causal inference. This is valid, it is unbiased thanks to the randomization.

• You use Neyman standard errors which are valid from randomization and needed because you are have an incomplete schedule of potential outcomes.

Congratulations.

Experiment Type 1: The Priming Experiment

Experiment Type 2: The Framing Experiment

Design: Vary question wording to assess whether form affects substance. The idea of an equivalence frame is that the same question is asked in different ways.

Experiment Type 3: The Conjoint

“A factorial survey experiment designed to measure multidimensional preferences”

— De la Cuesta, Egami, and Imai (2022)

Conjoints have two genuinely distinct use cases — rarely recognized:

Experiment Type 3: Decriptive Conjoints

Descriptive case 1: You ask would you react under each of these imagined situations?

• Ask about lots!
• Goal: Get people’s own read on their potential outcomes.
• Summarize beliefs using AMCE.

Descriptive case 2: You want to learn about an algorithm: what rule does a bank use when deciding to give credit? What rule does an AI model use to assess the validity of a statement.

• Send in multiple queries.
• Summarize rules using AMCE.

In both cases you might consider alternatives:

• Ask the bank what the rule is
• Inspect the algorithm directly
• Perhaps: ask many questions
• Use summaries other than AMCE

Experiment Type 3: Causal Conjoints

Classic candidate experiment: in a world where a given set of facts about attributes is available (and only these), how do choices depend on the elements of the sets.

• Treatment is information, not attributes
• Willing to believe subjects act ‘as if’ they are making a choice

Hainmueller et al external validity study two edged sword supporting this view: it might work! but the actual target applications are quite unusual

Go in peace

The Promise of Controls

Conjoint experiments are celebrated for the ability to control many features simultaneously.

Researchers describe this as addressing confounding:

Controls Change Which Estimand You Target

Controls Can Increase Uncertainty

A counterintuitive result?:

Adding controls at the intervention stage can increase variance in conjoint experiments.

Why? Controlling \(A_2\) at the intervention stage introduces variation that must then be removed in the analysis to recover the baseline effect. The net effect on variance can be positive.

Preferences “over an Attribute” Reflect Valuation and Informational Content

Focus not on distribution of levels but presence.

Setup: \(Y = A_2 \times A_3\). So \(A_1\) no causal role in quality. Yet whether \(A_1\) “matters” for preferences depends entirely on what else is known.

Joint distribution of \((A_1, A_2, A_3)\):

Key feature: \(A_1=1\) is a strong signal that \(A_2=1\) (and thus informative about \(Y\), via \(A_2\)).

Mental Model 1: Each Attribute Affects Quality Directly

Privilege, wealth, and ability each contribute directly to quality.

Figure 4: Left: respondent’s mental model. Right: what the experimenter can observe when only signals for privilege and wealth are provided — ability is unobserved.

In this world, the experiment cleanly recovers the mental model: signals map directly to beliefs about the attributes they describe.

Mental Model 2: Ability Matters, Wealth is a Collider

Wealth is produced by privilege and ability — so signalling privilege and wealth is informative about ability.

Figure 5: Left: respondent’s mental model — privilege and ability jointly produce wealth; ability drives quality. Right: what signals about privilege and wealth reveal about beliefs.

Adding the wealth control creates an apparent effect of privilege via collider bias: conditioning on wealth renders privilege informative about quality.

The Mental Models Problem: Key Implication

In Mental Model 2 (right column): subjects believe only ability drives quality, and that wealth is produced by privilege and ability. Privilege has no direct effect on quality.

But when both privilege and wealth are controlled, the implied “preferences” are identical in both worlds:

The Real Structure

In a conjoint, there is a causal hierarchy from features to outcomes:

\[\underbrace{A_1, A_2}_{\text{Attributes}} \;\rightarrow\; \underbrace{I_1, I_2}_{\text{Information provided}} \;\rightarrow\; \underbrace{B_1, B_2}_{\text{Beliefs formed}} \;\rightarrow\; \underbrace{Y}_{\text{Evaluation}}\]

Figure 6

Seven Estimands, One Design

The conjoint identifies information effects (rows 2, 4) only.

The Gap Between Claims and Evidence

Survey experiments are routinely used to make claims like these:

Maps to Nowhere

Before asking how to export results, ask: is the target estimand even well-defined?

Conjoints estimate effects of signals cleanly. The corresponding real-world estimands — effects of gender, regime type, corruption — may not be.

Three threats to estimand existence:

Threat 1: Attributes as Causes

Holland’s challenge: “attributes of units are never causes”

If you change the attribute, you may no longer be talking about the same unit.

Threat 2: SUTVA Violations — Hidden Treatment Versions

The Stable Unit Treatment Value Assumption requires no hidden versions of treatment.

For states rather than interventions, this is almost always violated.

Threat 3: Exclusion Restriction

If every lever we can imagine to change an attribute inevitably induces other effects on outcomes, the clean causal estimand does not exist.

What Would It Take? The Six Conditions

For AMCE = ATE (survey estimand = real-world causal effect):

The DAG: Promising vs. Threatened Cases

Figure 7: Left: a structure under which AMCE can recover the effect of attributes on votes. Right: three numbered threats to this inference. Arrow 1 = causal relation between attributes (violates A1). Arrow 2 = cross-attribute updating (violates A5). Arrow 3 = direct behavioral effect (violates A5).

Three Risks: Summary

What Survey Experiments Can Do Well

There is genuine power here, used right: