Design and Analysis of Experiments

Introduction to the class

Macartan Humphreys

1 Getting started

  • General aims and structure
  • Expectations

2 Aims and items

  • Primary focus is training on how to do experiments; not just knowledge of good experiments
  • Deep understanding of key ideas in causal inference
  • Transportable tools for understanding how to evaluate and improve design
  • Practice!

2.1 Syllabus and resources

2.2 The topics

Day Topic Activity
1: 1/9 Getting started Outline In-class exercise, intro to field experiment
2: 1/16 Doing experiments: Workflows; ethics Presentation of plans, ethics application
3: 1/30 Causality: Foundations, Inquiries Field reports
4: 2/6 Answer strategies: Estimation and Inference Discussion of findings 1; Proposals 2
5: 2/20 Data strategies and Design evaluation DeclareDesign, Assignments, Design evaluation Proposal refinements 2
6: 2/27 Topics: Survey experiments, Spillovers, Downstream experimentation Design declarations and ethics applications

2.3 Hands on!

We hope to run three experiments:

  1. Today, simulated experiments by hand
  2. Over the next four weeks (completed by 6 Feb): A field experimental variation of Choi, Poertner, and Sambanis (2023)
  3. Last weeks development of designs to implement in collective survey experiment (admitted based on external review and subject to ethics approval and implemented likely after class ends). Expect to have space for four projects (individual or collective).

3 Responsibilities

3.1 Required

  • Do the readings and take part in discussion (30%)
  • Take part in (field) experiment I (30%)
  • Design an experiment as a candidate for (survey) experiment II (40%)

3.2 Optional

  • Prepare a research design or short paper, perhaps building on existing work. Typically this contains:
    • a problem statement
    • a description of an experimental strategy to address the problem
    • empirical (ideally) or simulation based results (ok)
    • a discussion of implications
    • A passing paper will illustrate identify a causal effect of substantive interest credibly; a good paper will innovate on method or topic.

4 Support

  • Office hours: Friday afternoons after class: 2 - 4
  • Lennard: Office hours
  • Email

5 Let’s go 1

5.1 Task

We are going to do some experiments by hand. You choose assignment schemes and analysis plans.

  • You have been given an envelope with 20 cards. Each card has two numbers. One in black lettering (on white) and one in white lettering (on black). On the other side of the card there may or may not be writing.

  • Your job is to figure out whether–on average–the black numbers are larger or smaller than the white numbers or the same.

5.2 But

  • The catch: when you turn over a card you are only allowed to read the black number or the white number (honor code!!). What’s more: you have to decide before reading the card which number you will look at (though of course you can see the symbol on the card, if there is one, before reading the numbers)

  • To do: Access the complete procedure sheet, get a pack of cards, go!

6 Let’s go 2

6.1 Task

We are collectively going to try to replicate an experimental design that was implemented in Germany.

We are free to modify this design as will. Our goal is to:

  • Modify the design (or not)
  • Generate a pre-analysis plan, including power estimates, if we can
  • File an ethics application: HARD DEADLINE January 19
  • Actually implement it!
  • Run analysis
  • Short write up

We are going to make mistakes and learn to work as a team.

6.2 Starter pack I

AJPS

6.3 Starter pack I

Why do native Europeans discriminate against Muslim immigrants? Can shared ideas between natives and immigrants reduce discrimination? We hypothesize that natives’ bias against Muslim immigrants is shaped by the belief that Muslims hold conservative attitudes about women’s rights and this ideational basis for discrimination is more pronounced among native women. We test this hypothesis in a large-scale field experiment conducted in 25 cities across Germany, during which 3,797 unknowing bystanders were exposed to brief social encounters with confederates who revealed their ideas regarding gender roles. We find significant discrimination against Muslim women, but this discrimination is eliminated when Muslim women signal that they hold progressive gender attitudes. Through an implicit association test and a follow-up survey among German adults, we further confirm the centrality of ideational stereotypes in structuring opposition to Muslims. Our findings have important implications for reducing conflict between native–immigrant communities in an era of increased cross-border migration.

6.4 Starter pack I: Results

Results

6.5 Starter Pack I : A norms study

Summary:

  • Inquiry: Does prejudice depend on outgroup attitudes?
  • Population: People out and about
  • Treatment: Random exposure to outgroup attitudes; lab-in-the-field experimental variation
  • Difference in means (variant)

6.6 Starter Pack II : A protest study

Team

6.7 Starter Pack II : A protest study

Despite decades of scholarship on protest effects, we know little about how bystanders—citizens who observe protests without participating—are affected by them. Understanding the impact of protest on bystanders is crucial as they constitute a growing audience whose latent support, normative beliefs, and concrete actions can make or break a movement’s broader societal impact. To credibly assess the effects of protests on observers, we design and implement a field experiment in Berlin in which we randomly route pedestrians past (treatment) or away from (control) three large-scale Fridays for Future (FFF) climate strikes. Using data gathered on protest days as well as through a one-month follow-up survey, we find evidence for a substantial increase in immediate donations to climate causes but no detectable impact on climate attitudes, vote intentions, or norm perceptions. Our findings challenge the prevailing assumption in both scholarship and public discourse that protest effects operate via impacts on public opinion and call for renewed theorizing that centers on observers’ immediate behavioral activation.

6.8 Starter Pack II : A protest study {.smaller

Results

6.9 Starter Pack II : A protest study

Results

6.10 Starter Pack II : A protest study

Summary:

  • Inquiry: Does protest matter? Does observing a climate protest affect beliefs or behavior?
  • Population: People out and about
  • Treatment: Random exposure to protests; field experimental variation
  • Difference in means (variant)

6.11 Other ideas

  • Information experiments (take the stairs!)
  • Audit experiments (can you send me information?)

7 Let’s create some brainstorming groups

  • 4 groups
  • Each thinks through a possible intervention
    • Motivation: What question can you answer?
    • Treatment: What would be varied?
    • Measurement: What would outcome measures be?
    • Feasibility: Challenges to doing this rapidly and ethically in the coming weeks
    • Sketch of workteams for implementation
  • Initial feedback.
  • Plan to present next week (4 slides). Groups can collude.

8 Appendix: Coding Tips

8.1 Good coding rules

8.2 Good coding rules

  • Metadata first
  • Call packages at the beginning: use pacman
  • Put options at the top
  • Call all data files once, at the top. Best to call directly from a public archive, when possible.
  • Use functions and define them at the top: comment them; useful sometimes to illustrate what they do
  • Replicate first, re-analyze second. Use sections.
  • (For replications) Have subsections named after specific tables, figures or analyses

8.3 Aim

  • First best: If someone has access to your .qmd file they can hit render or compile and the whole thing reproduces first time. So: Nothing local, everything relative: so please do not include hardcoded paths to your computer

  • But: often you need ancillary files for data and code. That’s OK but aims should still be that with a self contained folder someone can open a main.Rmd file, hit compile and get everything. I usually have an input and an output subfolder.

8.4 Collaborative coding / writing

  • Do not get in the business of passing attachments around
  • Documents in some cloud: git, osf, Dropbox, Drive, Nextcloud
  • General rule: only post non sensitive, non proprietary material
  • Share self contained folders; folders contain a small set of live documents plus an archive. Old versions of documents are in archive. Only one version of the most recent document is in a main folder.
  • Data is self contained folder (in) and is never edited directly
  • Update to github frequently

8.5 Fin

Choi, Donghyun Danny, Mathias Poertner, and Nicholas Sambanis. 2023. “The Hijab Penalty: Feminist Backlash to Muslim Immigrants.” American Journal of Political Science 67 (2): 291–306.