puzzles for causal inference and experimental design

1 Experiments by hand

For each experiment

You have been given an envelope with 20 cards. Each card has two numbers. One in black lettering (on white) and one in white lettering (on black). On the other side of the card there may or may not be writing.

Your job is to figure out whether–on average–the black numbers are larger or smaller than the white numbers or the same.

The catch: when you turn over a card you are only allowed to read the black number or the white number (honor code!!). You have to decide before reading the card which number you will look at (though of course you can see the symbol on the card, if there is one, before reading the numbers)

Proceed as follows:

Step 1: Implement a first experiment

1. Put all the cards face down on the table.
   
2. Decide for which ones you will read the black number and for which ones you will read the white number (\(Z\)). For this you might first want to detemine a strategy. You are free to use any method.
3. Turn each card over and read the appropriate number
4. Fill in a table like the below in a spreadsheet (e.g. .csv file)
5. When all numbers are complete calculate:
   1. An estimate of the average difference between the black and white numbers
   2. A standard error associated with this estimate
   3. A \(p-\)value for the null hypothesis that there is no difference

[Discuss]

Step 2: Redesign and repeat

6. You have completed the first investigation. You are now doing to do 10 more. Write down a brief pre-analysis plan for these next investigations. This is your design. Based on what you found so far you might want to (a) revist your assignment strategy in step 2 and (b) you analysis strategy in step 4.
7. Repeat steps 1 - 5 10 times and show histograms of
   1. the estimates
   2. the standard errors
   3. the \(p-\)values

[Discuss]

Step 3: Evaluate your design

8. Now turn over all the cards:
   1. calculate the true average difference between the numbers
   2. using the calculations you have, estimate:
      1. whether your design was biased
         
      2. the power of your design
      3. the coverage of the confidence intervals
   3. calculate the correlation between the white numbers, the black numbers
   4. calculate the correlation between the black numbers and the prompt / clue (if any)

[Discuss]

ID	Z: Color read (W = 0, B = 1)	Y: Number observed	Y0: White number (if read)	Y1: Black number (if read)	X: Background data
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20