Using DeclareDesign for power analysis

Insights from the MIDA framework

Graeme Blair, Alex Coppock, Macartan Humphreys

WZB presentation | November 2022

Plan

Topics

  1. The MIDA framework and the declaration-diagnosis-redesign cycle
  2. Brief intro to DeclareDesign
  3. \(p\)-values review
  4. Power review
  5. Power via design diagnosis: the how to and advantages thereof
  6. Lessons & strategies
  7. Some applications

Timing

  • Part I: Topics 1-4 and exercises 1
  • Break: 10:13 - 10:33
  • Part II: Topics 5-6, 10:33 - 11:30
  • Break: 11:30 - 11:45
  • Part III: Topic 7, 11:45 - 12:30

The MIDA Framework

Declaration

Q: Is my research design good?

A: Well let’s simulate it to see how it performs.

Q: What should I put in the simulation?

A: All elements of a research design.

Q: What are the elements of a research design?

A: M! I! D! A!

Four elements of any research design

  • Model: set of models of what causes what and how
  • Inquiry: a question stated in terms of the model
  • Data strategy: the set of procedures we use to gather information from the world (sampling, assignment, measurement)
  • Answer strategy: how we summarize the data produced by the data strategy

Four elements of any research design

Declaration, Diagnosis, Redesign

Declaration

Telling the computer what M, I, D, and A are.

Diagnosis

Estimating “diagnosands” like power, bias, rmse, error rates, ethical harm, amount learned.

  • want to diagnose over model uncertainty

Redesign

Fine-tuning features of the data and answer strategies to understand how they change the diagnosands

  • Different sample sizes
  • Different randomization procedures
  • Different estimation strategies
  • Implementation: effort into compliance versus more effort into sample size

Very often you have to simulate!

  • This is too hard to work out from rules of thumb or power calculators
  • Specialized formulas exist for some diagnosands, but not all.

DeclareDesign

Key commands for making a design

  • declare_model()
  • declare_inquiry()
  • declare_assignment()
  • declare_measurement()
  • declare_inquiry
  • declare_estimator()

and there are more declare_ functions!

Key commands for using a design

  • draw_data(design)
  • draw_estimands(design)
  • draw_estimates(design)
  • get_estimates(design, data)
  • run_design(design), simulate_design(design)
  • diagnose_design(design)
  • redesign(design, N = 200)
  • design |> redesign(N = c(200, 400)) |> diagnose_designs()
  • compare_designs(), compare_diagnoses()

Cheat sheet

https://raw.githubusercontent.com/rstudio/cheatsheets/master/declaredesign.pdf