How useful was social science during the pandemic?

Macartan Humphreys | WZB

Plan

The pandemic was useful for social science, but how useful was social science for the pandemic?

A look at (a small!) selection of work in political science and economics on:

  • Predicting
  • Tracking
  • Solving
  • Critiquing

Was this useful? Where could we have done better?

General remarks

Rapid responses, broad responses

  • Oxford policy data tracker
  • Tracking dashboards
  • Pooled surveys: RECOVR, Y-RISE
  • Special issues (e.g. POP) and funding opportunities (e.g. IPA, Mercury)

“Staying in lane” issues fairly well negotiated

Predicting: Where will the Corona burden lie?

  • Lots of epi models: does social science have value added?
  • Social scientists are particularly good at predicting regular events after the fact
  • Real challenge to predict outside the routine
  • Is it too much to ask?

Predicting

Bosancianu et al 2023: lots of available theory (in all directions)

Predicting

Capacity arguments mixed: astonishing unconditional relations, capacity correlates early on, trust later on

Predicting

Not much traction from political variables

Predicting

Trust and inequality stand out

Implied learning

Expert predictions (generally very diverse) and implied updating

Model challenge

Over 100 entries across challenges: Golden, Slough, and Zhai (2023)

Model challenge

Social scientists better than chance but only in the tails

Model challenge

Trust and inequality again

Summary

  • Not clear much demand for social science predictions
  • Quite modest traction: but still, value added over epidemiological and machine learning models
  • Purely political arguments did not do a lot; theoretical logics thin
  • Trust and inequality consistently important

Tracking

A lot of social science is focused on measurement.

Did social science effectively help track the situation?

Dashboards

Sierra Leone dashboard making use of existing sample and team (esp Voors and Mereggi in country): resources

Dashboards

Drawn on by MoF

Dashboards

Similar dashboards in Uganda and elsewhere

Social costs

Huge costs cross-nationally from policies: Egger et al. (2021)

Vaccine acceptance

Despite bad press: broad willingness to use vaccines. Solı́s Arce et al. (2021)

Many many more

Many observational papers identifying who takes what kind of action

  • e.g. low trust Goldstein and Wiedemann (2022)
  • POP special issue

Summary

  • Social scientists stepped up to address measurement issues
  • Made use of existing infrastructures
  • But also flexible
  • Engaged with policy in developing areas

Solving

Are social science contributions helpful for formulating policy?

  • Survey work
  • Nudge work
  • Field work

Surveys

Incentives to take up vaccine, Klüver et al. (2021)

Surveys

Attitudes to restrictions (Germany), Hartmann et al (2023)

Surveys

Broad (non-strategic) support for vaccine sharing (Germany), Geissler et al. (2022)

Nudges

A focus for many:

Mixed messages on messaging: Dai et al. (2021)

But failures to replicate (Rabb et al. 2022)

Nudges

Regret lotteries: positive news at first, but no replication Milkman et al. (2022)

Field: Masks

Perhaps the largest masking study done by economists (Abaluck et al. 2022; Jefferson et al. 2023); WHO change in policy. Wild heterogeneity across studies.

Field: Last mile

Key study of vaccine delivery Mobarak et al. (2022)

Taking costing seriously

Summary

  • Some weaknesses: survey dependence, relying on easy fixes

but

  • Social science methods performing well (theory contributions less obvious)

  • Striking agility of some (often US based) social science researchers: access to data, willing to move quickly at scale, “shoe leather” research.

Critiquing

Some points of intersection some points of tension seen in lane crossings

Critiquing

  • Social science reading of health analyses suggests scope for gains
  • Statistical interpretations
    • \(p\) values
    • Weak cost - benefit analysis
    • Interpretation of nulls
    • Baseline balance
    • Selection biases
  • More attention to contextual heterogeneity: coordinated trials
  • Weak between community connections (Sunstein et al. 2022; Lees et al. 2021)

Critiquing

Null interpretations, experimental design, theory

Future

Implications for the next time.

Lessons

Infrastructure

  • Reliance on spotty coverage from project specific surveys: nothing like a global barometer. Can we imagine this?

Networks

  • Strengthen networks, share methods
  • Need: Consumer (government) side analysis

Questions

  • Health consequences of social divisions
  • Optimal responses model taking account of economic and political costs

References

Abaluck, Jason, Laura H Kwong, Ashley Styczynski, Ashraful Haque, Md Alamgir Kabir, Ellen Bates-Jefferys, Emily Crawford, et al. 2022. “Impact of Community Masking on COVID-19: A Cluster-Randomized Trial in Bangladesh.” Science 375 (6577): eabi9069.
Dai, Hengchen, Silvia Saccardo, Maria A Han, Lily Roh, Naveen Raja, Sitaram Vangala, Hardikkumar Modi, Shital Pandya, Michael Sloyan, and Daniel M Croymans. 2021. “Behavioural Nudges Increase COVID-19 Vaccinations.” Nature 597 (7876): 404–9.
Egger, Dennis, Edward Miguel, Shana S. Warren, Ashish Shenoy, Elliott Collins, Dean Karlan, Doug Parkerson, et al. 2021. “Falling Living Standards During the COVID-19 Crisis: Quantitative Evidence from Nine Developing Countries.” Science Advances 7 (6): eabe0997. https://doi.org/https://doi.org/10.1126/sciadv.abe0997.
Geissler, Ferdinand, Felix Hartmann, Macartan Humphreys, Heike Klüver, and Johannes Giesecke. 2022. PLOS ONE 17 (12): e0278337.
Golden, Miriam A, Tara Slough, and Haoyu Zhai. 2023. “Gathering, Evaluating, and Aggregating Social Scientific Models.” Available at SSRN 4570855.
Goldstein, Daniel AN, and Johannes Wiedemann. 2022. “Who Do You Trust? The Consequences of Partisanship and Trust for Public Responsiveness to COVID-19 Orders.” Perspectives on Politics 20 (2): 412–38.
Jefferson, Tom, Liz Dooley, Eliana Ferroni, Lubna A Al-Ansary, Mieke L van Driel, Ghada A Bawazeer, Mark A Jones, et al. 2023. “Physical Interventions to Interrupt or Reduce the Spread of Respiratory Viruses.” Cochrane Database of Systematic Reviews, no. 1.
Klüver, Heike, Felix Hartmann, Macartan Humphreys, Ferdinand Geissler, and Johannes Giesecke. 2021. “Incentives Can Spur COVID-19 Vaccination Uptake.” Proceedings of the National Academy of Sciences 118 (36): e2109543118. https://doi.org/https://doi.org/10.1073/pnas.2109543118.
Lees, Shelley, Salla Sariola, Megan Schmidt-Sane, Luisa Enria, Kit-Aun Tan, Angel Aedo, Koen Peeters Grietens, and David Kaawa-Mafigiri. 2021. “Key Social Science Priorities for Long-Term COVID-19 Response.” BMJ Global Health 6 (7): e006741.
Milkman, Katherine L, Linnea Gandhi, Sean F Ellis, Heather N Graci, Dena M Gromet, Rayyan S Mobarak, Alison M Buttenheim, et al. 2022. “A Citywide Experiment Testing the Impact of Geographically Targeted, High-Pay-Off Vaccine Lotteries.” Nature Human Behaviour 6 (11): 1515–24.
Mobarak, Ahmed Mushfiq, Niccolò Meriggi, Maarten Voors, Madison Levine, Vasudha Ramakrishna, Desmond Maada Kangbai, Michael Rozelle, Ella Tyler, and Sarah Cundy. 2022. “Solving Last-Mile Delivery Challenges Is Critical to Increase COVID-19 Vaccine Uptake: A Cluster Randomized Controlled Trial.”
Rabb, Nathaniel, Megan Swindal, David Glick, Jake Bowers, Anna Tomasulo, Zayid Oyelami, Kevin H Wilson, and David Yokum. 2022. “Evidence from a Statewide Vaccination RCT Shows the Limits of Nudges.” Nature 604 (7904): E1–7.
Solı́s Arce, Julio S., Shana S. Warren, Niccolò F. Meriggi, Alexandra Scacco, Nina McMurry, Maarten Voors, Georgiy Syunyaev, et al. 2021. “COVID-19 Vaccine Acceptance and Hesitancy in Low- and Middle-Income Countries.” Nature Medicine 27 (8): 1385–94. https://doi.org/https://doi.org/10.1038/s41591-021-01454-y.
Sunstein, Cass, Maria Augusta Carrasco, Varun Gauri, Gavin George, Ross Gordon, David Houéto, Ruth Kutalek, et al. 2022. “Behavioural and Social Sciences Are Critical for Pandemic Prevention, Preparedness, and Response. WHO Technical Advisory Group on Behavioural Insights and Sciences for Health [Open Letter].”