At the IPI group at WZB we have five projects innovating on aspects of what we call the “aggregation challenge.” The aggregation challenge is the challenge of combining findings from multiple studies to foster cumulative learning. We think of it as one of the biggest challenges facing social scientists and that addressing it will be key for strengthening the relevance and reliability of social science findings.
Coordinated trials 1. IPI contributes to the "metaketa" initiative housed at EGAP. Metaketas are coordinated randomized trials across multiple sites with harmonized measures and analysis strategies. See this summary paper in Science Advances summarizing the results from Metaketa 1 and this "shiny app" we developed that lets readers explore analyses and robustness of findings to the inclusion or exclusion of different studies.
Coordinated trials 2. We are working on the structure for a "rolling" metaketa on the contact hypotheses, building on work by Alexandra Scacco and Shana Warren and others. The framework connects existing contact experiments using a common causal logic and sets up a structure to allow future studies to enter a live meta-analysis.
Coordinated analysis of strategies to measures hidden populations. We are leading the meta-analysis for a multicountry collection of studies coordinated by APRIES to assess the prevalence of human trafficking prevalence estimation.
Integrated inferences. We are developing the CausalQueries R package that lets users define and combine causal models. See our guide for examples with applications to combine inferences from qualitative and quantitative analyses, inferences from observational and experimental studies, and inferences from multiple trials examining different parts of a common causal model.
Meta-models: Our Correlates of Corona project examines socioeconomic predictors of Covid mortality. Experimental stages now in the field (with Miriam Golden, Alexandra Scacco and colleagues) focus on aggregating disciplinary beliefs about logics driving Corona and strategies to connect observational patterns with causal logics. See here for our challenge.
Banerjee, Duflo, and Kremer have had an enormous impact on scholarship on the political economy of development. But as RCTs have become more central in this field, political scientists have struggled to draw implications from proliferating micro-level studies for longstanding macro level problems. We describe these challenges and point to recent innovations to help address them.
Critics of field experiments lament a turn away from theory and criticize findings for weak external validity. In this chapter, we outline strategies to address these challenges. Highlighting the connection between these twin critiques, we discuss how structural approaches can both help design experiments that maximize the researcher’s ability to learn about theories and enable researchers to judge to what extent the results of one experiment can travel to other settings. We illustrate with a simulated analysis of a bargaining problem to show how theory can help make external claims with respect to both populations and treatments and how combining random assignment and theory can both sharpen learning and alert researchers to over-dependence on theory.
Information, accountability, and cumulative learning: Lessons from Metaketa I
Throughout the world, voters lack access to information about politicians, government performance, and public services. Efforts to remedy these informational deficits are numerous. Yet do informational campaigns influence voter behavior and increase democratic accountability? Through the first project of the Metaketa Initiative, sponsored by the Evidence in Governance and Politics (EGAP) research network, this book aims to address this substantive question and at the same time introduce a new model for cumulative learning that increases coordination among otherwise independent researcher teams. It presents the overall results (using meta-analysis) from six independently conducted but coordinated field experimental studies, the results from each individual study, and the findings from a related evaluation of whether practitioners utilize this information as expected. It also discusses lessons learned from EGAP’s efforts to coordinate field experiments, increase replication of theoretically important studies across contexts, and increase the external validity of field experimental research.
Voter information campaigns and political accountability: Cumulative findings from a preregistered meta-analysis of coordinated trials
Voters may be unable to hold politicians to account if they lack basic information about their representatives’ performance. Civil society groups and international donors therefore advocate using voter information campaigns to improve democratic accountability. Yet, are these campaigns effective? Limited replication, measurement heterogeneity, and publication biases may undermine the reliability of published research. We implemented a new approach to cumulative learning, coordinating the design of seven randomized controlled trials to be fielded in six countries by independent research teams. Uncommon for multisite trials in the social sciences, we jointly preregistered a meta-analysis of results in advance of seeing the data. We find no evidence overall that typical, nonpartisan voter information campaigns shape voter behavior, although exploratory and subgroup analyses suggest conditions under which informational campaigns could be more effective. Such null estimated effects are too seldom published, yet they can be critical for scientific progress and cumulative, policy-relevant learning.
We develop an approach to multimethod research that generates joint learning from quantitative and qualitative evidence. The framework—Bayesian integration of quantitative and qualitative data (BIQQ)—allows researchers to draw causal inferences from combinations of correlational (cross-case) and process-level (within-case) observations, given prior beliefs about causal effects, assignment propensities, and the informativeness of different kinds of causal-process evidence. In addition to posterior estimates of causal effects, the framework yields updating on the analytical assumptions underlying correlational analysis and process tracing. We illustrate the BIQQ approach with two applications to substantive issues that have received significant quantitative and qualitative treatment in political science: the origins of electoral systems and the causes of civil war. Finally, we demonstrate how the framework can yield guidance on multimethod research design, presenting results on the optimal combinations of qualitative and quantitative data collection under different research conditions.