An estimate is a guess about the value of some quantity of interest (the “estimand”). Here are questions people have when wondering if an estimate is valid.
Probably not. Generally in an RCT you don’t have to control for anything in the following sense: in general, your results are unbiased when you don’t control for anything.1 This is true whether or not treatment and control groups look similar. Intuitively the reason is that any third thing, gender say, is ex ante no more likely to be overrepresented (or underrepresented) in the treatment group than in the control group, thanks to randomization. This is what trips people up: Unbiasedness is a statement about whether your estimates would be too high or too low on average if you repeated the experiment, not about whether you will be too high or too low in this instance.2 With that said, including background variables in estimation can help make sure that you make smaller errors (again, on average) and so you might want to see those variables included.
So: you generally don’t need to be worried about things that are not controlled for if you are worried about bias. You might be worried if you think that unnecessarily imprecise estimation is making it too likely that the null hypothesis will be maintained when it should be rejected. Or if you worry that your unbiased estimate might well be very far from the truth (see for example some of the arguments in favor of blocking in Multivariate Continuous Blocking to Improve Political Science Experiments).
Probably not.3 See here. Even if you have imbalance on one or more background covariates (see here).
Maybe. We sometimes control for background variables when looking at the difference between treatment and control outcomes. This is often done in order to produce more precise estimates. It’s often a good thing to do but there can be two things to worry about:
If a control is “post treatment” then including it can generate bias. You do not want to control for something that itself was affected by treatment. Say a medicine prevented death by reducing a fever. If you control for fever you might find that, conditional on fever, things look the same in the treatment and control groups and falsely conclude that the medicine had no effect. But of course its effect worked through the thing you just controlled for.
While simple differences in means is guaranteed to be unbiased in an experiment, regression with adjustment is only guaranteed to be “consistent” (informally: works well when you have lots of data), and so you might worry when you have small samples. (See Lin on this point and ways to address it.)
Including controls can change the precision of estimates and sometimes test of hypotheses about estimates are “significant” when controls are used but not when they are not. In this case you might be worried that researchers have selected the controls because they made the result significant. This is a form of “fishing” and can lead to false inferences. If researchers have pre-specified their controls in a pre-analysis plan then this is probably not a cause for concern.
For more on when to include covariates in your analysis (or when not to), see 10 Things to Know About Covariate Adjustment.
No. Baseline measures are often a good idea and they can let you implement a better random assignment and get tighter estimates. But they are not needed for inference from an RCT. The fact that there are no baseline measures should not lead you to read estimates of effects differently or interpret standard errors or confidence intervals differently, since lower precision will normally be captured by these numbers already.
There is confusion here because we intuitively think the goal is to estimate the difference between changes in the treatment and changes in the control group. But an amazing thing is that with randomization the differences in the average changes is the same as the difference in the average outcomes in treatment and control groups (in expectation).4 So we are not talking about different quantities. A better way to think about baseline data is that it provides excellent controls, which you may or may not make use of (see here and here).
Probably not. Inference based on randomization does not depend on homogeneity (a lovely reading on this is by Paul Holland). In particular there is no assumption that treatment works the same way for all people. Rather all we need (for unbiasedness) is that the average outcome in a random sample in any condition is the same (in expectation) as what the average outcome would be for the whole population in that condition.
Maybe. If you randomize then treatment and control groups should look fairly similar on pretreatment variables — gender, age, and so on. But of course in practice you can be certain they will look different.
xBalance command of the RItools R pacakge or (2) adjust the p-values so that you are not over-interpreting some few small p-values that arise by chance (say, using the p.adjust() command in R).Probably. The standard analysis assumes that each subject reacts only to their own treatment status. If people react to other people’s status then simple analyses can produce the wrong results. For intuition if everyone gets healthy because half the population got a treatment then you will estimate an effect of 0 when in truth there was a positive effect for everyone.
This is often called a problem of “spillovers” or, more technically, a violation of the “SUTVA” assumption. There are ways to address it and you should look to see whether people took account of this risk when doing analysis. Here is a nice treatment of these issues. And see 10 Things to Know about Spillovers for a less technical overview.
Maybe not. It’s not uncommon that you can randomly “assign” a treatment but whether people actually take up the treatment is another matter. It could well be that only the people for whom the treatment works will actually take it up and others won’t bother. This problem goes by the name of “noncompliance.” Surprisingly this is less of a problem than it might seem. There are a couple of ways that researchers likely respond to this:
The “intention to treat” analysis is perhaps the most common. Researchers report on the effects of assigning a treatment, not the effects of taking up a treatment. This is often a policy relevant quantity. The fact that not everyone takes it up is not a problem here (rather it can be thought of as one of the things that dampen the effects of assigning people to treatment).
Another approach is to focus on the “complier average treatment effect” or the “local average treatment effect”: the effect specifically for those that would take up the treatment if and only if they were assigned to treatment. For intuition you can estimate the effect of assignment on uptake and the effect of assignment on outcomes and then divide the latter by the former. For instance if giving a mask increases the probability of wearing a mask by 50 percentage points, and giving a mask reduced sickness by 10 percentage points. Then the estimated effect of wearing the mask (among those that wore a mask because you gave them a mask) is 20 percentage points; you can now think of the estimated 10 points as an average of 20 (for the compliers) and 0 (for the non compliers). The approach requires a few assumptions but it is often viable (the seminal reference is Angrist et al; see also here for discussion of some assumptions and references). And see the discussion of the ITT in Estimating Estimands with Estimators and 10 Things to Know About the Local Average Treatment Effect.
Another approach is to focus only on people that took up the condition that they were assigned to and ignore the data of “non compliers” (this is sometimes called “per-protocol”). You should worry if you see this because it can produce biased estimates.
Maybe. Sometimes you end up getting measures for some people but not for other people and then people for whom you do not have measures end up dropping out of your sample. This can carry a risk of bias.
For intuition imagine a drug has positive effects for for half your treatment group and negative effects for the other half. The ones that experienced negative effects are reasonably upset and stop taking part in the study. So you only see good outcomes in the treatment group. This is called “attrition.” If there is a lot of attrition you should check whether the authors have tried to address it. One approach is to try to demonstrate that the attrition was as good as random. Another is to show that attrition is unlikely large enough to change substantive conclusions. Possible responses are provided in Lin et al.’s standard operating procedures.
Probably not. Linear models generally work well for experimental data. People argue on this point but the differences in estimates of the ATE from different approaches is often not large.
In simple set ups with experimental data, linear (OLS) models return the difference in means estimate, which is unbiased, no matter the data type: and with a binary outcome the difference of means is a difference in proportions.5 With that said there are lots of different approaches that can be used to “model” different types of data generating processes, such as for binary data, ordinal data, or count data. These can get at quantities that linear models do not aim for and that can be a reason to use them. But warning: they might not be justified by randomization and might not be worth it if you are interested in the average treatment effect.6
Probably not.
That is my guess at the probability we would see such a difference between treatment and control groups if in fact the treatment had no effect. If the p value is low then I recommend that you stop thinking that there might not have been an effect.
Informally that is the conclusion we tend to draw but technically a very low p value is not a statement about the effect you estimated but about the hypothesis of no effect.
Frequentist statistics are not designed to learn about the probability that one or other effect is correct but rather whether you should doubt one effect or another. For claims about the probability of a given effect see Bayesian inference.
The confidence interval is a surprisingly obscure object.
The wrong interpretation: Informally people think of it as a describing a set of possible values such that there is a 95% probability that the truth lies within the confidence interval. This is wrong because the construction of the confidence interval does not use any information about how likely one or other possible effect is. It uses information about how likely the data are given some effect or other.
The technical feature that the confidence interval should have: The confidence interval is a pair of functions producing upper and lower bounds (an interval) designed so that if we repeated the experiment, and recalculated the interval, there would be a 95% chance (or more) that the interval would include the true effect (it would “cover” the truth 95% of the time).9 Now that’s a pretty hard idea to make sense of. And although it sounds a lot like it, what you cannot get from this definition is that the particular (confidence) interval we reported includes the true effect with 95% probability.10 It either does include it or it does not. This definition doesn’t help much in knowing what to do with the particular interval you have before you. (Though see here for a lead)
An interpretation we can understand. The interval can often be interpreted as the set of values that you cannot reject (at the 5% level) given the data.11 Values outside the set are values that you can reject (at the 5% level). So one (“frequentist”) approach is to reject all values outside the confidence interval and maintain all those inside. This interpretation is behind the common practice of saying “0 is inside my confidence interval so I cannot distinguish my estimated effect from 0.”
No (or maybe: no basis for the conclusion).
There is a sense that every number inside a confidence interval should be treated the same way: they are all values that you do not reject.
But that does not mean that they are all equally likely. For a statement about whether an estimate (or range of estimates) is likely you need some prior about whether they are likely. In other words you need information that the confidence interval doesn’t use and doesn’t provide. But see here.
Also, the p-values for values near the edges of the 95% CI will be near .05 and near the center of that interval the (two-sided) p-values will be higher than .05.
At a stretch. In practice people think of confidence intervals as if they were 95% Bayesian “credibility intervals” (a range that holds the true value with 95% probability) and think of numbers in the middle as more likely.
Technically this is incorrect. The confidence interval does not use or provide information about the probability of one effect or another.
But even though the informal practice is not technically correct, it is not completely crazy either. In some applications the confidence interval can look a lot like the credibility interval. For intuition: Imagine I thought that if the true effect were \(\tau\) the estimates I would get from my experiment would be distributed normally across experiments, \(t \sim N(\tau, 0.5)\). Say I estimate \(t = 1\). Then my confidence interval would be something like \([0.02, 1.98]\). If I further supposed that ex ante I thought any value of \(\tau\) equally likely, then my posterior belief would be proportional to the likelihood of observing 1 under each possible value of \(\tau\). The most likely value would indeed be \(\tau = 1\) and I would put about 95% probability mass on the possibilities in the credibility interval. I would put much lower weight on values near the edge of the confidence interval. (Bayesian plug: If you want to give a Bayesian interpretation ask for a Bayesian analysis from the get go).
Maybe. There is a sense in which if 0 is right at the edge of a confidence interval that your estimate was almost not significant. Or put differently, perhaps you can reject the null of 0 but you cannot reject the null of an absolutely tiny effect. This is the key thing: there is nothing truly sharp about these thresholds, there is no real difference being just inside or just outside a confidence interval in the same way as there is no real difference between \(p = .051\) and \(p = .049\). In particular, tiny differences have almost no bearing on what effects are “likely”. As Gelman and Stern nicely put it: the difference between “significant” and “not significant” is not itself statistically significant. See also this.
No. Weirdly it is possible that you can distinguish A’s effect from 0 and not distinguish B’s effect from 0, but still not be able to rule out that A and B have the same effect. It’s even possible that you have a significant effect for A and not B but that your estimated effect is bigger for B than for A.
For intuition, imagine you had a tight estimate for treatment A, with a mean of 1 and a confidence interval of [.9, 1.1]. For B your estimate is very noisy, with a mean of 2 but confidence interval of [-1, 5]. Here your estimate for B is so noisy that you cannot distinguish it from 0 or from 1. You would be wrong to infer that A is more effective than B.
If you are reading a paper and statements are made about the relative effects of treatments, look for the analyses that specifically back up these claims.
No. See here.
If you are reading a paper and statements are made about the different effects of treatments for different groups, look for the analyses that specifically back up these claims. These often come in the form of an interaction term: Treatment \(\times\) Gender.
Maybe. Your study is underpowered if (ex ante) you are not very likely to reject a null of no effect even when there is a real effect. The basic problem is that the design is not capable of generating sufficiently precise estimates to be able to distinguish real effects from noise. The problem with that is that you might (falsely) conclude that there is no effect even though there is one. Having more observations generally improves power and underpowered studies are often small. But other things matter also, such as how randomization was done, how measurement was done, and what estimators are used.
If you are worried about power it can be useful to look at confidence intervals. An underpowered study has big confidence intervals. Before discounting an intervention on the grounds that you cannot reject the null of no effect you might also ask whether you can reject the possibility that there are very large effects. If you cannot reject either large or small effects you might conclude that there is not too much to learn here.
You might think that low power is not a problem if in fact you get a significant result. But there’s a subtlety here. Conditional on being significant, an underpowered estimate is more likely than not to be an overestimate. If someone brings your attention to an underpowered published study with significant results and huge effects it’s probably too good to be true. (Gelman calls this the significance filter; we discuss implications for publishing here)
For more on power, see 10 Things to Know About Statistical Power.
No. People sometime say: “absence of evidence is not evidence of absence” which captures the idea that in general, null results should be interpreted as absence of evidence. A null result just means that data like this could have been produced easily enough by a process in which there is no true effect.12 The problem is not helped by the fact that unfortunately researchers often say “we found that this treatment didn’t work” when really they mean “we weren’t able to rule out the possibility that there is no effect.”
Better in this case to look at the confidence interval to see whether they are tight. If you have a null result with tight confidence intervals that means that large effects are rejected—bigger effects sizes, outside the confidence interval, are not likely to have produced data like this.
For more on interpreting null results, see 10 Things Your Null Result Might Mean.
No. A lot of the statistics commonly used for analyzing RCTs are for figuring out if the data support some or other hypothesis. But the decision to take an action should depend on what you believe the costs and benefits are and knowing that an estimate is “significant” does not help much with that decision.
As a concrete example consider a version of the problem described here and imagine that if the true effect were \(\tau_j\) the estimates I would get from my experiment would be distributed normally if I were to repeat my experiment many times, \(t \sim N(\tau_j, 1)\). Say I estimate \(t = 1\). Then my confidence interval would be something like [-1, 3]. So I would have a null result, with an estimate that is quite far from significant. Say though that my benefit from intervening was \(\tau\) but my cost was .5. Then my expected net benefit from intervening would be 1-.5 = .5. If my benefit were \(\tau^2\) then my expected net benefit would be 2 - .5 = 1.5. So I have reason to take an action even though I cannot reject the null of no effect.
The bigger point is that when using data to make inferences you should not focus on significance alone, or on the estimate of effect sizes alone, but try to think through the implications given your beliefs over a range of plausible effects.
Maybe. There is increasing attention to the importance of attending to ethics in experiments (and other research). RCTs invoke specific ethical concerns insofar as they, typically, manipulate some people in order for other people to learn things. With that said they are often implemented specifically to learn things that could be beneficial to participants and to others and in ways that are equitable and avoid harm to participants. There is however variation in the extent to which they involve proper informed consent—informed consent is common for measurement but less common for social science interventions.
If you have concerns about the ethics reach out to the authors. There may be features of the intervention that address your questions that are not in the paper. It’s useful to draw authors’ attention to these issues because such features should be in the paper.
There have been cases where beneficial treatments have been withheld from control groups for research purposes which raises obvious ethical concerns. If you are worried about this, first find out if that is what actually happened. A number of the items in Karlan and Udry’s framework help assess this point but the information is still often missing in writeups.
Sometimes what looks like straight up withholding benefits has some or more of the following features:
A main reason to use RCTs is that different types of “selection bias” can result in very misleading inferences when we look at patterns as they occur naturally in the world. For instance development programs might operate in easy-to-access areas. When we compare places with and without programs we might think that the program is very effective because many other good things are going on in easy-to-access areas and so we mistakenly think that the program is effective when it is not. Similarly we might find no differences between areas in which a program is in operation and those in which it is not and wrongly conclude that there is no effect—rather the program chose disadvantaged areas to work in and so the lack of a difference is actually evidence of an effect. The random assignment to treatment provides confidence that none of these selection biases are producing faulty inferences.
If you are worried about whether an RCT was justified it can be helpful to (i) think through what are the types of biases in this case that might undermine inferences from observational comparisons (or: are there reasons to think that you might be wrong about what you think an intervention does) (ii) compare the costs of the RCT to the costs and benefits of the intervention—a million dollar RCT that gives guidance on a billion dollar intervention can be money well spent (iii) try to think back to what you thought before seeing the result of the RCT and benchmark learning against that, likely either your beliefs about effect sizes changed or your confidence in your beliefs, or both (iv) try to think through how you might have changed your beliefs had the RCT produced outcomes different to what it did.
There are some exceptions: for instance if you randomize with different probabilities within blocks then you should control for those blocks: a great approach is estimate the effects within each block and then combine the estimates across blocks. ↩
In practice you may have more men in treatment and more women in control, say. And it may well be that your estimate of effects partly reflects this, in the sense that you would have gotten different estimates if it were the other way around. But your estimators is unbiased for the simple reason that bias is a statement about whether you expect to be right on average and not a statement about whether you are right this time. ↩
The rare cases in which you might worry are if the design used a randomization procedure — such as assigning different units with different probabilities — that calls for adjustment at the analysis stage. ↩
For a bit more intuition the average of the differences \((\overline{y}^T_{t=1} - \overline{y}^T_{t=0}) - (\overline{y}^C_{t=1} - \overline{y}^C_{t=0})\) can also be written as \((\overline{y}^T_{t=1} - \overline{y}^C_{t=1}) - (\overline{y}^T_{t=0} - \overline{y}^C_{t=0})\), where the first part is just the difference between endline outcomes and the second part is zero in expectation, thanks to randomization. A slightly deeper (and simpler) way to think about it is in terms of the estimand: if we are interested in average changes we are interested, in the average, over individuals, of \((Y_i^{t=1}(1) - Y_i^{t=0}) - (Y_i^{t=1}(0) - Y_i^{t=0}) = Y_i^{t=1}(1) - (Y_i^{t=1}(0))\), where \(Y_i^{t=1}(x)\) is a potential outcome—the value \(Y\) takes in condition \(x\)—but \(Y_i^{t=0}\) is a realized outcome. ↩
One word of caution: if the data is ordinal, say taking on values (0 = unhappy, 1 = happy, 2 = very happy) the ATE you estimate with OLS is the effect of the treatment on the scale you are using. If you decided that “very happy” should be coded as a 3 not a 2 you would get different answers. ↩
For discussion of gains in the case with covariates see Negi and Woolridge (Part 7). ↩
In fact using a Horvitz Thompson estimator you can get unbiased estimates if you have only one observation in total! ↩
e.g. the null that the treatment had no effect on all units. ↩
It is often conceived of as a type of set valued estimate, but in practice it is used to provide a sense of uncertainty about a point estimate or range of plausible hypotheses — hypotheses that, if tested, would not yield \(p < .05\) in the case of a 95% CI. ↩
Back in 1954 Savage complained about how routinely this has to be pointed out. ↩
For intuition: Say your p values are calculated correctly and so if the truth is 0 you will declare it to have a p value below 0.05 just 5% of the time, then 95% of the time 0 will be within the set that you do not reject. Similarly for other null hypotheses that you might consider. Greenland et al highlight many more fallacies and clarify conditions under which this interpretation is valid. ↩
At the same time, any Bayesian would likely argue that a null result suggests that there may be nothing going on. Expressing the idea properly requires a Bayesian set up however. See here. ↩
The discussion raises, I think, some interesting questions for work that depends on both normative and empirical claims—as does a lot of theoretical work in political science. Below I summarize where I think things ended up and where I think there are points of confusion. I end with a question about whether or when theory papers should provide support for empirical premises on which a normative conclusion depends.
There are lots of interesting parts to Mittiga’s paper, but I think the core of the argument relates to three propositions. (Note this is just my attempt to reconstruct the argument; it is not found in this form in the text so apologies to Ross if I mix it up.)
The propositions:
We want to get from A and B to C. In my reading we can’t quite get there directly since A and B only get us to “Non-authoritarian policies are not legitimate in situation X.” So we seem to need something more. One possibility might be to add an assumption:
Assumption O: In any situation some set of policies is always legitimate.
Under Assumption O, C is implied by A and B.
If we don’t maintain Assumption O then we could still get to C with variations on A and B (A’ : “Any policy that is required for the survival of a polity is ipso facto legitimate“; B’ : “Authoritarian policies are required for the survival of a polity” 1) Other strategies might involve moving beyond necessity and sufficiency statements to more continuous statements (such as “better able to ensure the survival of a polity”).
There are two further distinct types of conclusion that follow from C:
We might call D the strong conclusion—it follows if A and B are true (assuming O or similar); and E the weak conclusion, an analytic claim of the if-then variety that does not require A and B to be true.
I think the key contribution of the article is claim E not claim D (though I have some confusion on this, see below). That is, to point out that “if A and B are true then C and D are true.” Sure enough, given Assumption O, C follows from A and B. And D follows from C. D then is useful insofar as it connects the dots and highlights an important conclusion for anyone who thinks A and B are plausible.
A second contribution, I think, is to argue that A is true.
Beyond the missing assumption noted above, I see a few points of confusion (that is, where I was confused). Some of these have been addressed in the discussion but not all:
Regimes or policies? The term “authoritarian governance” is not too clear to me: at some points it sounds like authoritarian regimes are meant, not authoritarian policies implemented by democracies (the title mentions “authoritarianism”; the early discussion compares states of different regime types); Mittiga’s responses emphasize that policies not regimes are meant and this is also laid out quite clearly in the section “Authoritarian Climate Governance”.
Strong or weak claim? I think it is sometimes unclear whether Mittiga is making the stronger or weaker claim. At some points it sounds like Mittiga is in fact supposing A and B are indeed true and so inferring that C and D are indeed true. Thus claiming D itself, rather than the weaker claim “If A and B then D”. The last line of the abstract for instance seems to state D directly, but it is caveated in a way that suggests to me that the weaker claim is meant (because of the uses of “suggests” / “may” and, critically, an incorporation of a version of A as a condition of the statement itself). The conclusion uses somewhat similar wording that seems more in line with the weak claim.2 The tweeted responses suggest though that the strong (“disturbing”) claim is meant.3
Is A true by definition? There are parts of the text that suggests A is true by definition, but ultimately I think A is meant as a normative claim. When foundational legitimacy (FL) is defined we are told that “FL requires that citizens’ essential safety needs are met.” So meeting needs seems to be part of the definition of FL. Later we have “legitimacy requires that governments ensure the safety and security of their citizens” (emphasis added). I understand this to mean that FL is not definitional to legitimacy, but rather, following the argumentation in the text, that in fact FL is necessary (though not sufficient) for legitimacy. Thus CL and FL are not types of legitimacy, but rather are things that can contribute in some way to legitimacy (indeed, sometimes: “political legitimacy will depend more (or even exclusively) on whether security needs are met than on whether CL factors are satisfied”).
Is B true by definition? There’s a part of the text that suggests that B might be set up to be true by definition but ultimately I think B is meant as an empirical proposition. The challenges arise where there is a “state of exception,” where a “state of exception […] is precipitated by an emergency (or credible threat thereof) of sufficiently great magnitude that prevailing political institutions, processes, norms, etc. either impede the swift action needed to preserve/restore normal conditions or simply break down”. This sounds like the inability of democratic institutions to address the problem is part of the criteria determining whether you have an SOE. But such a tautology would I think defang claim D; and elsewhere versions of B seem to be treated as empirical claims not a tautology (indeed: “ultimately an empirical question whether authoritarian governance is better able to realize desired environmental outcomes”). So the key part of B , not contained in the definition, is the idea that authoritarian governance would fare better (or: some policies will work).
Is a still stronger claim implied?: some of the reactions were to a stronger claim than Mittiga made. But there might be a reason for this. If you in fact buy A and B and you believe we now are in situation X (understood as a crisis), then the same logic would lead you not to D but to: “D’: authoritarian governance is legitimate.” So a more extreme conclusion can follow depending on what you think about X — the state of degradation of the climate.
This discussion has been interesting because it was between an empirical and a theoretical researcher. Evaluation of the argument calls, I think, for that kind of discussion. The conclusions depend on what you think of A (a normative claim) and B (an empirical claim). So attention naturally goes to these.
A (the normative claim): Assuming A is not meant to be true by definition I’d like to learn more about why we should believe A. Addressing this requires a sharper statement of legitimacy (and not just of FL and CL) than we have in the text, I think. It doesn’t seem hard to me to imagine notions of legitimacy that are consistent with self sacrifice: taking actions that threaten survival but are good for other reasons purpose. Does A hold broad support among theorists?
B (the empirical claim): Wuttke is concerned with B and I think both sides agree that you can’t use Mittiga’s argument to get to D if you don’t buy B. Mittiga doesn’t do a lot to establish B however. Should he? More generally, what do we think is the responsibility of theorists to establish the plausibility of a condition on which a normative conclusion (here C and D) depends? I think the answer depends in part on whether claims like the strong claim or the weak claim are meant. If the strong claim is meant then B really is needed for the argument and Wuttke’s concerns are critical. If the weak claim is meant then the importance of the claim (rather than the validity of the claim) depends on whether B is plausible. In that case might it be enough to argue that many people believe B or at least that many believe B to be possible?
Notes
I invoke Assumption O rather than the stronger version of A because Mittiga emphasizes that ensuring survival may be necessary but is not sufficient for legitimacy (with a nice discussion of Hobbes and Williams). A difficulty with Assumption O is that it is inconsistent with A if no policies are consistent with survival. ↩
“But if adhering to CL factors proves incompatible with responding effectively to the climate crisis, then political legitimacy may require adopting a more authoritarian approach” (emphasis added). ↩
“In short, the article is an attempt to draw our attention to one of the most significant threats to our ability to maintain democratic institutions and robust human rights… Should those of us fortunate enough to live in liberal-democratic states fail to respond adequately to the climate crisis, we open the door to legitimating darker, more authoritarian forms of climate governance.” ↩
There has been a rich discussion going on on blogs and Twitter around evaluations of CDD following the release of a report by 3ie. See here especially: https://oxfamblogs.org/fp2p/the-last-word-in-the-community-driven-development-wonkwar-scott-guggenheim-responds-to-howard-white-and-radhika-menon/
The takeaways are still a bit confusing I think.
My read is that CDD started off as an approach to use existing social capital to deliver economic benefits more effectively, meaning better choices, perhaps better monitoring, and transfer of the decision-making power to the people that mattered most. That is still what CDD often is. Call it the KDP model. Somewhere along the way an idea was appended that in fact CDD could generate social capital. This version of the CDD model gained currency and influenced the design of many programs, especially in post conflict settings. Call it the CDR model. There are certainly others. The CDR model can differ from the KDP model not just in its goals but in its allocation of funding away from projects and towards activities intended to build social capital. In a way the idea is the antithesis of the original since instead of using existing capital you end up trying to replace it. The idea that local control produces better outcomes likely resonates widely. The idea that short term exposure to CDD alters social capital and local institutions seems a much harder proposition. The evidence on CDDs, I think, largely supports this view. In other words, the KDP model might work well for its purposes; the CDR model seems not to work for its social goals.
I think this conclusion has not come out clearly enough in the recent discussion. The discussion has been confused a bit by a denial that CDD has had these social capital ambitions in the first place and perhaps a concern that the KDP model is under attack when the CDR version is challenged.
In making sense of this discussion I think five points are worth emphasizing:
Lots of CDD programs have had building social capital as a goal.
3ie claim at the top of their report “CDD programme objectives have evolved over time. The programmes in the early 1990s had more of an emphasis on poverty reduction and infrastructure building; the programmes in the late 1990s and 2000s have focused on decentralisation and improving local governance and social cohesion.”
Guggenheim claims “except for one $2.5m project in Sierra Leone that I’ve spent nearly a decade trying to convince my friend Rachel Glennerster was actually a not very well thought through outlier, to the best of my knowledge, no CDD project has had the objective of building new social capital.” But, contrary to Guggenheim’s claim, multiple programs have had new social capital as an explicit goal. (though in some cases, research has focused on social capital related outcomes even though these were not explicit goals of the programs studied)
I give lots of references to back up this claim below.
There is not that much squabbling among the experts and most reviews seem to agree with the propositions that the bottom line that CDD is good for getting stuff to people but not good for changing social structures.
Guggenheim writes: “I can tell you that one reason why people resist some of this huge push for evidence-based policy is that there is so much constant squabbling by the specialists about what the evidence actually is.”
But in my read this is an area where there is a fairly broad consensus. It is one of few areas in the political economy of development where there has been an accumulation of RCTs and multiple reviews. Of course there are differences between the reviews but I think these agree on a core bottom line. A reflection piece by Sheree Bennett and Alyoscia D’Onofrio here describes the issues around differences in priorities over “Getting stuff to people or changing how people do things.” The reviews and studies suggest that CDD might be pretty good at the former and not very good at the latter. Wong writes: “nine projects reviewed in this study reported on income poverty impacts as part of their evaluations. Out of the nine, seven had statistically significant positive impacts on household living standards and welfare….Based on the limited evidence to date, most projects have no impact on social capital, or at best mixed impacts. There is very little evidence of social capital spillover effects. For local governance, the evidence is more positive, however also mixed.”
Broadly this is also the conclusion in two other very informative reviews:
Rachel Glennerster also provides a nice account of discussions on these themes following a World Bank “smackdown” in 2013. Team IE seemed in agreement with the goal of using CDD to “get goods on the ground.”
This conclusion resonates with most of what I have seen also.1
This bottom line is important because it matters for fund allocations.
There are a lot of nuances around what sort of economic benefits arise in what sectors and so on or what sort of social impacts have been assessed where. But still the headline conclusion that CDD may be effective for the hardware but not the software (to use Casey et al’s terms) is important because a belief that these social benefits derive from CDD can lead to an allocation of funds away from project components that work to those that do not.
This whole conversation is muddied by the diversity of CDD programs and suggests we are using the wrong unit of analysis for studying impact. The term CDD gets used quite loosely; some of the interventions studied self-describe as CDR, some as participatory development and so on. Many of the programs are extremely complex with many components. One of the biggest differences is that short term interventions that are not integrated with administrative structures get lumped into longer term institutionalized mechanisms. Inferring failure of the latter from weakness of the first is not helpful. If the packets are so heterogeneous then conclusions might be more clearly stated in terms of the effects of program components — when these can be isolated.
Although there are now multiple trials of CDR versus nothing, the more fundamental proposition that community driven development performs better than alternatives has still not been as well tested.
The 3ie report says it quite clearly “it is not clear if CDD programmes are a more cost-effective delivery mechanism, especially compared with local government.” Guggenheim writes “to be constructive and useful, the most useful evaluation would have been to compare CDD programs against the next best alternative,” though he did not feel there is a need for an RCT to show it. Glennerster also made the point here. Many of the studies that exist compare a CDD disbursement mechanism coupled with often costly social interventions to a control condition rather than comparing the participatory disbursement mechanisms itself to a viable alternative.
I am surprised by Guggenheim’s claim that CDD programs have not aimed at generating social capital. It is possible (and in some ways reasonable) that he excludes CDR programs from his accounting though these are very often considered part of the CDD family. I think there are some instances where researchers have studied the social impacts of CDD programs even if these were not the primary goals of the project.2
In these cases the hypotheses examined appear largely inspired by the broader literature, which is reasonable but perhaps asking a lot of the programs.
King and Sami provide the following useful summary of four programs, with variations of social capital appearing regularly (social cohesion, local governance, collective action capacity).
Afghanistan “The key objective of NSP is to build, strengthen, and maintain Community Development Councils (CDCs) as effective institutions for local governance and social-economic development” (Islamic Republic of Afghanistan, n.d.). Beath et al. add that the program “explicitly mentions promoting gender equality as one of the program’s main goals” (Beath et al., 2012b, p. 6).
DRC “To improve the stability and quality of life for communities in eastern DRC through structured, participatory, and inclusive collective action. By establishing and strengthening participatory local governance committees [the program aims…] to improve the understanding and practice of democratic governance, improve citizens’ relationships with local government, and improve social cohesion and thereby communities’ ability to resolve conflict peacefully. The conduit to achieve these purposes will be village- and community-level projects that themselves will contribute to socio-economic rehabilitation as DRC moves into a post-conflict and development period” (Humphreys et al., 2012, p. 11). “While people’s vision of democracy’s dividends [in anticipation of Congolese elections] is in all probability unrealistic in time and scope, it is nevertheless vital that they receive tangible returns for their enduring tolerance. It is thus crucial that the post-election period deliver peace dividends…” (International Rescue Committee, 2006, p. 11).
Liberia “…the project aims to improve material welfare, build institutions and promote community cohesion by bringing together all actors within the community, including local government, civil society and private sector to identify priority problems/needs and to develop community action plans for implementation.” (International Rescue Committee, 2006b, p. 1).” This model was adopted, in part, as a strategy for using local leadership to quickly generate material improvements in people’s lives. Given the state of the government after fifteen years of civil war communities could plausibly also move more quickly than government to deliver a tangible peace dividend” (Fearon et al., 2008, pp. 2–3).
Sierra Leone “Through intensive, long term facilitation, CDD aims to strengthen local institutions, make them more democratic and inclusive of marginalized groups, and enhance the capacity of communities to engage in collective action”” (Casey et al., 2011c, p. 1).
There are others too.
These echo a general statement of the goals of CDR as envisioned by IRC https://odihpn.org/magazine/community-driven-reconstruction-a-new-strategy-for-recovery/
A side note on two seeming outliers. A Liberia study I worked on seems to contradict this conclusion. For the nuance however the full study clarifies that the positive effects we find are only inside mixed groups constructed for measurement purposes; we have doubts that these would be the groups in fact tasked to resolve collective action problems in a less controlled setting, of the form studied in Congo. The Congo study (which I was also a researcher on) seems to speak against the positive economic conclusions; but a careful read of that study makes clear that it focused specifically on social components and was timed for before major investments were made. For these reasons these studies do not, I think, speak against this broader conclusion. ↩
For instance Alexandra Adveenko and Michael Gilligan have an interesting paper on “International Interventions to Build Social Capital: Evidence from a Field Experiment in Sudan.” It is not obvious from the paper or the program documentation that this was an aim however: (“The objective of the Project is to meet urgent community-driven recovery and development needs in the war-affected and underdeveloped areas of North Sudan including the Three Areas by providing social and economic services and infrastructure.” here and here. Tu Chi Nguyen and Matthias Rieger have an interesting paper on “Community-driven Development and Social Capital: Evidence from Morocco” where building social capital was not obviously an aim of the program, at least from the description they give here. ↩