Effects of economic and social incentives on bureaucratic quality

Experimental Evidence from Sierra Leone

Maarten Voors, Macartan Humphreys, Salif Jaiteh, Niccolo Mereggi, Carlo Prato, Peter van der Windt

Outline

  • Motivation
  • Model
  • Design
  • Results
  • Structural model

Motivation

Motivation

A challenge:

  • Quality of service provision is low in many developing countries

  • Recruiting and incentivizing service providers is difficult

A case:

  • Agricultural extension workers
  • The 2013–2016 Ebola virus epidemic in West Africa
  • Caught many by surprise, no early warning. Need for stronger decentralized bureaucracy.
  • \(>\) 7/10 emerging infectious diseases spill over from animals

Motivation

Promising work on improving service provision:

  1. Recruitment

  2. Community monitoring

  3. Financial incentives

We show:

  • evidence 2. and 3. matter with limited crowding out.
  • no good news on 1.
  • we use a structural model to explore why and when these might be more effective.

Past Work on Recruitment

Theoretically:

Empirically:

Past Work on Community Monitoring

  • View that bottom-up pressure by empowered citizens is key to improved service delivery

  • Overall mixed effects:

    • In health: Björkman and Svensson (2009); Björkman and Svensson (2017); Raffler, Posner, and Parkerson (2020); Christensen et al. (2021)

    • In education: Banerjee et al. (2012); Barr et al. (2012); Pradhan et al (2014); Andrabi et al. (2017)

Past Work on Financial Incentives

By and large a success:

  • Education: Muralidharan and Sundararaman (2011), Kahn et al (2014), Glewwe et al (2010), De Ree et al (2010), Leaver et al (2021)

  • Health status: Miller et al (2012) Mohanan et al. (2021)

  • Service delivery: Gertler and Vermeersch (2013), Desserano et al (2022)

  • Service provider inputs: teachers (Duflo et al, 2012, Leaver et al 2021), nurse attendance Banerjee and Duflo (2006)

The policy questions

  1. Are community monitoring mechanisms more or less effective than performance-based payment schemes?
  1. Are there interaction effects between recruitment strategies and incentive mechanisms?
  • E.g. community monitoring may be less effective when service providers are selected by the state (and so, less embedded)

Design

We study these questions in the context of a randomized agricultural intervention in Sierra Leone.

Core design elements (more on context to come)

treatment label
T1 Agent is selected by community (T1 = 1) or by chief (T1 = 0)
T2 Community can impose social sanctions on agent (T2 = 1) or not (T2=0)
T3 Agents receive pay for performance (T3 = 1) or not (T3=0)

Model

A worker has type \(\theta = (\eta,\mu)\) where:

  • \(\eta \in \{ \underline{\eta} , 1 \}\) denotes embeddedness
  • \(\mu \in \{ 0 ,\overline{\mu} \}\) denotes intrinsic motivation

Drawn according to:

\({\mu}{\backslash}\eta\) \(\underline{\eta}\) 1
0 \(\frac{1+ \rho}{4}\) \(\frac{1- \rho}{4}\)
\(\overline{\mu}\) \(\frac{1- \rho}{4}\) \(\frac{1+ \rho}{4}\)

So \(\rho\) governs the correlation between embeddedness and motivation: embedded people are possibly more effective, but are they more likely to be highly motivated?

T1 and type

  • Under chief selection (\(T_1=0\)) the worker is a random draw from the candidate pool.
  • Under villager selection (\(T_1=1\)), the worker is drawn from the pool with high embeddedness.

So paramount chief is more likely to select workers with high intrinsic motivation if and only if \(\rho < 0\).

Performance

  • Workers choose effort \(e\)

  • Performance depends on effort and embeddedness: \[\pi = e \eta\]

  • Stochastic performance signal \(\sigma\), which is high (\(\sigma=1\)) with probability \(\pi\) or low (\(\sigma=0\)) with probability \(1-\pi\)

  • Cost of effort: \(c(e) = \frac{e^2}{2\kappa}\)

Villager (Monitoring)

  • A representative villager selects a social sanctioning plan, which depend on the performance signal \(\sigma\): \[s=(s(0),s(1)) \in [-T_2, T_2]^2.\]

Interpretable as commitments to reward (\(s(\sigma)<0\)) or punish (\(s(\sigma)>0\)) performance.

Villager (Monitoring)

Punishment is possibly psychologically costly, according to \(\gamma\).

\[u(\pi, s) = \pi - \gamma s .\]

  • \(\zeta\) is probability punishment is not costly for the villager

CAHW utility

Under pay-for-performance (\(T_3=1\)) and high performance signal (\(\sigma=1\)), the worker receives a payment normalized to 1

In addition:

  • \(\phi\): sensitivity to social sanctions
  • \(\mu\): intrinsic benefits from performing well

Putting it all together utility is:

\[w(e;s,\sigma;\theta = (\mu, \eta)) = \sigma (T_3) + \pi\mu - \phi s(\sigma) T_2 - \frac{e^2}{2 \kappa},\]

where \(\pi = e\eta\)

Timeline

The sequence of events is as follows:

  1. worker is selected by either village (\(T_1=1\)) or chief (\(T_1=0\)).
  2. villagers and worker learn whether the latter is subject to community monitoring (\(T_2=1\)) and/or P4P (\(T_3=1\)).
  3. if \(T_2=1\), villager with guilt parameter \(\gamma\) commits to a conditional plan of social sanctioning \(\{s(\sigma)\}_{\sigma \in \{0,1\}}\).
  4. The worker chooses effort \(e\).
  5. The worker’s performance signal \(\sigma\) is realized.
  6. Payoffs are realized and the game ends.

Solution concept

An equilibrium is a pair of strategies \(\{ s,e(s;\theta) \}\) such that

  1. effort is individually rational for each type given the sanctioning strategy
  2. the sanctioning strategy maximizes the villager’s expected payoff given anticipated effect on the workers’ effort \(\hat{e}(s;\theta)\), and
  3. expectations are correct (\(\hat{e}(s;\theta)=e(s;\theta)\) for each \(\theta\)) :

\[e(s;\theta) \in \arg \max_{e \in [0,1]} \mathbb{E}_{\pi } \Big\{ w(e,s,\sigma;\theta) \Big\}\] \[s \in \arg \max_{\{ (s(0),s(1)) \in [-T_2,T_2]^2 \}} \mathbb{E}_{\theta,\pi} \Big\{ u(\pi,s) \mid \hat{e}(s;\theta) \Big\}\]

Results

  • Sanctioning.
    • following a high signal: always praise the worker
    • following a low signal: choose maximal shaming \(s(0)=1\) if susceptibility to guilt is low (\(\gamma=0\)) and minimal shaming \(s(0)=-1\) otherwise.
  • Effort (depending on citizen guilt):

\[e(s^*;\gamma,\theta) = \left\{\begin{array} .\kappa \eta (\mu + T_3 + T_2 2 \phi) & \mbox{if} \quad \gamma =0 \\ \kappa \eta (\mu + T_3) \quad & \mbox{otherwise}. \end{array}\right.\]

Results

Full closed form solution for all treatment conditions:

\((T_2,T_3) | T_1\) \(T_1=0\) \(T_1=1\)
\(\frac{\pi^*(T_1,0,0)}{\kappa}\) \(\frac{\overline{\mu}}{2} \frac{1 + \rho + \underline{\eta}^2(1-\rho)}{2}\) \(\frac{\overline{\mu}}{2} (1 + \rho)\)
\(\frac{\pi^*(T_1,0,1)}{\kappa}\) \(\frac{\overline{\mu}}{2} \frac{1 + \rho + \underline{\eta}^2(1-\rho)}{2}+ \frac{1+ \underline{\eta}^2}{2}\) \(\frac{\overline{\mu}}{2} (1 + \rho)\) \(+1\)
\(\frac{\pi^*(T_1,1,0)}{\kappa}\) \(\frac{\overline{\mu}}{2} \frac{1 + \rho + \underline{\eta}^2(1-\rho)}{2} +(1+ \underline{\eta}^2) \phi \zeta\) \(\frac{\overline{\mu}}{2} (1 + \rho) +2 \phi \zeta\)
\(\frac{\pi^*(T_1,1,1)}{\kappa}\) \(\frac{\overline{\mu}}{2} \frac{1 + \rho + \underline{\eta}^2(1-\rho)}{2} +(1+ \underline{\eta}^2) \phi \zeta + \frac{1+ \underline{\eta}^2}{2}\) \(\frac{\overline{\mu}}{2} (1 + \rho) +2 \phi \zeta +1\)

Hypotheses inspired by model

Hypothesis 1: Village selection increases performance.

  • Explanation: Requires gains from embeddedness not crowded-out by any reductions in intrinsic motivation

Hypothesis 2: Community monitoring increases performance.

  • Explanation: Community monitoring (T2=1) incentivizes worker performance through the associated probability of being publicly shamed for poor performance.

Hypothesis 3: Pay-for-performance increases performance.

  • Explanation: Pay for performance (T3=1) increases performance due to the probability of receiving a monetary reward.

Hypotheses

Hypothesis 4: There is a positive interaction between village selection and pay-for-performance.

  • Explanation: More productive workers (because embedded) are more responsive to extrinsic incentives (T3=1).

Hypothesis 5: There is a positive interaction between village selection and community monitoring.

  • Explanation: More productive workers (i.e. those with higher embeddedness, T1=1) as it is easier (less risky!) for villagers to commit to sanctioning (as long as embeddedness does not crowd-out intrinsic motivation too much).

Hypotheses

  • [Hypothesis 6: There is a positive interaction between community monitoring and pay-for-performance.

Explanation: The expected improvement in performance driven by the monetary incentive (T3=1) makes it easier for the village to commit to punish low performance (T2=1).]

Design

Context: Sierra Leone

Poor service provision

  • In 2020, U5 mortality is 108/1,000 live births (45/48)

Threat of zoonosis is high and expected to increase

  • Animals are essential for livelihoods (90% keep animals)

  • Low information and bad animal husbandry practices

  • Animal diseases are highly contagious

  • Impacts on households are enormous and largely preventable

Context

  • In the wake of the Ebola epidemic, together with the GoSL, we implement a “One Health” intervention, addressing animal and human health

  • 360 villages across 7 chiefdoms in Kono district

Context

  • Within the One Health program: we randomize recruitment and incentives across villages

One Health Program

  • Co-designed with GoSL’s MOH and MAF

  • Implemented from July 2017 to July 2019

  • Recruitment of Community Animal Health Worker: CAHW

  • 21-day training, implemented by veterinarians

  • Tool kit and drugs for treatment of basic animal diseases

  • Creation One Health platform

  • CAHW:

    • provides basic animal health services
    • demonstrates and provides information on best practices
    • reports suspected disease symptoms to MAF supervisors

One Health Program: 21-day training

One Health Program: Animal disease reporting form

Results

One Health Program: Results

  • March and April 2020: Household endline survey (344/363 villages)
  • Large increases in animal stocks (22%)

One Health Program

  • Strong enthusiasm for the program

    • reporting throughout the period and beyond

  • No impact on human health, but impacts on intermediary outcomes

  • From planner’s perspective it matters if we can improve CAHW performance:

    1. Recruitment

    2. Community monitoring

    3. Pay-for-performance

Inside One Health

Recruitment: Community selection versus “business as usual”

  • Community selection: MAF facilitators hold village meetings to nominate and select the CAHW candidate

  • Business as usual: CAHWs are recruited by state authorities in collaboration with Paramount Chiefs

  • Minimal ability requirements (tests)

  • Key design element: double listing

    • Obtain nominees from both Paramount Chief and community
    • Allows us to have the counter-factual

Recruitment and Incentives: Treatments

Community Monitoring

  • CAHW and community develop joint action plans

  • Interface meetings between ministry staff and community:

    • Two times: after 6 months and 1.5 years
    • CAHW performance revealed (relative and overall)
    • Allows community members to reward or impose social sanctions

Recruitment and Incentives: Treatments

Pay-for-Performance

  • CAHWs are paid based on their rank (across three groups)
  • Two times: after 6 months and 1.5 years

Note: rank ensure revenue neutrality but introduces small risk of non-independence / collusion

Recruitment and Incentives: Treatment Implementation

  • Community monitoring and pay-for-performance implemented as designed

  • The recruitment treatment not so much

    • Only in 100 villages do we have true double-listing (in many other cases either same person listed twice or no one listed by one party)

    • Implication: smaller sample and non-experimental interactions

Realized distribution of cases

Characteristic No financial incentives, N = 145 Pay for performance, N = 144
N Chief selects, N = 881 Community selects, N = 571 N Chief selects, N = 871 Community selects, N = 571
T2 145

144

    No community sanctions
44 (50%) 29 (51%)
44 (51%) 28 (49%)
    Community sanctioning
44 (50%) 28 (49%)
43 (49%) 29 (51%)
1 n (%)
  • Small amount of attrition where candidates not identified (pre-randomization)

Recruitment and Incentives: Manipulation Check 1

  • Does community selection impact CAHW type?
  • Only modest differences from CAHW survey

CAHWs baseline survey

Recruitment and Incentives: Manipulation Check

  • Does community selection impact CAHW type?
  • More differences from HH survey

household baseline survey

Recruitment and Incentives: Manipulation Check

  • Community monitoring and sanctioning

household baseline survey

Recruitment and Incentives: Manipulation Check

Recruitment and Incentives: Manipulation Check

Primary Outcome: Performance

Standard deviation of index in T2, T3 control condition: 0.19

Main design based results

Statistical models
  T1 T2 T3 All All (FE)
T1 (selection) 0.01     0.01  
  (0.03)     (0.02)  
T2 (community monitoring)   0.05*   0.05* 0.05*
    (0.02)   (0.02) (0.02)
T3 (pay for performance)     0.06* 0.06* 0.06*
      (0.02) (0.02) (0.02)
T1 x T2       -0.06 -0.06
        (0.05) (0.05)
T1 x T3       -0.04 -0.04
        (0.05) (0.05)
T2 x T3       -0.03 -0.03
        (0.05) (0.05)
T1 x T2 x T3       0.01 0.01
        (0.10) (0.10)
Two candidates (T1)       0.06*  
        (0.02)  
Constant       0.50***  
        (0.02)  
R2 0.56 0.26 0.27 0.06 0.29
Adj. R2 0.07 0.00 0.01 0.04 0.02
Num. obs. 287 287 287 287 287
RMSE 0.20 0.21 0.21 0.21 0.21
***p < 0.001; **p < 0.01; *p < 0.05

Alternative specifcations

Statistical models
  T1 (sub) T23_sat
T1 (selection) 0.01  
  (0.04)  
T2 (community monitoring)   0.05*
    (0.02)
T3 (pay for performance)   0.06**
    (0.02)
T2 x T3   -0.03
    (0.05)
R2 0.54 0.28
Adj. R2 0.03 0.03
Num. obs. 100 287
RMSE 0.20 0.21
***p < 0.001; **p < 0.01; *p < 0.05
  • Restricting T1 analysis to 100 pure randomizations
  • T2 x T3 analyses without T1 and with randomization strata fixed effects

Subcomponents

Statistical models
  reports animals findable all
T2 (community monitoring) 0.05 0.04 0.06 0.05*
  (0.03) (0.03) (0.03) (0.02)
T3 (pay for performance) 0.08** 0.06* 0.04 0.06**
  (0.03) (0.03) (0.03) (0.02)
T2 x T3 -0.04 -0.03 -0.02 -0.03
  (0.06) (0.06) (0.06) (0.05)
R2 0.29 0.25 0.25 0.28
Adj. R2 0.04 -0.01 -0.02 0.03
Num. obs. 287 287 287 287
RMSE 0.26 0.27 0.26 0.21
***p < 0.001; **p < 0.01; *p < 0.05

Summary

  • Evidence for both community monitoring and pay for performance
  • No evidence for selection
  • Noisy evidence for interactions: signs in wrong direction in two cases

Some of these results challenge the model more than others:

  • No effect for T1 can be consistent with the model
  • But we really do we expect from H 4: that more productive workers are more responsive to extrinsic incentives (T3=1).

Structural model

  • Our model provides predictions for effort as a function of parameters and treatments
  • We integrate these into a model with chiefdom level fixed effects and random errors
  • We then estimate model parameters using stan

\[e_{ij} = \alpha_{j} + \kappa \times p(T_1, T_2, T_3 | \bar{\mu}, \underline{\eta}, \rho, \nu, \phi, \zeta) + \epsilon_i\]

Parameter estimates

Posterior distributions on model parameters: movements not dramatic

Model based estimates of treatment effects

Posterior distributions on causal effects: close but note forced direction on interactions

Experiments: T1

Where and when might we expect stronger selection effects?

When might selection matter?

Experiments: T2

The Björkman and Svensson (2009) effect.

When might community sanctions work?

Experiments: T3

The payment effect.

When is payment most effective?

Discussion & Conclusion

Program

  • Successful intervention: No increase in human health, but increase in good animal and human health practices, and animal stock

Institutions

  • Evidence for two distinct institutional strategies
    • Consider now “invisible” costs
  • No evidence here that selection changes outcomes
    • But plausibly case specific issues

Discussion & Conclusion

Explanation

  • Highlights power of combining model and data for understanding lessons
  • But also risks
  • Current work incorporates measures of embededness and motivation
  • Perhaps set rival models in competition?

End

Extra slides

One Health Program

Created new positions in villages

One Health Program

Sustained engagement

Quality of Reporting

  • Surveillance performance driven by events, effort and ability

  • Records contain both Type 1 and Type 2 errors

  • Both matter for containing pandemics

    • triggering responses to false events
    • failing to report events

Quality of Reporting

We measure the quality of CAHW reporting

  • Practical: Central location to assess CAHW

    • Animal exam: CAHWs observe and diagnose sick animals, and record
    • Form review: Assess 5 (random) recently submitted forms

  • Village visit: MAF representatives visit the CAHW’s village

    • Sick animal observation: Verify if sick animals had been reported
    • Form follow-up: Verify reported sick animals

Results: Quality of Reporting

Also: there is strong relationship between surveillance quality and performance

Quality

Results: Animal Husbandry Practices

- Data on disease control, response, knowledge, preventative practices

Parameters

Symbol Name Interpretation
\(\mu \in \{0, \overline{\mu}\}\) (Intrinsic) motivation Strength of motivation
\(\eta \in \{\underline{\eta},1\}\) Embeddedness Strength of a worker’s ties to the community
\(\theta = (\mu,\eta)\) Worker’s type
\(\rho \in [-1,1]\) Type correlation \(Corr(\mu,\eta)\)
\(\phi\) Social sensitivity The worker’s susceptibility to social sanctioning
\(\gamma \in \{0,\overline{\gamma}\}\) Guilt The villager’s psychological cost of social sanctioning
\(\zeta \in [0,1]\) Strength of village-level norms Probability of low-guilt villager
\(\kappa\) Productivity Effort elasticity to wage
\(e \in [0,1]\) Effort How hard the worker works
\(\pi \in [0,1]\) Performance Outcome of worker’s activity (\(\pi= e \eta\))
\(\sigma \in \{0,1\}\) Performance signal Public signal of performance (\(\Pr(\sigma=1)=\pi\))
\(s(\sigma) \in [-1,1]\) Social sanctioning plan Villager commits to it before observing \(\sigma\)
Ashraf, Nava, Oriana Bandiera, and Scott Lee. 2020. “Losing Prosociality in the Quest for Talent? Sorting, Selection, and Productivity in the Delivery of Public Services.” American Economic Review 110: 1355–94.
Banerjee, Abhijit, Rukmini Banerji, Esther Duflo, Rachel Glennerster, and Sharmila Khemani Barr. 2012. “Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India.” American Economic Journal: Economic Policy 2 (1): 1–3.
Banerjee, Abhijit, and Esther Duflo. 2006. “Addressing Absence.” Journal of Economic Perspectives 20 (1): 117–32.
Björkman, Martina, and Jakob Svensson. 2009. “Power to the People: Evidence from a Randomized Field Experiment on Community-Based Monitoring in Uganda.” The Quarterly Journal of Economics 124 (2): 735–69.
Christensen, D, O Dube, J Haushofer, B Siddiqi, and M Voors. 2021. “Healthcare Delivery During Crises: Experimental Evidence from Sierra Leone’s Ebola Outbreak.” Quarterly Journal of Economics 136 (2): 1145–98.
Dal Bó, Ernesto, Frederico Finan, and Martı́n A Rossi. 2013. “Strengthening State Capabilities: The Role of Financial Incentives in the Call to Public Service.” The Quarterly Journal of Economics 128 (3): 1169–1218.
Deserranno, E. 2017. “Financial Incentives as Signals: Experimental Evidence from the Recruitment of Health Workers.”
Francois, Patrick. 2000. “‘Public Service Motivation’as an Argument for Government Provision.” Journal of Public Economics 78 (3): 275–99.
Gertler, Paul, and Christel Vermeersch. 2013. “Using Performance Incentives to Improve Medical Care Productivity and Health Outcomes.” National Bureau of Economic Research.
Mohanan, Manoj, Katherine Donato, Grant Miller, Yulya Truskinovsky, and Marcos Vera-Hernández. 2021. “Different Strokes for Different Folks: Experimental Evidence on the Effectiveness of Input and Output Incentive Contracts for Health Care Providers with Different Levels of Skills.” American Economic Journal: Applied Economics.
Muralidharan, Karthik, and Venkatesh Sundararaman. 2011. “Teacher Performance Pay: Experimental Evidence from India.” Journal of Political Economy 119 (1): 39–77.
Raffler, Pia, Daniel Posner, and Donald Parkerson. 2020. “Can Citizen Pressure Be Induced to Improve Public Service Provision?”