Experiments

5.3 Innovative designs

Randomization of

• encouragements to gain citizenship (New York)
• where police are stationed (India)
• how government tax collectors get paid (do they get a share?) (Pakistan)
• the voting rules for determining how decisions get made (Afghanistan)
• of populations to peacekeepers (Liberia)
• of ex-combatants out of their networks (Indonesia)
• students to ethnically homogeneous or ethnically diverse schools (Ethiopia)

6.1 Constraint: Is it ethical to manipulate subjects for research purposes?

• There is no foundationless answer to this question.

• Belmont principles commonly used for guidance:

  1. Respect for persons
  2. Beneficence
  3. Justice

• Unfortunately, operationalizing these requires further ethical theories. (1) is often operationalized by informed consent (a very liberal idea). (2) and (3) sometimes by more utiliarian principles

• The major focus on (1) by IRBs might follow from the view that if subjects consent, then they endorse the ethical calculations made for 2 and 3 — they think that it is good and fair.

• Trickiness: can a study be good or fair because of implications for non-subjects?

6.2 Is it ethical to manipulate subjects for research purposes?

• Many (many) field experiments have nothing like informed consent.

• For example, whether the government builds a school in your village, whether an ad appears on your favorite radio show, and so on.

• Consider three cases:

  1. You work with a nonprofit to post (true?) posters about the crimes of politicians on billboards to see effects on voters
  2. You hire confederates to offer bribes to police officers to see if they are more likely to bend the law for coethnics
  3. The British government asks you to work on figuring out how the use of water cannons helps stop rioters rioting

6.3 Is it ethical to manipulate subjects for research purposes?

• Consider three cases:

  • You work with a nonprofit to post (true?) posters about the crimes of politicians on billboards to see effects on voters
  • You hire confederates to offer bribes to police officers to see if they are more likely to bend the law for coethnics
  • The British government asks you to work on figuring out how the use of water cannons helps stop rioters rioting

• In all cases, there is no consent given by subjects.

• In 2 and 3, the treatment is possibly harmful for subjects, and the results might also be harmful. But even in case 1, there could be major unintended harmful consequences.

• In cases 1 and 3, however, the “intervention” is within the sphere of normal activities for the implementer.

6.4 Constraint: Is it ethical to manipulate subjects for research purposes?

• Sometimes it is possible to use this point of difference to make a “spheres of ethics” argument for “embedded experimentation.”

• Spheres of Ethics Argument: Experimental research that involves manipulations that are not normally appropriate for researchers may nevertheless be ethical if:

  • Researchers and implementers agree on a division of responsibility where implementers take on responsibility for actions
  • Implementers have legitimacy to make these decisions within the sphere of the intervention
  • Implementers are indeed materially independent of researchers (no swapping hats)

6.5 Constraint: Is it ethical to manipulate subjects for research purposes?

• Difficulty with this argument:
  • Question begging: How to determine the legitimacy of the implementer? (Can we rule out Nazi doctors?)

Otherwise keep focus on consent and desist if this is not possible

6.7 APSA Ethics: General [Abbr]

1. Political science researchers should respect autonomy, consider the wellbeing of participants and other people affected by their research, and be open about the ethical issues they face.

2. Political science researchers have an individual responsibility to consider the ethics of their research related activities and cannot outsource ethical reflection to review boards, other institutional bodies, or regulatory agencies.

3. These principles describe the standards of conduct and reflexive openness that are expected of political science researchers. … [In cases of reasonable deviations], researchers should acknowledge and justify deviations in scholarly publications and presentations of their work.

6.8 APSA Ethics: Power

4. When designing and conducting research, political scientists should be aware of power differentials between researcher and researched, and the ways in which such power differentials can affect the voluntariness of consent and the evaluation of risk and benefit.

1. especially with low-power or vulnerable participants
2. covert or deceptive research with more than minimal harm may sometimes be ethically permissible in research with powerful parties

6.9 APSA Ethics: Consent

5. Political science researchers should generally seek informed consent from individuals who are directly engaged by the research process, especially if research involves more than minimal risk of harm or if it is plausible to expect that engaged individuals would withhold consent if consent were sought.

1. informed and voluntary
2. continuing consent
3. observation of public behavior does not usually invoke this principle of consent.
4. forgoing perhaps if minimal risk, or seeking consent increases the risks for participants
5. discuss consent in publications and presentations

6.11 APSA Ethics: Harm and Trauma

7. Political science researchers should consider the harms associated with their research.

• Researchers should generally avoid harm when possible, minimize harm when avoidance is not possible, and not conduct research when harm is excessive.

• do not limit concern to physical and psychological risks to the participant.

8. Political science researchers should anticipate and protect individual participants from trauma stemming from participation in research.

6.12 APSA Ethics: Confidentiality

9. Political science researchers should generally keep the identities of research participants confidential; when circumstances require, researchers should adopt the higher standard of ensuring anonymity.

1. Researchers should clearly communicate assurances of confidentiality / anonymity
2. If confidentiality bit provided, communicate this and justify c./d. consider risks at all stages
3. Researchers who determine that it would be unethical to share materials derived from human subjects should be prepared to justify their decision to journal editors, to reviewers, etc

6.13 APSA Ethics: Impact

10. Political science researchers conducting studies on political processes should consider the broader social impacts of the research process as well as the impact on the experience of individuals directly engaged by the research. In general, political science researchers should not compromise the integrity of political processes for research purposes without the consent of individuals that are directly engaged by the research process.

1. cases in which research that produces impacts on political processes without consent of individuals directly engaged by the research might be appropriate. [examples]

2. Studies of interventions by third parties do not usually invoke this principle on impact. [details]

3. This principle is not intended to discourage any form of political engagement by political scientists in their non-research activities or private lives.

4. researchers should report likely impacts

6.15 APSA Ethics: Shared Responsibility

12. The responsibility to promote ethical research goes beyond the individual researcher or research team.

1. Mentors, advisors, dissertation committee members, and instructors

2. Graduate programs in political science should include ethics instruction in their formal and informal graduate curricula;

3. Editors and reviewers should encourage researchers to be open about the ethical decisions …

4. Journals, departments, and associations should incorporate ethical commitments into their mission, bylaws, instruction, practices, and procedures.

8.2 Contentious Issues

Experimental researchers are deeply engaged in the movement towards more transparency social science research.

Contentious issues (mostly):

• Data. How soon should you make your data available? My view: as soon as possibe. Along with working papers and before publication. Before it affects policy in any case. Own the ideas not the data.

  • Hard core: no citation without (analytic) replication. Perhaps. Non-replicable results should not be influencing policy.

• Where should you make your data available? Dataverse is focal for political science. Not personal website (mea culpa)

• What data should you make available? Disagreement is over how raw your data should be. My view: as raw as you can but at least post cleaning and pre-manipulation.

8.3 Open science checklist

Experimental researchers are deeply engaged in the movement towards more transparency social science research.

• Should you register?: Hard to find reasons against. But case strongest in testing phase rather than exploratory phase.

• Registration: When should you register? My view: Before treatment assignment. (Not just before analysis, mea culpa)

• Registration: Should you deviate from a preanalysis plan if you change your mind about optimal estimation strategies. My view: Yes, but make the case and describe both sets of results.

8.18 Myth: Concerns about fishing presuppose researcher dishonesty

• Fishing can happen in very subtle ways, and may seem natural and justifiable.

• Example:

  • I am interested in whether more democratic institutions result in better educational outcomes.
  • I examine the relationship between institutions and literacy and between institutions and school attendance.
  • The attendance measure is significant and the literacy one is not. Puzzled, I look more carefully at the literacy measure and see various outliers and indications of measurement error. As I think more I realize too that literacy is a slow moving variable and may not be the best measure anyhow. I move forward and start to analyze the attendance measure only, perhaps conducting new tests, albeit with the same data.

8.22 Myth: Registration only makes sense for experimental studies, not for observational studies

• The key distinction is between prospective and retrospective studies.

• Not between experimental and observational studies.

• A reason (from the medical literature) why registration is especially important for experiments: because you owe it to subjects

• A reason why registration is less important for experiments: because it is more likely that the intended analysis is implied by the design in an experimental study. Researcher degrees of freedom may be greatest for observational qualitative analyses.