Research integrity

Fishing and preregistration

  • Even though everyone knows it is wrong, it is common practice in empirical social science to select what results to report only after analysing data. This practice of “data fishing” can result in enormous bias and an unreliable body of published research. Peter van der Windt and Raul Sanchez de la Sierra and I did a paper on the possible merits of a non-binding but comprehensive registration scheme. We describe the scope for bias under weak registration systems and discuss likely effects of registration on the sort of research that gets produced and reported here.

  • Here is a little app I made that illustrated the problem with data fishing. The point of the app is not to show that generating real random numbers is hard; it is to show that for some problems you can always choose your tests to get the result you want.

  • Al Fang and Grant Gordon and I have another working paper on the effects of registration in medical sciences. We don’t find much evidence that the way it is done there makes a difference. (Maybe we would have found more if we didn’t preregister) See here.

  • See here for a paper drafted jointly with representatives from three sections at APSA to think through what a APSA sponsored political sciences registry would look like.

  • 2013 blog post Monkey Business