This website uses cookies.

By clicking the "Accept" button or continuing to browse our site, you agree to first-party and session-only cookies being stored on your device to enhance site navigation and analyze site performance and traffic. For more information on our use of cookies, please see our Privacy Policy.

  1. Home
  2. Journals
  3. Journal of Economic Perspectives
  4. Summer 2015
  5. Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible

Journal of Economic Perspectives

ISSN 0895-3309 (Print) | ISSN 1944-7965 (Online)

Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible

  • Lucas C. Coffman
  • Muriel Niederle
Journal of Economic Perspectives
vol. 29, no. 3, Summer 2015
(pp. 81–98)
Download Full Text PDF
(Complimentary)

Abstract

The social sciences—including economics—have long called for transparency in research to counter threats to producing robust and replicable results. In this paper, we discuss the pros and cons of three of the more prominent proposed approaches: pre-analysis plans, hypothesis registries, and replications. They have been primarily discussed for experimental research, both in the field including randomized control trials and the laboratory, so we focus on these areas. A pre-analysis plan is a credibly fixed plan of how a researcher will collect and analyze data, which is submitted before a project begins. Though pre-analysis plans have been lauded in the popular press and across the social sciences, we will argue that enthusiasm for pre-analysis plans should be tempered for several reasons. Hypothesis registries are a database of all projects attempted; the goal of this promising mechanism is to alleviate the "file drawer problem," which is that statistically significant results are more likely to be published, while other results are consigned to the researcher's "file drawer." Finally, we evaluate the efficacy of replications. We argue that even with modest amounts of researcher bias—either replication attempts bent on proving or disproving the published work, or poor replication attempts—replications correct even the most inaccurate beliefs within three to five replications. We offer practical proposals for how to increase the incentives for researchers to carry out replications.

Citation

Coffman, Lucas C., and Muriel Niederle. 2015. "Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible." Journal of Economic Perspectives, 29 (3): 81–98. DOI: 10.1257/jep.29.3.81

JEL Classification

  • C38 Multiple or Simultaneous Equation Models: Classification Methods; Cluster Analysis; Principal Components; Factor Models
There are no comments for this article.
Login to Comment