Journal of Economic Perspectives
ISSN 0895-3309 (Print) | ISSN 1944-7965 (Online)
Promises and Perils of Pre-analysis Plans
Journal of Economic Perspectives
vol. 29,
no. 3, Summer 2015
(pp. 61–80)
(Complimentary)
Abstract
The purpose of this paper is to help think through the advantages and costs of rigorous pre-specification of statistical analysis plans in economics. A pre-analysis plan pre-specifies in a precise way the analysis to be run before examining the data. A researcher can specify variables, data cleaning procedures, regression specifications, and so on. If the regressions are pre-specified in advance and researchers are required to report all the results they pre-specify, data-mining problems are greatly reduced. I begin by laying out the basics of what a statistical analysis plan actually contains so those researchers unfamiliar with it can better understand how it is done. In so doing, I have drawn both on standards used in clinical trials, which are clearly specified by the Food and Drug Administration, as well as my own practical experience from writing these plans in economics contexts. I then lay out some of the advantages of pre-specified analysis plans, both for the scientific community as a whole and also for the researcher. I also explore some of the limitations and costs of such plans. I then review a few pieces of evidence that suggest that, in many contexts, the benefits of using pre-specified analysis plans may not be as high as one might have expected initially. For the most part, I will focus on the relatively narrow issue of pre-analysis for randomized controlled trials.Citation
Olken, Benjamin A. 2015. "Promises and Perils of Pre-analysis Plans." Journal of Economic Perspectives, 29 (3): 61–80. DOI: 10.1257/jep.29.3.61Additional Materials
JEL Classification
- C38 Multiple or Simultaneous Equation Models: Classification Methods; Cluster Analysis; Principal Components; Factor Models
There are no comments for this article.
Login to Comment