American Economic Review
ISSN 0002-8282 (Print) | ISSN 1944-7981 (Online)
Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics
American Economic Review
vol. 110,
no. 11, November 2020
(pp. 3634–60)
Abstract
The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal "revise and resubmit" process mitigates the problem; (iii) things are improving through time.Citation
Brodeur, Abel, Nikolai Cook, and Anthony Heyes. 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics." American Economic Review, 110 (11): 3634–60. DOI: 10.1257/aer.20190687Additional Materials
JEL Classification
- A14 Sociology of Economics
- C12 Hypothesis Testing: General
- C52 Model Evaluation, Validation, and Selection