Registration, Transparency, and Publication
Paper Session
Friday, Jan. 3, 2025 10:15 AM - 12:15 PM (PST)
- Chair: Lars Vilhuber, Cornell University
Measuring Publication Bias With Standardized Reporting in Economics
Abstract
We developed an approach to standardize and record the hypotheses of studies registered in the AEA RCT Registry. We apply this approach to the registration of 300 studies registered in the AEA Registry between 2015 and 2017. This corresponds to thousands of hypotheses, and we proceed to search for the corresponding results in the published literature, publicly available working papers, and other research reports. This is among the largest follow-up studies in economics aimed at measuring publication bias and selective reporting, and is novel in focusing on results at the research hypothesis level. Preliminary analyses show substantial heterogeneity in the number of registered hypothesis, many of which we were unable to find in the published literature up to nine years later. This result suggests that preregistration alone may not solve the problem of selective reporting. In addition to measuring publication bias, we also develop an innovative approach to standardized reporting of research results in economics – a practice already widely used in other fields, including medicine and public health – that could be of interest to journals, funders, and professional associations.Improving Results Reporting in RCT Registries: Evidence from Interviews with Registrants
Abstract
The complete and accurate registration of trials is a critical component of credible science. Comprehensive registries provide a record of trials conducted in a given field – regardless of the studies’ publication – thereby helping combat publication bias. Thanks in part to requirements by journals and funders, the social sciences have seen a marked increase in registration over time.Yet the benefits of registries cannot be realized if researchers know which trials were started but have incomplete information on post-trial outcomes (trial completion, results, etc.). Failure to update registrations with post-trial information is a challenge in both medicine, where government-sponsored scientists comply less than 50% of the time with results-reporting requirements, and the social sciences. In 2021, only 29% of registrations in the AEA RCT Registry had any post-trial fields completed one year post-trial end date.
This study explores barriers to updating AEA RCT Registry trials and interventions to increase post-trial reporting. We conducted qualitative interviews with 48 registrants randomly chosen from 5,752 trials registered before May, 2022 and at least one year past their trial end date. Sampled registrants completed a self-guided survey and follow-up semi-structured interview covering attitudes towards research transparency, registration, updating, and the design of messages to increase updates. Results informed the design of an RCT that aimed to identify methods to increase registry post-trial update rates.
We identify preferences for messaging about publication bias and nudges with pre-filled information and fields to be completed. Respondents also identify journal requirements and peer norms as the most likely incentives to increase reporting. Absence of perceived benefits and of staff/own time are found to be primary constraints to reporting. Respondents identify the importance of building a culture of transparency, but also see a gap between personal importance and the perceived importance to the economics field.
Discussant(s)
Taisuke Imai
,
Osaka University
Jonathan Alan Libgober
,
University of Southern California
Sarah Kopper
,
J-PAL/MIT
JEL Classifications
- A1 - General Economics
- B4 - Economic Methodology