Research Highlights Article
February 11, 2022
Publishing the most valuable findings
What are the tradeoffs journal editors must weigh when choosing which papers to accept?
Journal editors must weigh the value of a paper's findings when choosing what to publish.
Bigstock
The publishing decisions of journal editors have recently come under intense scrutiny. Critics are concerned with publication bias toward research with statistically significant results, many of which may be misleading.
But journal space is limited. Editors have to decide which research deserves to be included. While there are some proposals to accept scientific papers prior to knowing the results or remove gatekeepers altogether, these models may lead to drowning out important research.
In a paper in the American Economic Journal: Microeconomics, authors Alexander Frankel and Maximilian Kasy weigh the costs and benefits of publishing only significant findings and develop an optimal rule to guide research journals constrained by space. Their findings build on a wave of research that has come amid a reproducibility crisis among the social sciences.
“We say, all right, what are the actual trade-offs here?” Frankel said in an interview with the AEA. “What would an optimal publication system look like? Are there problems with moving to purely non-selective publication?”
Before answering that, however, researchers must answer a more fundamental question: why are studies valuable? Is it more important to get all information out there, or is research valuable because it changes our understanding of the world?
Frankel and Kasy assumed the latter.
“If your goal is not just pure academic knowledge of what is true in the world, if your goal is to do something with this information, if you're going to make policy based on the results of these studies, then some results will be more useful for policy,” Frankel said.
Credit where credit is (not) due
Findings that fail to replicate not only get published, but they get cited just as often as studies that pass replication. The chart below shows that citation rates for replicating and nonreplicating psychology papers published between 2009 and 2017 were indistinguishable.
The authors develop a model that formalizes some of the problems that come with publication bias, such as the unreliability of confidence intervals that come with extreme results, and then examine the costs and benefits from publishing a range of findings.
Because the value of research lies in moving beliefs, choosing whether to publish necessarily depends on prevailing opinion. A result that confirms that what you're doing is what you should be doing is less useful than a result that says what you are doing is wrong.
In an ideal world, all information would be free and available for consumption. But there are costs that come with non-selective publishing. One relates to the limits on attention itself. The most valuable studies—those that change our beliefs—might get buried beneath thousands of studies that do nothing to change policy.
If your goal is not just pure academic knowledge of what is true in the world, if your goal is to do something with this information, if you're going to make policy based on the results of these studies, then some results will be more useful for policy.
Alexander Frankel
Also, the type of policy matters. With policy that might be continuously changing, like tax rates, you can make mistakes in either direction by setting a tax rate that is either too high or too low. In that case, new studies that find surprisingly large effects are valuable for telling us to raise rates, and studies that find surprisingly small effects are valuable for telling us to lower rates.
By contrast, in the case of a binary policy where we either do something or don’t do it, such as passing a law, it might be much more helpful to get surprising results in one direction versus the other. A surprisingly large effect might convince us to implement the policy, while a surprisingly small effect confirms that we shouldn’t.
“The structure of the policy is going to affect the value of information,” Frankel said.
The authors are not suggesting that only extreme results be published. Again, it depends on what the prevailing opinion is. If a study found a high result when you were already expecting a high result, that would do nothing to shift beliefs. It also depends on how confident people are in their opinions. If there is a research-backed consensus about a certain outcome, then a paper showing a radically different result is going to have to justify itself.
The authors say that context matters, and ignoring it with a blunt rule that calls for abolishing the gatekeepers may do more harm than good.
“I think our headline result is basically just to say to the reformers that we have to be careful here,” Frankel said. “If we can't get to the first-best world where all data is published . . . if there is going to be some gatekeeper deciding where to focus our attention, then there actually is a benefit of highlighting the surprising studies that move your beliefs more rather than the less surprising studies.”
♦
“Which Findings Should Be Published” appears in the February 2022 issue of the American Economic Journal: Microeconomics.