, 3 min read
The case against double-blind peer review
Many scientific journals use double-blind peer review. That is, the authors submit their work in a way that cannot be traced back to them. Meanwhile, the authors do not know who the reviewers are. In this way, the reviewers are free to speak their mind. It feels fair because the reviewers cannot be influenced (in theory) by the declared affiliation of the authors or their relative fame.
How well does it work in practice? You would expect double-blind reviewing to favor people from outside academia. Yet Blank (1991) reported that the opposite is true: authors from outside academia have a lower acceptance rate under double-blind peer review. Moreover, Blank indicates that double-blind peer review is overall harsher. This is not a surprise: It is easier to pull the trigger when the enemy wears a mask.
Meanwhile, there is at best a slight increase in the quality of the papers due to double-blind peer review (De Vries et al., 2009), everything else being equal. However, not everything is equal under double-blind peer review. What is the subtext? That somehow, the research paper is a standalone artefact, an anonymous, standardized piece of LEGO. That it should not be viewed as part of a stream of papers produced by an author. It sends a signal that an original research program is a bad idea. Researchers should be interchangeable. And to assess them, we might as well count the number of their papers since these papers are standard artifacts anyhow.
But that is counter-productive! Research papers are often only interesting when put in a greater context. It is only when you align a series of papers, often from the same authors, that you start seeing a story develop. Or not. Sometimes you only realize how poor someone’s work is by collecting their papers and noticing that nothing much is happening: just more of the same.
Researchers must make verifiable statements, but they should also try to be original and interesting. They should also be going somewhere. Research papers are not collection of facts, they represent a particular (hopefully correct) point of view. A researcher’s point of view should evolve, and how it does is interesting. Yet it is a lot easier to understand a point of view when you are allowed to know openly who the authors are.
Are there cliques and biases in science? Absolutely. But the best way to limit the biases is transparency, not more secrecy. Let the world know who rejected which paper and for what reasons.
Source: This blog post came about through an online exchange with Philippe Beaudoin.
References:
- Blank, R.M., The effects of double-blind versus single-blind reviewing: Experimental evidence from the American Economic Review, The American Economic Review 81 (5), 1991.
- De Vries, D.R. and Marschall, E.A. and Stein, R.A., Exploring the Peer Review Process: What is it, Does it Work, and Can it Be Improved? Fisheries 34 (6), 2009.
Update: Mark Wilson has another argument against double-blind peer review. What if you pick up good ideas from double-blind papers that are later rejected and remain unpublished? How do you acknowledge the contribution of the authors of the unpublished work?
Update 2: Patrick Lam points out that the programming languages community now uses a variant of double-blind review for some conferences (like PLDI or POPL, the top PL conferences) where the authors are asked to submit blinded papers, but the identities are revealed to the reviewers after they submit their first-draft reviews.
Further reading: I have more comprehensive argument in a latter blog post.