, 3 min read
The perils of filter-then-publish
Why do I prefer the publish-then-filter system, which dominates social media such as blogs, to the traditional filter-then-publish system used by scientific journals? Because the conventional peer review system (filter-then-publish) has disastrous consequences:
- In the conventional peer review system, you seek to please the reviewers who in turn try to please the editor who in turn is trying to guess what the readers want. It should not be a surprise that the papers are optimized for peer review, not for the reader. While you will eventually get your work published, you may have to drastically alter it to make it pass peer review. A common theme is that you will need to make it look more complicated. In a paper I published a few years ago, I had to use R*-trees, not because I needed them, but because other authors had done so. When I privately asked them why they had used R*-trees, the answer was “it was the only way to get our paper in a major conference”. So my work has been made more complicated for the sole purpose of impressing the reviewers: “look, I know about R*-trees too!” Several times, during the course of peer review, I was asked to remove material which was judged to be “textbook material”: didactic material is frowned upon in many circles (hint: it is not fancy enough). Be warned: if you find an easy way to prove a result, and it ends up looking trivial in retrospect, your work may become unpublishable. You will need to invent complex related problems to pass peer review. It explains why several important results appear as remarks in long and complicated papers. Either purposefully, or by habit, people will write in a way to make their paper pass peer review even if it makes the work inaccessible. Do you think research papers have to be boring? If so, you have been brainwashed.
- The conventional system is legible: you can count and measure a scientist’s production. The incentive is to produce more of what the elite wants. In a publish-then-filter system nobody cares about quantity: only the impact matters. And impact can mean different things to different people. It allows for more diversity in how people produce and consume science. Thus, if you think it would be better if we stopped counting research papers, then you should reject conventional peer review and favor the publish-then-filter system.
- The difference between filter-then-publish and publish-then-filter is analogous with the difference between Soviet central planning and a free market. You either let a select few decide, or you let the market decide. You can either trust that the people will be smart enough, or you can delegate the selection to a few trusted experts.
- The conventional peer review system pretends to delegate the assessment of scientists to review boards. Instead of reading each other, we trust brands. The net result is that people hire and promote each others without reading the work. Thus, the conventional system kills any incentive to build a coherent and interesting body of work: you are just a machine that produces research papers as commodities. You know how you succeed in science these days? Take a few ideas, then try every small variation of these ideas and make a research paper out of each one of them. Each paper will look good and be written quickly, but your body of work will be highly redundant. Instead of working toward deep contributions, we encourage people to repeat themselves more and more and collect many shallow contributions. We sacrifice scholarship for vanity.
Further reading: Become independent of peer review and Three myths about scientific peer review.
Source: This post was inspired by a comment made by Sylvain Hallé.