, 2 min read
Peer review is an honor-based system
It would take too long to expose all of the flaws of peer review, here are some:
- some work is just flat wrong because the reviewers cannot analyze all of the mathematical results, and because they cannot redo the experiments;
- numerous researchers cheat, sometimes in small ways (“2 out of 3 experiments agree with by theory, let us drop the third one”), sometimes in big ways (“I don’t have time to run these experiments, so let me make up some data”);
- peer review may perpetuate some biases and prevent researchers from putting into question some fundamental questions (“we decided that this is the right way, if you question it, you are a loony”).
However, for all its faults, peer review remains essential in science. I want other researchers to read and criticize my work. I enjoy it very much when people try to find flaws in my work. I think that my work is serious enough that when people point out flaws, I am usually aware of them at some level and I can respond easily (and enjoy the process).
The type of peer review I do not enjoy is the country-club approach: 1) does the paper agrees with the goals and views of the reviewers 2) is the paper written by someone we can respect? Fortunately, you can navigate the system and stay away (mostly) from country-club peer review.
But why do I still like peer review despite its obvious flaws? Because I see it as an honor-based system. In such a system, you have to accept that there will be cheaters. A lot of them. And there will lots of mistakes. All we have to do is be open about it. That is, you cannot say “but my work was peer reviewed so you cannot question it!” or “I am very good, look at these prestigious publications!”. The peer review is there to help the authors. It is not, however, an insurance against fraud or mistakes. I like peer review because it helps me become better, but I do not use the system to determine how good someone else is.
So, what do we do if we want to know how good someone is? You read his work. You reproduce his experiments. You redo his math. Of course, this scales poorly. If you have to hire someone, you cannot read the work of 50 or 500 candidates. So? I think we have to be realistic. It is hard to know how good someone is. You can get to know 10 or 20 researchers in your life. That is about all.
Hiring processes are flawed. You will hire cheaters. Get over it.