28th October 2008, 1 min read

When in doubts, prefer unimpressive negative results

When you first hear about peer review, you are lead to believe that the purpose of peer review is detect mistakes and improve papers. Then, one day, you submit a beautiful paper presenting a correct result.__ Surprise!__ It gets rejected! What happened? Someone decided your work was not impressive enough.

In Why Current Publication Practices May Distort Science, Young et al. explains why the artificial scarcity of publication venues in science may systematically favor wrong results. The short story is this:

journals and conferences pick papers that are most likely to get cited;
papers with impressive positive results are more likely to get cited;
hence, papers with impressive positive results are favoured against unimpressive negative results;
however, unimpressive negative results are actually more likely to be correct!

Hence, the modern version of peer review may actually favour misguided, but impressive results over carefully crafted, but boring results.

One of their conclusions is particularly interesting:

To exorcise the winner’s curse, the quality of experiments rather than the seemingly dramatic results in a minority of them would be the focus of review, but is this feasible in the current reality?

Maybe my advice to submit your papers where they are likely to be accepted might actually lead to better science?

Source: Guillaume Guité