17th September 2010, 3 min read

Can Science be wrong? You bet!

A common answer to my post on the reliability of science, was that fraud was marginal and that, ultimately, science is self-correcting. That is true on one condition: that the science in question is bona fide science. Otherwise, I disagree that institutional science is self-correcting. It is self-correcting about as much as human beings are rational. That is, not often. A lot of what passes for science is actually cargo cult science. What looks like rigorous science, may not be, no matter what the experts tell you. Don’t fool yourself: science is not the process of getting published in prestigious journals or a tool to get a tenured job. Richard Feynman defined science as the belief in the ignorance of experts.

Institutional science can be wrong or not even wrong for decades without any remorse:

• Economists failed to predict or explain the last financial crisis. Yet they can’t put into questions their models. Philip Mirowski explains why: “The range in which dissent happens is so narrow. (…) The field got rid of methodological self-criticism.”
• A large fraction of AI researchers have convinced themselves that intelligence must emerge from Prolog-like reasoning engines. This gave us twenty years of predictions that the future was in expert systems, and the last ten years spent predicting the rise of the Semantic Web. This ever-growing community of AI researchers are oblivious to their own failure to produce any useful result.
• Like Fred Brooks, I’m amazed that in 2010, the waterfall method is taught in software engineering school as the reference model. There is no evidence that it is beneficial and, in fact, much evidence that it is hurtful. That is, students would be better off learning nothing rather than learning to use the waterfall method. Yet, entire Ph.D. thesis are still built on the assumption that the waterfall method is sound. Accordingly, criticizing the waterfall method on campus is a risky business.
• The dominant paradigm of modern Theoretical Physics is String theory, which is not even a scientific theory.

We should not trust that self-correction will happen. Instead, biases are often self-reinforcing. Rather, we must ask how self-correction can happen. I think that all science must be verified by independently designed and reproduced experiments. For example, it is insufficient to verify the speed of light with one reproducible experiment. It must be possible for different researchers to come up independently with different experiments, which are all reproduced independently several times. And if everyone is working from the same data, the limitations of the data may never be revealed. And if there is no experiment, you are doing Mathematics or art, not science.

Peer review does not lead to self-correction. Peer review increases quality, but it can also reinforce biases. In Information Retrieval, we often talk about the trade-off between precision and recall. Peer review improves precision, but degrades recall. If your primary goal is to please your peers, you won’t be tempted to point out the flaws in their research!

However, I am optimistic for the future. The rise of Open Scholarship will allow outsiders to participate in the research process and keep it more honest.