It seems that Assange started with the idea that the public can process the available information (that was back when Wikileaks started). However, after releasing documents, he realized that the general public does not care or have the ability to process the information effectively. So, he ended up giving the information to the media, since the professional journalists are the ones that can take the data and make an effective story.
So, in a sense, he wants people to have access to the raw data (for verifiability), but he does not believe that the general public can effectively process the data to come up with conclusions.
It is not that experts can be replaced. But experts need to be checked, for all the reasons that you list above.
It is not that experts can be replaced. But experts need to be checked, for all the reasons that you list above.
Right.
So, I can read your research papers, and learn quite a bit… but if I am a scientist, I won’t trust you entirely ever. If I care particularly about one statement you make, I’ll try to verify it for myself.
That is what science is. It does not mean that everyone will check the speed of light for themselves in their basement…
By making it easy for people to verify your research, you encourage a scientific frame of mind.
So, I don’t know, maybe you could publish your work as a web service so that anyone can use your API?
I agree strongly with your post. I have been thinking for many years that the scientific method, or way of thinking, is too valuable to be confined to scientists and the entire population should be using it. It has proven very powerful in medicine and should be extended to the social and political arena, not to mention commercial businesses. The very fact that the term “evidence-based X” is used implies that standard practice in area X is not evidence-based, which seems very scary.
While I agree that we should always practice what is often called a “healthy skepticism”, I was quite disturbed by some of your provocative statements, which need to be explained a little bit more if they are to be considered as a starting point to a far-ranging discussion about science and those who contribute to it – or claim to do so.
You write: “Even when they speak in good faith, experts are wrong all the time”. Well, you don’t define who is considered an expert, and if “speak”, refer to giving an interview to a journalist, or if it includes any form of communication (for instance, papers in peer-reviewed journals). From what follows this statement, you seem to imply that experts include researchers’ papers in peer-reviewed journals. If so, does it mean that those papers report wrong findings or reach wrong conclusions all the time, or if there’s always something wrong in each and every of them.
You write further, quoting Richard Feynman that “science is the belief in the ignorance of experts”. Well, I read the entire (rather clumsy, in my opinion; certainly not Feynman at his best), and it’s not really clear who are those “experts”: Feynman speaks vaguely of “pseudoscientific advisers” who imitate science by somewhat blindly applying its methods.
And finally, you say that “running experiments or doing mathematics” are not science (a point also made by Feynman, though I feels that you both meant “by themselves”), but “merely the tools most scientists use” and that “once you understand what science is, you see that anyone can be a scientist”.
Stating that science is not “running experiments or doing mathematics” but “believing in the ignorance of experts” doesn’t seem to me to shed much light on what science can be. But it certainly could be used, in a rather naïve way, to justify your last statement: to be a scientist, no need to know the methods of science, just don’t believe what the experts (scientists?) say.
I could wait for your reaction before going on, but here is my personal view of science.
Science is made of:
– a set of objects and phenomena which are deemed to fall under its scrutiny (for instance, God(s), the soul, do not);
– questions and(or) hypotheses about these (a point also made clear in Feynman’s speech);
– a set of methods and tools (including, among many others, mathematics) used to answer questions or validate hypotheses;
– a social system which regulates it, mostly on a single-discipline basis, defining which objects, phenomena, questions, methods, tools, etc. are relevant and how resources and rewards are to be allocated (and here we find ample room for the biases you allude to);
As to the significance of doubt, or skepticism, although it is a highly praised value, or attitude, in practice one must make a trade-off between doubt and efficiency. As you say, no need to check the speed of light in one’s basement. In fact, only unexpected results (like cold fusion, for instance) are systematically rechecked. Sure, every scientist know that there is a probability that any published result is in fact wrong, either due to good-faith error or scientific fraud). But one must assume it’s quite small.
Back in the Middle Ages, we had many activities which could be classified as science in that they had (1) a set of objects (2) questions (3) methods and tools (4) a social system. I am sure that many things were “demonstrated” using experiments and mathematics all throughout history. And to some extend, we can probably trace back to origin of science to the beginnings of civilization. Of course, you could argue that science is defined by specific methods.
Yet, I don’t care about how some people do science right now: I argue that this can change drastically in the near future. And indeed, people like Michael Nielsen are predicting such drastic changes. The social organization, the disciplines and the methods could change, but my definition of science wouldn’t.
I’m interested in science as the object that brought so much scholarly progress so fast. And I think that the fundamental change, the revolution if you want, is that knowledge went from something that relied on “authority” (truth comes from God or our Great Leaders) to something that relied on independent, autonomous, verification.
While the business of science recognizes leaders, experts and authorities, science itself does not. And that is why I’m not interested in defining what an expert is.
You can either consider truth to be something that is handed out to you… or as something that you can, and should, verify for yourself.
Obviously, my definition does not fit with the conventional one which limits science to Physics, Chemistry and maybe Biology (though Feynman would exclude Biology, I suppose). Rather, I think that much of what is done in the name of science, in Physics with String Theory for example, is based authoritative knowledge. Everyone does it, if you dare doubt, you won’t find a job in theoretical physics, and so on. This is not science. I don’t think that science is limited to some objects (mostly physical). Any object, any truth, can be approached scientifically as long as it is verifiable. The essential step is the rejection of authority: truth knows no master, truth does not care about famous and powerful people. Truth can be found by anyone, as long as he does the work and has the skills to find it.
If so, does it mean that those papers report wrong findings or reach wrong conclusions all the time, or if there’s always something wrong in each and every of them.
I meant “all the time” in the colloquial sense. Specifically, I am not surprised when a research article or a scientist is wrong.
Ranjit Chandra also comes to mind. He published 272 research papers (in Scopus) which were cited over 3000 times according to Scopus. He received millions in funding and the order of Canada. Yet, it now looks like a sizeable fraction of his work was fabricated. The research funding he received was handed out to his family.
(…) to be a scientist, no need to know the methods of science, just don’t believe what the experts (scientists?) say.
If you doubt the experts and investigate the facts for yourself, then you need to learn quite a bit to be effective. If I doubt the currently accepted measure of the speed of light, and I want to measure it myself, I need to learn how it is done. If I want to determine for myself how many people were killed in Irak, I need to learn strategies to get independent counts to get an overal reliable picture. If I doubt the economists, I need to learn how to model the economy, verify my models and so on.
But I make a difference between the business of science, whereas I need to follow certain guidelines for my results to be “accepted”, and science itself. It is likely that if I measure the speed of light in my basement and find the widely accepted measure to be wrong, I wouldn’t be taken seriously. This would be especially true if my methods were entirely novel. So the business of science would reject me. Nevertheless, I would still be doing science [maybe it would be bad science, but that’s another issue].
Sure, every scientist know that there is a probability that any published result is in fact wrong, either due to good-faith error or scientific fraud). But one must assume it’s quite small.
I never expect any research paper to be entirely truthful and complete (outside maybe mathematics, and even in mathematics, many proofs are wrong when scrutinized). I should add that I don’t even trust my own research papers.
In fact, trusting research papers implicitly, without verification, is accepting authoritative truth. Because something comes from an influential individual, or a prestigious venue, it must be true. Well, that is unscientific to me. It is precisely what people before the scientific era: they accepted as true what they were given because it came for an authoritative source.
I don’t know why you imply that we need to assume that research papers are entirely truthful to be effective? I don’t. I routinely check published results for myself before building upon them. Of course, I don’t have time to check everything all the time. And that is not the point. You can doubt everything, including everything you do and think, and still go on living and be productive.
I’ve long accepted that I’m often wrong. I have no problem with this… I don’t know why anyone would have problems accepting that the research papers I publish might be wrong.
If one day we have the technology to automatically verify the mathematics used in research papers, I promise you that we will learn that many, many papers, maybe most of them, contains mistake. I think that a vast number of mathematics research papers are flat wrong.
But it does not matter. We don’t need absolute truth to go on. In fact, that is precisely what science is. The recognition that you can doubt and things do not fall apart.
Back in the Middle Ages, we had many activities which could be classified as science in that they had (1) a set of objects (2) questions (3) methods and tools (4) a social system.
You’re right. And there are some still in place; astrology is an example. And it is not trivial to prove (as far as one can really “prove” this sort of thing) that it’s not, on a theoretical basis, as good a science as meteorology, or economics, for instance, which both share astrology’s lack of predictive accuracy (an argument often used to disqualify astrology).
One of the most serious arguments, in my opinion, relates to the coherence of science: although each discipline has its own largerly autonomous territory, not covered by other disciplines, there is always some overlap. For instance, many of the objects studied by biology are not in the realm of physics, but one couldn’t accept a biological explanation or theory concerning those objects which would violate the laws of conservation of energy or momentum, which are part of the legitimate territory of physics.
As to astrology, there is the problem of making such links with notions from physics (the negligible variation of the gravitational field due to planet alignments), and then biology and(or) psychology (the effect of this variation on the physiology or behavior of living organisms).
So I should maybe add a to my definition something about the overall coherence of science.
The social organization, the disciplines and the methods could change, but my definition of science wouldn’t. […] But I make a difference between the business of science […] and science itself.
It is here where we disagree: As is quite obvious in my definition of science, the “business of science”, as you call it, is part of the definition (what I call the “social system”. Personally, I don’t see the interest of defining science independent of the way it is practiced and socially recognized.
I meant “all the time†in the colloquial sense. Specifically, I am not surprised when a research article or a scientist is wrong.
But you also said “researchers […] are also systematically wrong”, which is quite stronger than the colloquial meaning of “all the time” (as in “it happens all the time”, if I understand well).
I was also thinking of Ioannidis’s work, for example Ioannidis, Why Most Published Research Findings Are False.
It has been pointed out (see the comments on the paper) that Ioannidis’s conclusion applied to a quite specific type of research, certainly important in his field (clinical medicine) and that its title was way too much general. It is also recognized, first of all by Ioannidis himself, that it doesn’t make those papers useless (though some may think otherwise).
Ranjit Chandra also comes to mind. […] it now looks like a sizeable fraction of his work was fabricated.
I had read the blog post where you discussed this case (among others) of scientific fraud. But it is generally agreed that such over-the-board fraudulent behavior is extremely uncommon. We’re talking of outliers here. One has to know that they exist, but they don’t characterize, much less define a phenomenon. Scientific fraud doesn’t define science, not more than business fraud (although well documented) defines business in general.
Nevertheless [finding a different value for the speed of light using novel methods], I would still be doing science (maybe it would be bad science, but that’s another issue).
That’s an interesting example: according to my definition of science, it could be either bad science or science, according to the nature of your “novel methods”. If they are based upon completely different assumptions, or even postulates, that what constitutes science (according to the “businessmen” of science), I would say it’s simply not science.
For instance, you could have devised a method consisting of using meditation to reach a holistic understanding of the Universe, which allows you to perceive directly in you mind the value of the speed of light, which is infinite, of course. Would it be science because you didn’t believe the experts who say it’s not the correct value, and not the way to determine it? I’m the one being provocative here (but I suspect you can can enjoy).
Anyway, I’ll agree with you that even when one uses “correct” methods, it can be real hard to challenge accepted results or ideas.
I don’t know why you imply that we need to assume that research papers are entirely truthful to be effective?
That’s not what I meant. I simply said that for efficiency (otherwise stated, for practical purposes) one cannot systematically mistrust what has been published, and must accept the risk that some results are inaccurate or even plainly false.
But I believe that you cannot build much upon false results; at a certain point, you won’t be able to obtain the results you were hoping for on the basis of those previous results. But it will often be hard to know if it’s you or those results which are to blame.
But you also said “researchers […] are also systematically wrongâ€, which is quite stronger than the colloquial meaning of “all the time†(as in “it happens all the timeâ€, if I understand well).
I take “systematically” to mean “in a consistent manner”. Researchers do not make independent and random mistakes. It corresponds to what Ioannidis has shown in “Why Most Published Research Findings Are False”.
It has been pointed out (see the comments on the paper) that Ioannidis’s conclusion applied to a quite specific type of research (…)
True, but it is scientific research nonetheless. And the fact that he only addressed one type of scientific research does not leave out other types in the clear.
It is also recognized, first of all by Ioannidis himself, that it doesn’t make those papers useless (…)
No, it does not make them useless. Mistakes are ok.
Computer Science have taught us that probabilistic algorithms are tremendously useful. Yet, they get things wrong with non-zero probabilities. It is ok.
But it is generally agreed that such over-the-board fraudulent behavior is extremely uncommon.
It illustrates that peer review journals publish false or even misleading results.
We’re talking of outliers here.
And how do I verify this “fact”? I must trust it because it is often repeated by experts?
The problem is that, as a researcher, I encounter results that I cannot reproduce all the time. And there is no sane process by which I can address this. I cannot even report them.
(I did, once, on this blog, point to a flawed paper which appeared in one of the best Computer Science venue. I was asked to take down the blog post. I did. The paper was not withdrawn or corrected. You can still download it. It has been cited numerous times. Yet, it is fundamentally flawed. The research results have been made up, or were really badly executed.)
(Now that I think about it… we need a Wikileaks for science: anonymously report the results you could not verify or reproduce. But I don’t want to become the “Assange of Science”. I have kids. I don’t want to end up in jail for rape.)
For instance, you could have devised a method consisting of using meditation to reach a holistic understanding of the Universe, which allows you to perceive directly in you mind the value of the speed of light, which is infinite, of course.
What you are describing sounds a lot like how one could come up with String Theory. Jokes aside, you make a very good point. Yet, even with instruments and mathematics, you can fool yourself. In fact, it takes very little effort to fool yourself… you don’t need meditation for it. (See above references for proof of that.)
But I believe that you cannot build much upon false results; at a certain point, you won’t be able to obtain the results you were hoping for on the basis of those previous results. But it will often be hard to know if it’s you or those results which are to blame.
It depends. If I tell you that X is better than Y. You might work using X the rest of your life, without knowing that Y is really better.
So, if someone tells you that X is better than Y, and everyone thinks it is a bad idea to check… how are we ever going to find out?
I was watching the Wikileaks documentary by the Swedish TV at http://svtplay.se/v/2264028/wikirebels___the_documentary?cb,a1364145,1,f,-1/pb,a1364142,1,f,-1/pl,v,,2264028/sb,p118750,1,f,-1
It seems that Assange started with the idea that the public can process the available information (that was back when Wikileaks started). However, after releasing documents, he realized that the general public does not care or have the ability to process the information effectively. So, he ended up giving the information to the media, since the professional journalists are the ones that can take the data and make an effective story.
So, in a sense, he wants people to have access to the raw data (for verifiability), but he does not believe that the general public can effectively process the data to come up with conclusions.
It is not that experts can be replaced. But experts need to be checked, for all the reasons that you list above.
It is not that experts can be replaced. But experts need to be checked, for all the reasons that you list above.
Right.
So, I can read your research papers, and learn quite a bit… but if I am a scientist, I won’t trust you entirely ever. If I care particularly about one statement you make, I’ll try to verify it for myself.
That is what science is. It does not mean that everyone will check the speed of light for themselves in their basement…
By making it easy for people to verify your research, you encourage a scientific frame of mind.
So, I don’t know, maybe you could publish your work as a web service so that anyone can use your API?
I agree strongly with your post. I have been thinking for many years that the scientific method, or way of thinking, is too valuable to be confined to scientists and the entire population should be using it. It has proven very powerful in medicine and should be extended to the social and political arena, not to mention commercial businesses. The very fact that the term “evidence-based X” is used implies that standard practice in area X is not evidence-based, which seems very scary.
Hi Daniel,
While I agree that we should always practice what is often called a “healthy skepticism”, I was quite disturbed by some of your provocative statements, which need to be explained a little bit more if they are to be considered as a starting point to a far-ranging discussion about science and those who contribute to it – or claim to do so.
You write: “Even when they speak in good faith, experts are wrong all the time”. Well, you don’t define who is considered an expert, and if “speak”, refer to giving an interview to a journalist, or if it includes any form of communication (for instance, papers in peer-reviewed journals). From what follows this statement, you seem to imply that experts include researchers’ papers in peer-reviewed journals. If so, does it mean that those papers report wrong findings or reach wrong conclusions all the time, or if there’s always something wrong in each and every of them.
You write further, quoting Richard Feynman that “science is the belief in the ignorance of experts”. Well, I read the entire (rather clumsy, in my opinion; certainly not Feynman at his best), and it’s not really clear who are those “experts”: Feynman speaks vaguely of “pseudoscientific advisers” who imitate science by somewhat blindly applying its methods.
And finally, you say that “running experiments or doing mathematics” are not science (a point also made by Feynman, though I feels that you both meant “by themselves”), but “merely the tools most scientists use” and that “once you understand what science is, you see that anyone can be a scientist”.
Stating that science is not “running experiments or doing mathematics” but “believing in the ignorance of experts” doesn’t seem to me to shed much light on what science can be. But it certainly could be used, in a rather naïve way, to justify your last statement: to be a scientist, no need to know the methods of science, just don’t believe what the experts (scientists?) say.
I could wait for your reaction before going on, but here is my personal view of science.
Science is made of:
– a set of objects and phenomena which are deemed to fall under its scrutiny (for instance, God(s), the soul, do not);
– questions and(or) hypotheses about these (a point also made clear in Feynman’s speech);
– a set of methods and tools (including, among many others, mathematics) used to answer questions or validate hypotheses;
– a social system which regulates it, mostly on a single-discipline basis, defining which objects, phenomena, questions, methods, tools, etc. are relevant and how resources and rewards are to be allocated (and here we find ample room for the biases you allude to);
As to the significance of doubt, or skepticism, although it is a highly praised value, or attitude, in practice one must make a trade-off between doubt and efficiency. As you say, no need to check the speed of light in one’s basement. In fact, only unexpected results (like cold fusion, for instance) are systematically rechecked. Sure, every scientist know that there is a probability that any published result is in fact wrong, either due to good-faith error or scientific fraud). But one must assume it’s quite small.
@Couture
Back in the Middle Ages, we had many activities which could be classified as science in that they had (1) a set of objects (2) questions (3) methods and tools (4) a social system. I am sure that many things were “demonstrated” using experiments and mathematics all throughout history. And to some extend, we can probably trace back to origin of science to the beginnings of civilization. Of course, you could argue that science is defined by specific methods.
Yet, I don’t care about how some people do science right now: I argue that this can change drastically in the near future. And indeed, people like Michael Nielsen are predicting such drastic changes. The social organization, the disciplines and the methods could change, but my definition of science wouldn’t.
I’m interested in science as the object that brought so much scholarly progress so fast. And I think that the fundamental change, the revolution if you want, is that knowledge went from something that relied on “authority” (truth comes from God or our Great Leaders) to something that relied on independent, autonomous, verification.
While the business of science recognizes leaders, experts and authorities, science itself does not. And that is why I’m not interested in defining what an expert is.
You can either consider truth to be something that is handed out to you… or as something that you can, and should, verify for yourself.
Obviously, my definition does not fit with the conventional one which limits science to Physics, Chemistry and maybe Biology (though Feynman would exclude Biology, I suppose). Rather, I think that much of what is done in the name of science, in Physics with String Theory for example, is based authoritative knowledge. Everyone does it, if you dare doubt, you won’t find a job in theoretical physics, and so on. This is not science. I don’t think that science is limited to some objects (mostly physical). Any object, any truth, can be approached scientifically as long as it is verifiable. The essential step is the rejection of authority: truth knows no master, truth does not care about famous and powerful people. Truth can be found by anyone, as long as he does the work and has the skills to find it.
If so, does it mean that those papers report wrong findings or reach wrong conclusions all the time, or if there’s always something wrong in each and every of them.
I meant “all the time” in the colloquial sense. Specifically, I am not surprised when a research article or a scientist is wrong.
I give a reference at the end of my post:
Jonah Lehrer, The Truth Wears Off: Is there something wrong with the scientific method?
http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz186MYaBDA
I was also thinking of Ioannidis’s work, for example:
Ioannidis, Why Most Published Research Findings Are False
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124
Ranjit Chandra also comes to mind. He published 272 research papers (in Scopus) which were cited over 3000 times according to Scopus. He received millions in funding and the order of Canada. Yet, it now looks like a sizeable fraction of his work was fabricated. The research funding he received was handed out to his family.
(…) to be a scientist, no need to know the methods of science, just don’t believe what the experts (scientists?) say.
If you doubt the experts and investigate the facts for yourself, then you need to learn quite a bit to be effective. If I doubt the currently accepted measure of the speed of light, and I want to measure it myself, I need to learn how it is done. If I want to determine for myself how many people were killed in Irak, I need to learn strategies to get independent counts to get an overal reliable picture. If I doubt the economists, I need to learn how to model the economy, verify my models and so on.
But I make a difference between the business of science, whereas I need to follow certain guidelines for my results to be “accepted”, and science itself. It is likely that if I measure the speed of light in my basement and find the widely accepted measure to be wrong, I wouldn’t be taken seriously. This would be especially true if my methods were entirely novel. So the business of science would reject me. Nevertheless, I would still be doing science [maybe it would be bad science, but that’s another issue].
Sure, every scientist know that there is a probability that any published result is in fact wrong, either due to good-faith error or scientific fraud). But one must assume it’s quite small.
I never expect any research paper to be entirely truthful and complete (outside maybe mathematics, and even in mathematics, many proofs are wrong when scrutinized). I should add that I don’t even trust my own research papers.
In fact, trusting research papers implicitly, without verification, is accepting authoritative truth. Because something comes from an influential individual, or a prestigious venue, it must be true. Well, that is unscientific to me. It is precisely what people before the scientific era: they accepted as true what they were given because it came for an authoritative source.
I don’t know why you imply that we need to assume that research papers are entirely truthful to be effective? I don’t. I routinely check published results for myself before building upon them. Of course, I don’t have time to check everything all the time. And that is not the point. You can doubt everything, including everything you do and think, and still go on living and be productive.
I’ve long accepted that I’m often wrong. I have no problem with this… I don’t know why anyone would have problems accepting that the research papers I publish might be wrong.
If one day we have the technology to automatically verify the mathematics used in research papers, I promise you that we will learn that many, many papers, maybe most of them, contains mistake. I think that a vast number of mathematics research papers are flat wrong.
But it does not matter. We don’t need absolute truth to go on. In fact, that is precisely what science is. The recognition that you can doubt and things do not fall apart.
And yes, btw, feel free to doubt me.
@Daniel
Back in the Middle Ages, we had many activities which could be classified as science in that they had (1) a set of objects (2) questions (3) methods and tools (4) a social system.
You’re right. And there are some still in place; astrology is an example. And it is not trivial to prove (as far as one can really “prove” this sort of thing) that it’s not, on a theoretical basis, as good a science as meteorology, or economics, for instance, which both share astrology’s lack of predictive accuracy (an argument often used to disqualify astrology).
One of the most serious arguments, in my opinion, relates to the coherence of science: although each discipline has its own largerly autonomous territory, not covered by other disciplines, there is always some overlap. For instance, many of the objects studied by biology are not in the realm of physics, but one couldn’t accept a biological explanation or theory concerning those objects which would violate the laws of conservation of energy or momentum, which are part of the legitimate territory of physics.
As to astrology, there is the problem of making such links with notions from physics (the negligible variation of the gravitational field due to planet alignments), and then biology and(or) psychology (the effect of this variation on the physiology or behavior of living organisms).
So I should maybe add a to my definition something about the overall coherence of science.
The social organization, the disciplines and the methods could change, but my definition of science wouldn’t. […] But I make a difference between the business of science […] and science itself.
It is here where we disagree: As is quite obvious in my definition of science, the “business of science”, as you call it, is part of the definition (what I call the “social system”. Personally, I don’t see the interest of defining science independent of the way it is practiced and socially recognized.
I meant “all the time†in the colloquial sense. Specifically, I am not surprised when a research article or a scientist is wrong.
But you also said “researchers […] are also systematically wrong”, which is quite stronger than the colloquial meaning of “all the time” (as in “it happens all the time”, if I understand well).
I was also thinking of Ioannidis’s work, for example Ioannidis, Why Most Published Research Findings Are False.
It has been pointed out (see the comments on the paper) that Ioannidis’s conclusion applied to a quite specific type of research, certainly important in his field (clinical medicine) and that its title was way too much general. It is also recognized, first of all by Ioannidis himself, that it doesn’t make those papers useless (though some may think otherwise).
Ranjit Chandra also comes to mind. […] it now looks like a sizeable fraction of his work was fabricated.
I had read the blog post where you discussed this case (among others) of scientific fraud. But it is generally agreed that such over-the-board fraudulent behavior is extremely uncommon. We’re talking of outliers here. One has to know that they exist, but they don’t characterize, much less define a phenomenon. Scientific fraud doesn’t define science, not more than business fraud (although well documented) defines business in general.
Nevertheless [finding a different value for the speed of light using novel methods], I would still be doing science (maybe it would be bad science, but that’s another issue).
That’s an interesting example: according to my definition of science, it could be either bad science or science, according to the nature of your “novel methods”. If they are based upon completely different assumptions, or even postulates, that what constitutes science (according to the “businessmen” of science), I would say it’s simply not science.
For instance, you could have devised a method consisting of using meditation to reach a holistic understanding of the Universe, which allows you to perceive directly in you mind the value of the speed of light, which is infinite, of course. Would it be science because you didn’t believe the experts who say it’s not the correct value, and not the way to determine it? I’m the one being provocative here (but I suspect you can can enjoy).
Anyway, I’ll agree with you that even when one uses “correct” methods, it can be real hard to challenge accepted results or ideas.
I don’t know why you imply that we need to assume that research papers are entirely truthful to be effective?
That’s not what I meant. I simply said that for efficiency (otherwise stated, for practical purposes) one cannot systematically mistrust what has been published, and must accept the risk that some results are inaccurate or even plainly false.
But I believe that you cannot build much upon false results; at a certain point, you won’t be able to obtain the results you were hoping for on the basis of those previous results. But it will often be hard to know if it’s you or those results which are to blame.
But you also said “researchers […] are also systematically wrongâ€, which is quite stronger than the colloquial meaning of “all the time†(as in “it happens all the timeâ€, if I understand well).
I take “systematically” to mean “in a consistent manner”. Researchers do not make independent and random mistakes. It corresponds to what Ioannidis has shown in “Why Most Published Research Findings Are False”.
It has been pointed out (see the comments on the paper) that Ioannidis’s conclusion applied to a quite specific type of research (…)
True, but it is scientific research nonetheless. And the fact that he only addressed one type of scientific research does not leave out other types in the clear.
It is also recognized, first of all by Ioannidis himself, that it doesn’t make those papers useless (…)
No, it does not make them useless. Mistakes are ok.
Computer Science have taught us that probabilistic algorithms are tremendously useful. Yet, they get things wrong with non-zero probabilities. It is ok.
But it is generally agreed that such over-the-board fraudulent behavior is extremely uncommon.
It illustrates that peer review journals publish false or even misleading results.
We’re talking of outliers here.
And how do I verify this “fact”? I must trust it because it is often repeated by experts?
The problem is that, as a researcher, I encounter results that I cannot reproduce all the time. And there is no sane process by which I can address this. I cannot even report them.
(I did, once, on this blog, point to a flawed paper which appeared in one of the best Computer Science venue. I was asked to take down the blog post. I did. The paper was not withdrawn or corrected. You can still download it. It has been cited numerous times. Yet, it is fundamentally flawed. The research results have been made up, or were really badly executed.)
(Now that I think about it… we need a Wikileaks for science: anonymously report the results you could not verify or reproduce. But I don’t want to become the “Assange of Science”. I have kids. I don’t want to end up in jail for rape.)
For instance, you could have devised a method consisting of using meditation to reach a holistic understanding of the Universe, which allows you to perceive directly in you mind the value of the speed of light, which is infinite, of course.
What you are describing sounds a lot like how one could come up with String Theory. Jokes aside, you make a very good point. Yet, even with instruments and mathematics, you can fool yourself. In fact, it takes very little effort to fool yourself… you don’t need meditation for it. (See above references for proof of that.)
But I believe that you cannot build much upon false results; at a certain point, you won’t be able to obtain the results you were hoping for on the basis of those previous results. But it will often be hard to know if it’s you or those results which are to blame.
It depends. If I tell you that X is better than Y. You might work using X the rest of your life, without knowing that Y is really better.
So, if someone tells you that X is better than Y, and everyone thinks it is a bad idea to check… how are we ever going to find out?