Epidemiologists are not idiots. Yes, the science press may be misleading (and the academics may be to blame for overstating their results) but you can’t pretend that observational studies do not teach us anything. We know a lot about the harms of smoking without ever having run a RCT. Even recently SIDS has dropped by something like 75% thanks in part to recommendations like back sleeping for which there were zero RCTs: http://ije.oxfordjournals.org/content/34/4/874.full
I don’t think I make the mistake of thinking that medicine is easy.
We know a lot about the harms of smoking without ever having run a RCT.
It is true that I don’t know of an experiment where you asked some random group to smoke cigarettes while getting another random group to smoke a placebo. That does not mean that people did not run extensive scientific experiments. They certainly did. They still do. E.g., see
Prenatal Nicotine Exposure in Rhesus Monkeys Compromises Development of Brainstem and Cardiac Monoamine Pathways Involved in Perinatal Adaptation and Sudden Infant Death Syndrome: Amelioration by Vitamin C, Neurotoxicology and Teratology 33 (2011): 431-4.
We got just about every animal model full of tobacco and we have measured with great accuracy the results. Sure, it is conceivable that all mammals suffer from tobacco except human beings. But then, again, in science we never really prove anything, we just mount evidence.
I agree that observational studies can serve as evidence… and my blog post says so… but you have to recognize that even if you put all the statistical wizardry into it, it is still a pre-scientific process. What it means in concrete terms is that it is unreasonably easy to derive the wrong conclusions from observational studies.
Epidemiologists are not idiots.
Neither are priests, historians, politicians, CEOs…
There is plenty of poor science all over, but as my post makes clear, if there is one place where you should have robust science, it is in medical research. Firstly because they have all the money. Secondly because getting the facts wrong has dire consequences.
Aramsays:
Medical research is uniquely constrained by ethical constraints, making it hard to be “robust” in the ways we would like. For example, once we are reasonably certain that one treatment is best, it can become unethical to experiment with other treatment.
I think that your standards would say that astronomy is not a science. After all, they just “observe” but there are no “repeatable experiments”.
I think that your standards would say that astronomy is not a science.
Observational astronomy goes way back, centuries before the scientific revolution. It is no different than geography. Please note that I qualify my statement with the adjective “observational”.
I am pretty strong on the opinion that science requires experiments capable of falsifying theories. I am open to alternative models… yet I am not willing to throw everything into a bag and call it science.
Medical research is uniquely constrained by ethical constraints, making it hard to be “robust†in the ways we would like.
Ethics is one of many things that make medical research difficult.
once we are reasonably certain that one treatment is best, it can become unethical to experiment with other treatment.
… which is an argument for putting the bar very high. It echoes a point made in my blog post about why we should hold medical research to a high scientific standard.
Pierresays:
Thanks for the article. While I agree with what you said, it would be nice to have sources for the correlation between smoking and cancer, or cardio-vascular diseases and the 100 fold increase between 40 to 90.
I am not going to document the link between cancer and smoking. It has been done to death. So to speak.
The exponential growth of cancer and cardiovascular diseases with age has also been thoroughly documented, but is rarely reported in the news. Here are two plots I found after searching for 10 seconds in Google…
You can find more. It is best if you look for yourself. It is not hard. Just use Google.
It is a very well established fact that the risks of cancer and cardiovascular diseases go up exponentially with age.
In fact, it is safe to say that if you were to cure aging, you’d cure cancer and cardiovascular diseases. As long as aging exists, no amount of kale and exercise is going to keep you free of these diseases (except that you may die first of something else like Alzheimer’s).
Petersays:
In like your article. But I’d also like to add a comment:
Observational studies are not science by themselves, but they are part of the scientific process. Hence, people doing them engage in the scientific process which is abbreviated as “doing science”.
In that sense I find it completely legit to say “doing science” in that context.
Of course, it’s wrong if you stop there and just write a press release. (because cargo-science)
It’s like turning the steering wheel. Just turning it doesn’t equate to “driving a car”. But if you’re driving a car, turning the steering wheel is an essential part. And people usually would call that “driving” although at each specific point in time the driver does just turn the wheel or push some pedal or push a button or look right and left.
I’d like to add the selection bias on so many levels in medical research. People are already not doing studies which will give a negative result. If they do a study, they might not even try to publish if the result is negative, or it just gets less effort and stays in the drawer. And if the result is negative it is then difficult to get it published, because in journals the fancy positive studies are much better received.
In sciences like physics this is probably less of an issue, because a “negative” result is a positive result which pushes some boundary to more extreme values.
My 2ct
jpsays:
I think Jon Oliver must have based last nights show on this article. Also, “Believing something because impressive people say it must be so is the illustration of pre-scientific thought.” where does this put NDT?
David Taylorsays:
As Peter says, observations are an important part of science. If some people didn’t publish their data, then other people wouldn’t be able to analyse it and find underlying mechanisms (which is when science really gets going).
Respectable journals don’t allow you to publish dodgy statistics (believe me I’ve tried!) or to write conclusions that are not proven by your data. What I think you are talking about Daniel are the media reports that inflate these findings for journalistic purposes.
Ok. Let us get serious here. Let me take an example…
An international panel of experts convened by the World Health Organization concluded Monday that eating processed meat like hot dogs, ham and bacon raises the risk of colon cancer
That’s from the New York times.
From the WHO’s web site:
The IARC Working Group concluded that eating processed meat causes colorectal cancer.
Repeat after me: you can’t establish causality with observational studies.
What you seem to be referring to is not “science”, but rather, the scientific method. Statistics is definitely a science. Its methods have been proven by the scientific method. The abuse of statistics seems to mostly come from people in other scientific disciplines.
I get that drawing conclusions whole cloth from statistics is poor science. Performing tests to verify our hypothesis is using the scientific method to complete the square as it were. I absolutely agree that many, many studies are performed, even intentionally, with parameters that unduly elevate the appearance of significance when in real life, the effects are insignificant.
Then there is an additional dimension: nowhere in the calculus of health is the quality of life measured. How many millions of people suffer feeling inadequate, feeling ugly, never happy in their own skins – just because of a number called BMI. Because Doctors masquerading as statisticians say (with very limited mathematical training) that a high BMI increases risks of this that and the other illness. Which is quite simply unproven. Real human beings, many teenagers, die of anorexia. Many more suffer their whole lives with depression, rapidly fluctuating weight (and illnesses apparently related to fluctuating weight). Then misogynistic jerks on the internet take these statistics as truth – because, you know, science, and hammer away at fat people sanctimoniously because, you know, science has given them permission to be unconscionably rude. And almost always, to women.
Because Doctors masquerading as statisticians say (with very limited mathematical training) that a high BMI increases risks of this that and the other illness. Which is quite simply unproven.
That’s a great example but I don’t think it comes from doctors. BMI is used all over in research papers.
But yes, that’s a perfect example of widespread non-science. Take a group of people. Ask them if they have cancer. Then ask for their BMI. Then say “oh! oh! people with high BMI are more likely to have cancer!”.
That’s fine scholarship, but if you want to do science, you will have to state your hypothesis “a high BMI causes cancer”.
Asrafulsays:
Insightful . I am not sure how to react . This is complete intuitive natural explanation.
Readers may get confused to think this is medical science related random writing, but this is much more then that . This specific reasoning is about how we are getting fooled.
This reminds me of this cartoon.
https://xkcd.com/793/
Epidemiologists are not idiots. Yes, the science press may be misleading (and the academics may be to blame for overstating their results) but you can’t pretend that observational studies do not teach us anything. We know a lot about the harms of smoking without ever having run a RCT. Even recently SIDS has dropped by something like 75% thanks in part to recommendations like back sleeping for which there were zero RCTs:
http://ije.oxfordjournals.org/content/34/4/874.full
A smart biologist who just read news articles about CS, like this one http://www.tomshardware.com/news/zuckerman-random-number-generation-breakthrough,31821.html would conclude that computer scientists were full of baloney.
This reminds me of this cartoon. https://xkcd.com/793/
I don’t think I make the mistake of thinking that medicine is easy.
We know a lot about the harms of smoking without ever having run a RCT.
It is true that I don’t know of an experiment where you asked some random group to smoke cigarettes while getting another random group to smoke a placebo. That does not mean that people did not run extensive scientific experiments. They certainly did. They still do. E.g., see
We got just about every animal model full of tobacco and we have measured with great accuracy the results. Sure, it is conceivable that all mammals suffer from tobacco except human beings. But then, again, in science we never really prove anything, we just mount evidence.
I agree that observational studies can serve as evidence… and my blog post says so… but you have to recognize that even if you put all the statistical wizardry into it, it is still a pre-scientific process. What it means in concrete terms is that it is unreasonably easy to derive the wrong conclusions from observational studies.
Epidemiologists are not idiots.
Neither are priests, historians, politicians, CEOs…
There is plenty of poor science all over, but as my post makes clear, if there is one place where you should have robust science, it is in medical research. Firstly because they have all the money. Secondly because getting the facts wrong has dire consequences.
Medical research is uniquely constrained by ethical constraints, making it hard to be “robust” in the ways we would like. For example, once we are reasonably certain that one treatment is best, it can become unethical to experiment with other treatment.
I think that your standards would say that astronomy is not a science. After all, they just “observe” but there are no “repeatable experiments”.
I think that your standards would say that astronomy is not a science.
Observational astronomy goes way back, centuries before the scientific revolution. It is no different than geography. Please note that I qualify my statement with the adjective “observational”.
I am pretty strong on the opinion that science requires experiments capable of falsifying theories. I am open to alternative models… yet I am not willing to throw everything into a bag and call it science.
Medical research is uniquely constrained by ethical constraints, making it hard to be “robust†in the ways we would like.
Ethics is one of many things that make medical research difficult.
once we are reasonably certain that one treatment is best, it can become unethical to experiment with other treatment.
… which is an argument for putting the bar very high. It echoes a point made in my blog post about why we should hold medical research to a high scientific standard.
Thanks for the article. While I agree with what you said, it would be nice to have sources for the correlation between smoking and cancer, or cardio-vascular diseases and the 100 fold increase between 40 to 90.
@Pierre
I am not going to document the link between cancer and smoking. It has been done to death. So to speak.
The exponential growth of cancer and cardiovascular diseases with age has also been thoroughly documented, but is rarely reported in the news. Here are two plots I found after searching for 10 seconds in Google…
http://www.gov.scot/Resource/Img/233941/0068573.gif
http://www2.nau.edu/~gaud/bio372/class/cancer/ages.gif
You can find more. It is best if you look for yourself. It is not hard. Just use Google.
It is a very well established fact that the risks of cancer and cardiovascular diseases go up exponentially with age.
In fact, it is safe to say that if you were to cure aging, you’d cure cancer and cardiovascular diseases. As long as aging exists, no amount of kale and exercise is going to keep you free of these diseases (except that you may die first of something else like Alzheimer’s).
In like your article. But I’d also like to add a comment:
Observational studies are not science by themselves, but they are part of the scientific process. Hence, people doing them engage in the scientific process which is abbreviated as “doing science”.
In that sense I find it completely legit to say “doing science” in that context.
Of course, it’s wrong if you stop there and just write a press release. (because cargo-science)
It’s like turning the steering wheel. Just turning it doesn’t equate to “driving a car”. But if you’re driving a car, turning the steering wheel is an essential part. And people usually would call that “driving” although at each specific point in time the driver does just turn the wheel or push some pedal or push a button or look right and left.
I’d like to add the selection bias on so many levels in medical research. People are already not doing studies which will give a negative result. If they do a study, they might not even try to publish if the result is negative, or it just gets less effort and stays in the drawer. And if the result is negative it is then difficult to get it published, because in journals the fancy positive studies are much better received.
In sciences like physics this is probably less of an issue, because a “negative” result is a positive result which pushes some boundary to more extreme values.
My 2ct
I think Jon Oliver must have based last nights show on this article. Also, “Believing something because impressive people say it must be so is the illustration of pre-scientific thought.” where does this put NDT?
As Peter says, observations are an important part of science. If some people didn’t publish their data, then other people wouldn’t be able to analyse it and find underlying mechanisms (which is when science really gets going).
Respectable journals don’t allow you to publish dodgy statistics (believe me I’ve tried!) or to write conclusions that are not proven by your data. What I think you are talking about Daniel are the media reports that inflate these findings for journalistic purposes.
Ok. Let us get serious here. Let me take an example…
An international panel of experts convened by the World Health Organization concluded Monday that eating processed meat like hot dogs, ham and bacon raises the risk of colon cancer
That’s from the New York times.
From the WHO’s web site:
The IARC Working Group concluded that eating processed meat causes colorectal cancer.
Repeat after me: you can’t establish causality with observational studies.
What you seem to be referring to is not “science”, but rather, the scientific method. Statistics is definitely a science. Its methods have been proven by the scientific method. The abuse of statistics seems to mostly come from people in other scientific disciplines.
I get that drawing conclusions whole cloth from statistics is poor science. Performing tests to verify our hypothesis is using the scientific method to complete the square as it were. I absolutely agree that many, many studies are performed, even intentionally, with parameters that unduly elevate the appearance of significance when in real life, the effects are insignificant.
Then there is an additional dimension: nowhere in the calculus of health is the quality of life measured. How many millions of people suffer feeling inadequate, feeling ugly, never happy in their own skins – just because of a number called BMI. Because Doctors masquerading as statisticians say (with very limited mathematical training) that a high BMI increases risks of this that and the other illness. Which is quite simply unproven. Real human beings, many teenagers, die of anorexia. Many more suffer their whole lives with depression, rapidly fluctuating weight (and illnesses apparently related to fluctuating weight). Then misogynistic jerks on the internet take these statistics as truth – because, you know, science, and hammer away at fat people sanctimoniously because, you know, science has given them permission to be unconscionably rude. And almost always, to women.
Because Doctors masquerading as statisticians say (with very limited mathematical training) that a high BMI increases risks of this that and the other illness. Which is quite simply unproven.
That’s a great example but I don’t think it comes from doctors. BMI is used all over in research papers.
But yes, that’s a perfect example of widespread non-science. Take a group of people. Ask them if they have cancer. Then ask for their BMI. Then say “oh! oh! people with high BMI are more likely to have cancer!”.
That’s fine scholarship, but if you want to do science, you will have to state your hypothesis “a high BMI causes cancer”.
Insightful . I am not sure how to react . This is complete intuitive natural explanation.
Readers may get confused to think this is medical science related random writing, but this is much more then that . This specific reasoning is about how we are getting fooled.