Publishing is for people working at universities. If you work for a private company, and find some interesting results, you cannot publish, you do not want the competition to know it. You just have to really make sure you validated your own work, because management will make decisions based on it and the outcome has to be what you predicted or you are in trouble !
People who are not in academia don’t need to play this game so they just don’t submit their research (…)
Good point.
Raising the barrier to entry does prevent people from outside the system from playing a useful role.
That is why I seek to promote open scholarship (as opposed to mere “open access”). I think that the world would be a better place if more people pursued scholarship and published their findings. Limiting scholarship to professional researchers is counterproductive.
You just have to really make sure you validated your own work, because management will make decisions based on it and the outcome has to be what you predicted or you are in trouble !
Good point. Putting your work into practice, where everyone can see it, is good validation.
Publishing is for people working at universities. If you work for a private company, and find some interesting results, you cannot publish, you do not want the competition to know it.
Ah. But you do release it eventually, where everyone can see it, or reverse engineer it, don’t you?
Wait… what exactly do you expect? That scientists will retroactively remove citations to fraudulent papers in their own published work?
I use this as evidence that you cannot rely on citations to determine the reliability of the work.
He had a lot more citations before the papers were pulled out. These are the citations that remain on the papers that were not removed from Scopus.
And how many of those citations followed the word “fraudulent�
He was highly cited before getting caught. Since then, he has gathered far fewer citations.
If, as you say, citations are not validation, then why are citations to fraudulent papers objectionable?
They are evidence that people will gladly build on doctored papers.
“He lost his research job, but found an engineering position in Germany.†(…) Are you suggesting that he should not hold an engineering position? What sort of job should he be allowed to hold, then?
At least in Quebec (Canada) where I live, it would not be possible to remain a certified engineer following such a high profile fraud. Or rather, I would hope not.
Engineers are supposed to be able to sign off on public safety issues, for example. Making up data for years is a pretty serious breach of ethics. I’d say it is incompatible with an engineering job.
I completely agree. You also neglected that peer review is not nearly as anonymous as we pretend. If am reviewing a paper and it has Lemire(in press) cited, I know that it must be by either Lemire or one of his students or colleagues. If Lemire is very important in my field than I am going to think twice about rejecting his article.
I have been a reviewer for journals that strongly encouraged us to reject papers with minor problems with a “Revised and resubmit” decision so we could keep our reject rate high and look selective.
People who are not in academia don’t need to play this game so they just don’t submit their research many times.
There are a lot of problems with peer reviewed publications.
JeffEsays:
“Despite all the retracted papers, Jan Hendrik Schön has still 1,200 citations according to Scopus.”
Wait… what exactly do you expect? That scientists will retroactively remove citations to fraudulent papers in their own published work? How do you suggest publishers and citation indexers maintain those changes?
And how many of those citations followed the word “fraudulent”?
If, as you say, citations are not validation, then why are citations to fraudulent papers objectionable?
“He lost his research job, but found an engineering position in Germany.”
Are you suggesting that he should not hold an engineering position? What sort of job should he be allowed to hold, then?
@Zyxo You can publish. I work for a private company. We occasionally publish work after the project is complete, but not that often. The reason isn’t that we are afraid people will steal our ideas. It’s that the benefit to us of publishing is not worth the time we take away from away from paid work to jump through the hoops.
@Lemire I’d like to see both open access and open scholarship. There have been other discussions about the illogic of paying journals for articles reviewed (for free) that were written (for free) by researchers using research conducted with government funds.
Of course, you can get promoted making up data, but if it becomes known, your reputation will be gone.
That’s what we would hope. And certainly, if you go far enough, that is what happens.
But the problem I’d like to underline is that there are many small ways to cheat and mislead. By assuming that science is self-correcting, we are lead to think that cheating is no big deal. Ah! But it is… I think… because people are going to assume that whatever has been published is reliable.
For example, the climate-gate events showed that a lot of the global warming science is based on a few measures that were doctored. Nobody checked this data, nobody asked to see the details of how it was processed… and people built on it. There is no doubt that global warming is real, but a lot of the science might be misleading… maybe only in small ways… but… how do we know?
Why is it that only a few people had access to the raw climate data? Why wasn’t this raw data made widely available? Why is it that no journal reviewer asked to see the raw data?
I would like to point out that important parts of science are confidence and credit. Of course, you can get promoted making up data, but if it becomes known, your reputation will be gone. Of course, it may happen that you keep your former degrees and rewards, and get a decent job, but how you imagine talking to people after that? No one will ever trust you and consider you seriously.
Another thing is that the science system as it is now strongly encourages misconduct. It doesn’t matter how good your results are if they aren’t published in one of the top journals. But you referred to this subject many times in your blog.
The amazing thing is that the process still works. Who knows how many errors and deceptions go uncaught, how many citations lead to bogus facts? And yet our knowledge keeps growing.
In one sense you can extrapolate arbitrarily far from reality as the result of a small error, but in practice scientists do seem to notice when a field diverges significantly. Nobody checked the Climategate data specifically, but plenty of other data was being gathered. If it had been at complete odds with everything else, we’d have noticed.
I think forgetfulness plays a critical role in the process. At worst a whole field might pursue a totally bogus course. Senior scientists might squelch efforts to question some incorrect dogma. But in the long run those scientists die and those papers are forgotten. Rather than citing a 200 year old paper, some new researcher will rerun the experiment and hopefully see the mistake this time. If not, maybe in another 200 years.
Writing off a couple generations of a field of study is high cost, obviously we should strive for efficiency and better practices. But if you take a long view the process seems to be proceeding pretty cleanly.
Why worry so much worry too about isolated cases of outright fraud?
I subscribe to the opinion that most of the time we follow false leads, even when everyone involved is diligent.
Influential fraudsters are much rarer than influential honest scientists, so the risk is relatively low. Thing worth replicating will be replicated. If they are fraudsters, they will be uncovered, sooner or later. The time period depends on how influential they become.
So fraud seems to be a problem only for the non-influential members of academia, who run the risk of being outperformed by fraudsters. But if they are non-influential, you can also consider the outcome of their research to be “spam” (according to Suresh’s definition: http://geomblog.blogspot.com/2010/06/bad-research-as-spam.html).
Why worry so much worry too about isolated cases of outright fraud? (…) if they are non-influential, you can also consider the outcome of their research to be “spamâ€.
Because their work was not recognized as spam. Quite the contrary. To receive the Order of Canada for your research is quite an accomplishment that very, very few Canadian scientists will ever achieve.
Their work went straight into the “priority inbox”.
If they are fraudsters, they will be uncovered, sooner or later.
By the time Ranjit Chandra was found out, he was on his way to retirement. And most of his papers are still out there, they are still gathering citations. (Admittedly, much less than before.)
And the only reason we know about these frauds is that they were huge. Had these people merely “doctored” their results a little bit, it would have gone unnoticed.
Regarding Ranjit Chandra, it seems that the fraudulent research was uncovered within a year or so. The false findings were important enough for people to pay attention.
My point is that important fraud gets uncovered. If fraudulent results do not get uncovered this means that they were not important.
The fact that there was a false positive in giving the award is not a big deal, imho.
clvrmnkysays:
“Scientific fraud is a low-risk, high-reward activity.”
You have cherry picked three high-profile examples to reach this conclusion.
Can you actually demonstrate that, in general, this is actually the case?
I think your points are generally valid, and interesting. Laziness, business, group-think and all sorts of other human flaws make any work tend to have holes when scrutinized, and the effort require to find original source data, is very hard, in many cases. One might say everything is wrong, at some level.
Would you extrapolate to say that most peer reviewed science is wrong (enough to have flaws significant enough to invalidate the fundamental assertions?)
The point about citations, however, I think is frivolous, or maybe more accurately an example of your earlier point: citations imply rigor but are only of value if truly fact-checked to back to the original source.
That many people (especially bloggers, myself included) rely on second-hand sources may be a whole different issue indeed. That people believe claims based on some “trusted” source, especially complex ones they don’t have the skill to understand completely, is downright chilling.
I recently wrote a review on my blog of a book called Merchants of Doubt, which discusses how peer reviewed science has been systematically undermined by picking at details — the author strongly asserts that true peer review tends to produce valid results.
I think their case is pretty solid, rigorous and academic … but then again, I only followed about 3 of the literally thousands of citations and footnotes in the book, so perhaps I have been duped.
While I didn’t write about this in my post, I wonder what your take is on the issue in the case of this current book. Take a peek on my site and share your thoughts, if you like.
(…) the author strongly asserts that true peer review tends to produce valid results. (…) Would you extrapolate to say that most peer reviewed science is wrong (…)
Ah. But what is peer review? Is this the process by which you expose your results to your peers, including all the data and algorithms, and they get to tear it apart for years until everyone is convinced your work must be correct?
The conventional peer review process is not like that at all! You submit a paper to a journal. You do not include your data, your software, your lab books… just the paper which sometimes can be one or two pages long, with all details omitted. The journal then sends it to, say, two experts. How these experts are picked is often a black box. Maybe the journal picks friends of your theories, maybe the journal picks ennemies… who knows? Anonymously, these experts then produce a review of your work. This is not a point-by-point review. Sometimes the review is only one line or ten lines. I’ve once had a full page review, but this is a rarity… And these reviewers are anonymous, and these reviews remain hidden. Quite often, the expert did not grasp the paper as is apparent from the comments. So there is a fair amount of luck involved in the process. Moreover, the reviewers might be influenced by many secondary factors: does this help promote their own agenda?, are the authors well-known or from a well-known institution?, and so on. Then the journal takes a decision based on the reviews it gets back.
And once a paper is rejected, that is not definitive: the authors can resubmit to another journal and so on. Until the paper is eventually accepted. Now, with some luck, the reviewers were clever enough to see all of the flaws, point them out to the author, and the author fixed them all. But given that the reviewers do not get to see the data or the software, there is often quite a bit of trust involved. So, if you cheated, and you are smart, the reviewers may not be able to see it…
My point is that the current system is honor-based. We rely on the authors for not gaming the system. If they game the system, we have very little control over it. It is almost impossible for the reviewers to pick it up, and if they do, there is relatively little they can do… since the entire review process is hidden from view.
We could do better. The entire peer review process could be in the open. We could decide that not all papers are equal: a paper that underwent years of scrutiny in the open is more reliable. And so on.
I recently wrote a review on my blog of a book called Merchants of Doubt, which discusses how peer reviewed science has been systematically undermined by picking at details (…)
I would say that what this book describes is not science, but rather politics and the media. These people who casted doubts were addressing the media and politicians, not other scientists.
Tom Harrisonsays:
I see your points — obvious a system is only as good as the people behind it — things like honor, trust, diligence are required for peer-review to work properly.
In the case of Merchants of Doubt, while true that the scientists enlisted to undermine peer-reviewed science were used to manipulate press, they have created enough doubt that further, and further, even more diligent peer review has occurred (subjects include findings on cigarettes, ozone, acid rain, climate change) — perhaps this is the exact opposite situation as you describe! Or perhaps this is what should happen? It seems like there’s as much harm of having too little (or low quality) review, as there is having too much.
Like any human institution, it’s imperfect as best.
glad to have been introduced to you. i’ve been reading your posts all morning.
i’ve been in ed 20 years. what is troubling to me, is that we question very little about how we have always done things. i mean at the core.
ie: i’m listening to wise 2010 (world innovation summit of ed) -world leaders in ed are referring all possible improvements in public ed to pisa. and usually, when standardized tests are used to defend things, they say math is the most reliable means to compare countries.
well – what if we are doing math wrong?
solutions, they say, are to send teachers for more schooling in math, at schools that, according to pisa, etc, aren’t producing very well.
this irony seems obvious, but no one seems to address or notice it. and maybe that’s a good thing because i don’t think crafting schools for better pisa results is the answer.
again, my biggest worry, why are we relying on pisa in the first place?
we are basing public ed, which i believe if done right could be the vehicle to social change, on things most people don’t even believe in or desire. we’re just accepting the reliability clout of things like pisa’s following. why aren’t we bold enough to question what really matters, what people really want, what makes the most sense?
how can we get at the core? i’ll keep reading your stuff.. curious if you’ve already fluently addressed this.
thank you Daniel. looking forward to learning more from your insight. Erica McWilliams wrote a paper about being usefully ignorant. i feel that in your writing and questioning. i love it.
we are basing public ed, which i believe if done right could be the vehicle to social change, on things most people don’t even believe in or desire.
We founded public education on the need to train a generations of factory workers. You arrive on time. You value the standards. You memorize the time codes. You pay your taxes on time.
Public education is an industry, like any other. It provides jobs to people like me and you. It offers a service to society. Any industry is “good” as it enables certain things. Without the oil industry we couldn’t drive around in our cars. Without public education we wouldn’t have masses of standardized workers. Indeed, why do we even grade or categorize students to begin with? There is no serious evidence that grading students help them learn. If you think about it, the only reason we have to grade and rank students is as a service to employers. That is a fine service to render to human ressources departments. But we should be under no illusion that the goal is “social change”.
Public education has nothing to do with social change. If anything, like any large industry, it is a conservative force. Anyone who has worked in public education knows how much this is true.
Obviously, improving public education may cause social change because the graduates may get access to better jobs, which is obviously good. But, similarly, improving productivity in the oil industry may provide cheaper gaz which may improve the economic well being of the consumers.
The Web is the great social equalizer, not public education. Consider than any poor kids with an access to Internet has a comparable access to information than even the richest kids. Anyone interested in social change should be primarily concerned with the Web.
But we should be under no illusion that the goal is “social changeâ€.
i am under that illusion.
Public education has nothing to do with social change.
exactly my point. however, with the connections to people and info the web now allows, we can personalize public ed. that changes the game completely.
and we’re missing it.
and i’m not talking about:
..improving public education may cause social change because the graduates may get access to better jobs, which is obviously good.
i’m talking about school becoming life. any learner, any age, using life as their lessons. currently 12+ years, 7 hours a day, with our most valuable resource, people. it’s seems silly to me that all we think we can gain from that is a good job. what if life isn’t about getting jobs as much as it is about living it. doing what matters to an individual, and their respective community.
i’ve seen both of those videos.
via Toffler – school resembles the factory. but today i’d say – school can and should resemble life. we now have the tech that allows working through messy real problems as opposed to pseudo problems that fit nicely with ie: algebra. again – with the web – we can move beyond factory prep capabilities.
via Gates – what if the learners are completely immersed, not because they are spending 10 hour days in a program, but because what they are learning is actual life and not a set curricula, determined by someone else, that they may or may not have any need for.
the web is the great equalizer. that’s why i believe so much in digital equity and connect with people like Kosta Grammatis http://www.youtube.com/watch?v=ZT3NBbD_ml8
today you can choose to do/learn whatever you want, when, where, how, and with whom. but 3/4 of the world doesn’t get to play. yet.
public ed is packed with some great resources, namely people. if we mash-up the great resources in public ed and the capabilities the web now allows. if learners now truly can own their learning. if Clay Shirky’s cognitive surplus grows as exponentially as it has the potential to do so. if we could just give ourselves and others permission and time and space to own our learning..
how could that not change a society. for good.
Publishing is for people working at universities. If you work for a private company, and find some interesting results, you cannot publish, you do not want the competition to know it. You just have to really make sure you validated your own work, because management will make decisions based on it and the outcome has to be what you predicted or you are in trouble !
@AnnMaria
People who are not in academia don’t need to play this game so they just don’t submit their research (…)
Good point.
Raising the barrier to entry does prevent people from outside the system from playing a useful role.
That is why I seek to promote open scholarship (as opposed to mere “open access”). I think that the world would be a better place if more people pursued scholarship and published their findings. Limiting scholarship to professional researchers is counterproductive.
@Zyxo
You just have to really make sure you validated your own work, because management will make decisions based on it and the outcome has to be what you predicted or you are in trouble !
Good point. Putting your work into practice, where everyone can see it, is good validation.
Publishing is for people working at universities. If you work for a private company, and find some interesting results, you cannot publish, you do not want the competition to know it.
Ah. But you do release it eventually, where everyone can see it, or reverse engineer it, don’t you?
@ JeffE
Wait… what exactly do you expect? That scientists will retroactively remove citations to fraudulent papers in their own published work?
I use this as evidence that you cannot rely on citations to determine the reliability of the work.
He had a lot more citations before the papers were pulled out. These are the citations that remain on the papers that were not removed from Scopus.
And how many of those citations followed the word “fraudulent�
He was highly cited before getting caught. Since then, he has gathered far fewer citations.
If, as you say, citations are not validation, then why are citations to fraudulent papers objectionable?
They are evidence that people will gladly build on doctored papers.
“He lost his research job, but found an engineering position in Germany.†(…) Are you suggesting that he should not hold an engineering position? What sort of job should he be allowed to hold, then?
At least in Quebec (Canada) where I live, it would not be possible to remain a certified engineer following such a high profile fraud. Or rather, I would hope not.
Engineers are supposed to be able to sign off on public safety issues, for example. Making up data for years is a pretty serious breach of ethics. I’d say it is incompatible with an engineering job.
I completely agree. You also neglected that peer review is not nearly as anonymous as we pretend. If am reviewing a paper and it has Lemire(in press) cited, I know that it must be by either Lemire or one of his students or colleagues. If Lemire is very important in my field than I am going to think twice about rejecting his article.
I have been a reviewer for journals that strongly encouraged us to reject papers with minor problems with a “Revised and resubmit” decision so we could keep our reject rate high and look selective.
People who are not in academia don’t need to play this game so they just don’t submit their research many times.
There are a lot of problems with peer reviewed publications.
“Despite all the retracted papers, Jan Hendrik Schön has still 1,200 citations according to Scopus.”
Wait… what exactly do you expect? That scientists will retroactively remove citations to fraudulent papers in their own published work? How do you suggest publishers and citation indexers maintain those changes?
And how many of those citations followed the word “fraudulent”?
If, as you say, citations are not validation, then why are citations to fraudulent papers objectionable?
“He lost his research job, but found an engineering position in Germany.”
Are you suggesting that he should not hold an engineering position? What sort of job should he be allowed to hold, then?
@Zyxo You can publish. I work for a private company. We occasionally publish work after the project is complete, but not that often. The reason isn’t that we are afraid people will steal our ideas. It’s that the benefit to us of publishing is not worth the time we take away from away from paid work to jump through the hoops.
@Lemire I’d like to see both open access and open scholarship. There have been other discussions about the illogic of paying journals for articles reviewed (for free) that were written (for free) by researchers using research conducted with government funds.
That, though, is a topic for another day
@Misha
Of course, you can get promoted making up data, but if it becomes known, your reputation will be gone.
That’s what we would hope. And certainly, if you go far enough, that is what happens.
But the problem I’d like to underline is that there are many small ways to cheat and mislead. By assuming that science is self-correcting, we are lead to think that cheating is no big deal. Ah! But it is… I think… because people are going to assume that whatever has been published is reliable.
For example, the climate-gate events showed that a lot of the global warming science is based on a few measures that were doctored. Nobody checked this data, nobody asked to see the details of how it was processed… and people built on it. There is no doubt that global warming is real, but a lot of the science might be misleading… maybe only in small ways… but… how do we know?
Why is it that only a few people had access to the raw climate data? Why wasn’t this raw data made widely available? Why is it that no journal reviewer asked to see the raw data?
Hi Daniel, I like your post.
I would like to point out that important parts of science are confidence and credit. Of course, you can get promoted making up data, but if it becomes known, your reputation will be gone. Of course, it may happen that you keep your former degrees and rewards, and get a decent job, but how you imagine talking to people after that? No one will ever trust you and consider you seriously.
Another thing is that the science system as it is now strongly encourages misconduct. It doesn’t matter how good your results are if they aren’t published in one of the top journals. But you referred to this subject many times in your blog.
Misha
@Paul
(…) if you take a long view the process seems to be proceeding pretty cleanly.
“The long run is a misleading guide to current affairs. In the long run we are all dead”
The amazing thing is that the process still works. Who knows how many errors and deceptions go uncaught, how many citations lead to bogus facts? And yet our knowledge keeps growing.
In one sense you can extrapolate arbitrarily far from reality as the result of a small error, but in practice scientists do seem to notice when a field diverges significantly. Nobody checked the Climategate data specifically, but plenty of other data was being gathered. If it had been at complete odds with everything else, we’d have noticed.
I think forgetfulness plays a critical role in the process. At worst a whole field might pursue a totally bogus course. Senior scientists might squelch efforts to question some incorrect dogma. But in the long run those scientists die and those papers are forgotten. Rather than citing a 200 year old paper, some new researcher will rerun the experiment and hopefully see the mistake this time. If not, maybe in another 200 years.
Writing off a couple generations of a field of study is high cost, obviously we should strive for efficiency and better practices. But if you take a long view the process seems to be proceeding pretty cleanly.
Why worry so much worry too about isolated cases of outright fraud?
I subscribe to the opinion that most of the time we follow false leads, even when everyone involved is diligent.
Influential fraudsters are much rarer than influential honest scientists, so the risk is relatively low. Thing worth replicating will be replicated. If they are fraudsters, they will be uncovered, sooner or later. The time period depends on how influential they become.
So fraud seems to be a problem only for the non-influential members of academia, who run the risk of being outperformed by fraudsters. But if they are non-influential, you can also consider the outcome of their research to be “spam” (according to Suresh’s definition: http://geomblog.blogspot.com/2010/06/bad-research-as-spam.html).
@Panos
Why worry so much worry too about isolated cases of outright fraud? (…) if they are non-influential, you can also consider the outcome of their research to be “spamâ€.
Because their work was not recognized as spam. Quite the contrary. To receive the Order of Canada for your research is quite an accomplishment that very, very few Canadian scientists will ever achieve.
Their work went straight into the “priority inbox”.
If they are fraudsters, they will be uncovered, sooner or later.
By the time Ranjit Chandra was found out, he was on his way to retirement. And most of his papers are still out there, they are still gathering citations. (Admittedly, much less than before.)
And the only reason we know about these frauds is that they were huge. Had these people merely “doctored” their results a little bit, it would have gone unnoticed.
Regarding Ranjit Chandra, it seems that the fraudulent research was uncovered within a year or so. The false findings were important enough for people to pay attention.
My point is that important fraud gets uncovered. If fraudulent results do not get uncovered this means that they were not important.
The fact that there was a false positive in giving the award is not a big deal, imho.
“Scientific fraud is a low-risk, high-reward activity.”
You have cherry picked three high-profile examples to reach this conclusion.
Can you actually demonstrate that, in general, this is actually the case?
I might seem impolite, or perhaps even a spammer 🙂 , but you might have a look at http://www.readersourcing.org , and specifically at http://www.readersourcing.org/docs .
@Micah
Yes, fraud is just an extreme case of “being wrong”. I’ve covered this topic before:
Why senior researchers and managers should analyze data themselves…
Most scientific papers are probably wrong
When in doubts, prefer unimpressive negative results
This topic of fraud in scientific research is interesting, but there is a growing concern that most research results are actually false.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/
Daniel —
I think your points are generally valid, and interesting. Laziness, business, group-think and all sorts of other human flaws make any work tend to have holes when scrutinized, and the effort require to find original source data, is very hard, in many cases. One might say everything is wrong, at some level.
Would you extrapolate to say that most peer reviewed science is wrong (enough to have flaws significant enough to invalidate the fundamental assertions?)
The point about citations, however, I think is frivolous, or maybe more accurately an example of your earlier point: citations imply rigor but are only of value if truly fact-checked to back to the original source.
That many people (especially bloggers, myself included) rely on second-hand sources may be a whole different issue indeed. That people believe claims based on some “trusted” source, especially complex ones they don’t have the skill to understand completely, is downright chilling.
I recently wrote a review on my blog of a book called Merchants of Doubt, which discusses how peer reviewed science has been systematically undermined by picking at details — the author strongly asserts that true peer review tends to produce valid results.
I think their case is pretty solid, rigorous and academic … but then again, I only followed about 3 of the literally thousands of citations and footnotes in the book, so perhaps I have been duped.
While I didn’t write about this in my post, I wonder what your take is on the issue in the case of this current book. Take a peek on my site and share your thoughts, if you like.
@Harrison
(…) the author strongly asserts that true peer review tends to produce valid results. (…) Would you extrapolate to say that most peer reviewed science is wrong (…)
Ah. But what is peer review? Is this the process by which you expose your results to your peers, including all the data and algorithms, and they get to tear it apart for years until everyone is convinced your work must be correct?
The conventional peer review process is not like that at all! You submit a paper to a journal. You do not include your data, your software, your lab books… just the paper which sometimes can be one or two pages long, with all details omitted. The journal then sends it to, say, two experts. How these experts are picked is often a black box. Maybe the journal picks friends of your theories, maybe the journal picks ennemies… who knows? Anonymously, these experts then produce a review of your work. This is not a point-by-point review. Sometimes the review is only one line or ten lines. I’ve once had a full page review, but this is a rarity… And these reviewers are anonymous, and these reviews remain hidden. Quite often, the expert did not grasp the paper as is apparent from the comments. So there is a fair amount of luck involved in the process. Moreover, the reviewers might be influenced by many secondary factors: does this help promote their own agenda?, are the authors well-known or from a well-known institution?, and so on. Then the journal takes a decision based on the reviews it gets back.
And once a paper is rejected, that is not definitive: the authors can resubmit to another journal and so on. Until the paper is eventually accepted. Now, with some luck, the reviewers were clever enough to see all of the flaws, point them out to the author, and the author fixed them all. But given that the reviewers do not get to see the data or the software, there is often quite a bit of trust involved. So, if you cheated, and you are smart, the reviewers may not be able to see it…
My point is that the current system is honor-based. We rely on the authors for not gaming the system. If they game the system, we have very little control over it. It is almost impossible for the reviewers to pick it up, and if they do, there is relatively little they can do… since the entire review process is hidden from view.
We could do better. The entire peer review process could be in the open. We could decide that not all papers are equal: a paper that underwent years of scrutiny in the open is more reliable. And so on.
I recently wrote a review on my blog of a book called Merchants of Doubt, which discusses how peer reviewed science has been systematically undermined by picking at details (…)
I would say that what this book describes is not science, but rather politics and the media. These people who casted doubts were addressing the media and politicians, not other scientists.
I see your points — obvious a system is only as good as the people behind it — things like honor, trust, diligence are required for peer-review to work properly.
In the case of Merchants of Doubt, while true that the scientists enlisted to undermine peer-reviewed science were used to manipulate press, they have created enough doubt that further, and further, even more diligent peer review has occurred (subjects include findings on cigarettes, ozone, acid rain, climate change) — perhaps this is the exact opposite situation as you describe! Or perhaps this is what should happen? It seems like there’s as much harm of having too little (or low quality) review, as there is having too much.
Like any human institution, it’s imperfect as best.
glad to have been introduced to you. i’ve been reading your posts all morning.
i’ve been in ed 20 years. what is troubling to me, is that we question very little about how we have always done things. i mean at the core.
ie: i’m listening to wise 2010 (world innovation summit of ed) -world leaders in ed are referring all possible improvements in public ed to pisa. and usually, when standardized tests are used to defend things, they say math is the most reliable means to compare countries.
well – what if we are doing math wrong?
solutions, they say, are to send teachers for more schooling in math, at schools that, according to pisa, etc, aren’t producing very well.
this irony seems obvious, but no one seems to address or notice it. and maybe that’s a good thing because i don’t think crafting schools for better pisa results is the answer.
again, my biggest worry, why are we relying on pisa in the first place?
we are basing public ed, which i believe if done right could be the vehicle to social change, on things most people don’t even believe in or desire. we’re just accepting the reliability clout of things like pisa’s following. why aren’t we bold enough to question what really matters, what people really want, what makes the most sense?
how can we get at the core? i’ll keep reading your stuff.. curious if you’ve already fluently addressed this.
thank you Daniel. looking forward to learning more from your insight. Erica McWilliams wrote a paper about being usefully ignorant. i feel that in your writing and questioning. i love it.
@hardy
we are basing public ed, which i believe if done right could be the vehicle to social change, on things most people don’t even believe in or desire.
We founded public education on the need to train a generations of factory workers. You arrive on time. You value the standards. You memorize the time codes. You pay your taxes on time.
See :
Alvin Toffler on Education
http://www.youtube.com/watch?v=04AhBnLk1-s
Public education is an industry, like any other. It provides jobs to people like me and you. It offers a service to society. Any industry is “good” as it enables certain things. Without the oil industry we couldn’t drive around in our cars. Without public education we wouldn’t have masses of standardized workers. Indeed, why do we even grade or categorize students to begin with? There is no serious evidence that grading students help them learn. If you think about it, the only reason we have to grade and rank students is as a service to employers. That is a fine service to render to human ressources departments. But we should be under no illusion that the goal is “social change”.
Public education has nothing to do with social change. If anything, like any large industry, it is a conservative force. Anyone who has worked in public education knows how much this is true.
Obviously, improving public education may cause social change because the graduates may get access to better jobs, which is obviously good. But, similarly, improving productivity in the oil industry may provide cheaper gaz which may improve the economic well being of the consumers.
The Web is the great social equalizer, not public education. Consider than any poor kids with an access to Internet has a comparable access to information than even the richest kids. Anyone interested in social change should be primarily concerned with the Web.
See:
Bill Gates on in-person vs. online education
http://www.youtube.com/watch?v=p2Qg80MVvYs
But we should be under no illusion that the goal is “social changeâ€.
i am under that illusion.
Public education has nothing to do with social change.
exactly my point. however, with the connections to people and info the web now allows, we can personalize public ed. that changes the game completely.
and we’re missing it.
and i’m not talking about:
..improving public education may cause social change because the graduates may get access to better jobs, which is obviously good.
i’m talking about school becoming life. any learner, any age, using life as their lessons. currently 12+ years, 7 hours a day, with our most valuable resource, people. it’s seems silly to me that all we think we can gain from that is a good job. what if life isn’t about getting jobs as much as it is about living it. doing what matters to an individual, and their respective community.
i’ve seen both of those videos.
via Toffler – school resembles the factory. but today i’d say – school can and should resemble life. we now have the tech that allows working through messy real problems as opposed to pseudo problems that fit nicely with ie: algebra. again – with the web – we can move beyond factory prep capabilities.
via Gates – what if the learners are completely immersed, not because they are spending 10 hour days in a program, but because what they are learning is actual life and not a set curricula, determined by someone else, that they may or may not have any need for.
the web is the great equalizer. that’s why i believe so much in digital equity and connect with people like Kosta Grammatis http://www.youtube.com/watch?v=ZT3NBbD_ml8
today you can choose to do/learn whatever you want, when, where, how, and with whom. but 3/4 of the world doesn’t get to play. yet.
public ed is packed with some great resources, namely people. if we mash-up the great resources in public ed and the capabilities the web now allows. if learners now truly can own their learning. if Clay Shirky’s cognitive surplus grows as exponentially as it has the potential to do so. if we could just give ourselves and others permission and time and space to own our learning..
how could that not change a society. for good.