Is it a matter of having the right incentives for researchers? Instead of “publish or perish”, “publish, produce, share and/or teach”?
An example of a researcher having a wide impact is Conrad Sanderson through his C++ Armadillo library. I suspect producing this software while also managing to publish extensively in peer reviewed journals requires a combination of dedication and ability that goes beyond the capability of most people.
If producing counted as much as writing papers, then this model would be more accessible to the average researcher who may not otherwise be capable of publishing enough papers.
Will the academic world naturally move away from reviewed publication as the primary measure of performance as the culture changes, or is reform needed?
I don’t know Conrad Sanderson but I would not presume that working on software necessarily slows down his academic work. The thing is… productivity is not a zero-sum game.
Will the academic world naturally move away from reviewed publication as the primary measure of performance as the culture changes, or is reform needed?
Ultimately, it is a resource allocation problem. There are only so many research grants, jobs and so forth. People only have so much time. So people are pushed to certain things, and they are discouraged from doing others. No system is even perfect, but we have to somehow decide that some people should be funded before other people. There is no way not to make this choice. I don’t think that using publications as a measure of performance is necessarily bad. I don’t think that people refuse to share data and software so that they can publish more (though this might be their excuse), it is all deeply rooted in the culture.
The problem we are facing here is that Conboy et al. felt that it was in their own best self-interest to delay by about ten years work that looks to me to be highly important for all of us. They could have published their work elsewhere, but they chose not to do so. Instead, they express frustration at the fact that prestigious venues would not publish their work. Clearly, they held on to their work and tried to get it published in prestigious venues but they kept failing. I understand their frustration well. At the same time, they could have taken their work to PLoS Biology or some other alternative journal. Or heck! They could have posted the work as a technical report on their home page. They make it seem like their work would then not have counted. This might have been reasonable from their point of view… but… I think it is less and less reasonable.
It is getting harder and harder to stand by the belief that unless works appear in a highly prestigious venue, it does not count. That’s no accident. Smart and talented people have invested much of their time in the last ten years to make it happen.
And this matters very much because it means that it is less likely today that people like Conboy will hold on to important work in the hope of scoring a prestigious paper. I am not saying it won’t happen… it still does all the time… but we have made progress. And it is progress that matters very much.
A well done peer review is very useful. However, this is becoming a rare commodity as reviewer received (almost) no reward for doing an honest peer review.
The other important point is that MOST paper in biomedical science are wrong, because of the poor research practices. It would be even worst without peer review to filter the field. The corollary is that you can always manage to publish a paper if you are ready to move to another less prestigious journal.
The quest for prestige and the quest for truth are different things.
In the open-source world, anyone can claim to have software that does miraculous things. And indeed, people make extravagant claims all the time. Yet our civilization has come to rely critically on open-source software. We have no need for the equivalent of Nature. Do we have peer review? Oh yes! We do. The reason Linux is being used for mission-critical purposes is that there is an extensive review process of each and every contribution… a process that is far more thorough than the process of submitting a paper to Nature.
Science should be vetted in a similar manner. Before I can trust your work, I need to see extensive peer review of it… But I should not care if some powerful editor vet it or not.
I agree. But, unless you have a strong mechanism that enforce good vs bad work, you are likely to end up with a selection process that enforce popular vs impopular. This is how the pseudoscience work by the way.
Open source softwares are ongoing a permanent testing process. This is not the case of most scientific papers, because they are very hard to replicate.
A large fraction of the scientific controversy a simple product of poor research practice. And, outside the scientific community, users of the scientific knowledge have very little opportunity to test it.
Open source softwares are ongoing a permanent testing process. This is not the case of most scientific papers, because they are very hard to replicate.
Yes. And that is exactly the problem we must solve.
The old incentives in science were almost backward. You wanted your competitors to have a hard time reproducing your results, precisely so that you alone could pursue them (and to make sure they could not contradict you). Given that nobody can reproduce your work how do they trust you? Because you have written your work in such a way that people in high places will like it and recommend that it be accepted in a prestigious venue. It is not quite as bad as all that, but some of what I wrote is true.
It is really a pre-scientific, authority-based process.
Think about it… conceptually, there is no difference between accepting a truth because it comes from the Pope and accepting a truth because it comes from Nature.
Scientists should shy away from reasoning on the basis of authority.
The whole idea that without prestigious venues we would not know what to believe is just sad.
From my own experience, peer review is significantly reduce the number of rubbish results published. Overall, I reject roughly 50% of the paper I review because they are pure garbage. Hence peer review is certainly useful.
As a researcher, I seek references in Google Scholar, and whether it appeared in a pretigious venue or just arXiv is not a factor in my decision to check out a reference.
What traditional peer review is good at is filtering work that is obviously wrong. But that’s hardly our problem, is it?
Traditional peer review will not and cannot protect you from people who make up data, either intentionally or not. The only protection we have against that is people going “hmmm… funny, this does not work here… why is that?” And then, with a twist of irony, traditional peer review makes it impossible to publish a paper saying that “Smith et al. got it wrong, we can’t reproduce their work” (you hardly ever see such papers as they are tremendously difficult to get beyond peer review).
Personally, I have seen many papers saying that article of X is wrong. I written that a few time myself.
Peer review is not the cause of the lack of reproducibility. Research practice is.
Richard David Stafford, Ph.D.says:
Peer review essentially stopped me dead in the waters. I have a Ph.D. in theoretical physics but have been blocked by peer review on every attempt. I went into physics because physicists were the only teachers who would make a serious attempt to explain their views (my desire was to understand the universe I found myself in). One of the responses I often received was “you will have to find someone more educated than I as I really don’t understand the explanation.”
So I went to graduate school to learn the foundations of their beliefs. I am now approaching 80 years and I have never met a person with an advanced education who had any interest in thinking about their beliefs. They are all interested in “being authorities”. I have come to be convinced that modern science is a religion and not a science.
I would love to encounter someone interested in understanding the universe. I could show them things they have never even considered which provide answers beyond common belief.
Einstein’s theories of “space-time” are as erroneous as belief in a flat earth!
Great article.
Is it a matter of having the right incentives for researchers? Instead of “publish or perish”, “publish, produce, share and/or teach”?
An example of a researcher having a wide impact is Conrad Sanderson through his C++ Armadillo library. I suspect producing this software while also managing to publish extensively in peer reviewed journals requires a combination of dedication and ability that goes beyond the capability of most people.
If producing counted as much as writing papers, then this model would be more accessible to the average researcher who may not otherwise be capable of publishing enough papers.
Will the academic world naturally move away from reviewed publication as the primary measure of performance as the culture changes, or is reform needed?
I don’t know Conrad Sanderson but I would not presume that working on software necessarily slows down his academic work. The thing is… productivity is not a zero-sum game.
Will the academic world naturally move away from reviewed publication as the primary measure of performance as the culture changes, or is reform needed?
Ultimately, it is a resource allocation problem. There are only so many research grants, jobs and so forth. People only have so much time. So people are pushed to certain things, and they are discouraged from doing others. No system is even perfect, but we have to somehow decide that some people should be funded before other people. There is no way not to make this choice. I don’t think that using publications as a measure of performance is necessarily bad. I don’t think that people refuse to share data and software so that they can publish more (though this might be their excuse), it is all deeply rooted in the culture.
The problem we are facing here is that Conboy et al. felt that it was in their own best self-interest to delay by about ten years work that looks to me to be highly important for all of us. They could have published their work elsewhere, but they chose not to do so. Instead, they express frustration at the fact that prestigious venues would not publish their work. Clearly, they held on to their work and tried to get it published in prestigious venues but they kept failing. I understand their frustration well. At the same time, they could have taken their work to PLoS Biology or some other alternative journal. Or heck! They could have posted the work as a technical report on their home page. They make it seem like their work would then not have counted. This might have been reasonable from their point of view… but… I think it is less and less reasonable.
It is getting harder and harder to stand by the belief that unless works appear in a highly prestigious venue, it does not count. That’s no accident. Smart and talented people have invested much of their time in the last ten years to make it happen.
And this matters very much because it means that it is less likely today that people like Conboy will hold on to important work in the hope of scoring a prestigious paper. I am not saying it won’t happen… it still does all the time… but we have made progress. And it is progress that matters very much.
A well done peer review is very useful. However, this is becoming a rare commodity as reviewer received (almost) no reward for doing an honest peer review.
The other important point is that MOST paper in biomedical science are wrong, because of the poor research practices. It would be even worst without peer review to filter the field. The corollary is that you can always manage to publish a paper if you are ready to move to another less prestigious journal.
The quest for prestige and the quest for truth are different things.
In the open-source world, anyone can claim to have software that does miraculous things. And indeed, people make extravagant claims all the time. Yet our civilization has come to rely critically on open-source software. We have no need for the equivalent of Nature. Do we have peer review? Oh yes! We do. The reason Linux is being used for mission-critical purposes is that there is an extensive review process of each and every contribution… a process that is far more thorough than the process of submitting a paper to Nature.
Science should be vetted in a similar manner. Before I can trust your work, I need to see extensive peer review of it… But I should not care if some powerful editor vet it or not.
I agree. But, unless you have a strong mechanism that enforce good vs bad work, you are likely to end up with a selection process that enforce popular vs impopular. This is how the pseudoscience work by the way.
Open source softwares are ongoing a permanent testing process. This is not the case of most scientific papers, because they are very hard to replicate.
A large fraction of the scientific controversy a simple product of poor research practice. And, outside the scientific community, users of the scientific knowledge have very little opportunity to test it.
Open source softwares are ongoing a permanent testing process. This is not the case of most scientific papers, because they are very hard to replicate.
Yes. And that is exactly the problem we must solve.
The old incentives in science were almost backward. You wanted your competitors to have a hard time reproducing your results, precisely so that you alone could pursue them (and to make sure they could not contradict you). Given that nobody can reproduce your work how do they trust you? Because you have written your work in such a way that people in high places will like it and recommend that it be accepted in a prestigious venue. It is not quite as bad as all that, but some of what I wrote is true.
It is really a pre-scientific, authority-based process.
Think about it… conceptually, there is no difference between accepting a truth because it comes from the Pope and accepting a truth because it comes from Nature.
Scientists should shy away from reasoning on the basis of authority.
The whole idea that without prestigious venues we would not know what to believe is just sad.
From my own experience, peer review is significantly reduce the number of rubbish results published. Overall, I reject roughly 50% of the paper I review because they are pure garbage. Hence peer review is certainly useful.
As a researcher, I seek references in Google Scholar, and whether it appeared in a pretigious venue or just arXiv is not a factor in my decision to check out a reference.
What traditional peer review is good at is filtering work that is obviously wrong. But that’s hardly our problem, is it?
Traditional peer review will not and cannot protect you from people who make up data, either intentionally or not. The only protection we have against that is people going “hmmm… funny, this does not work here… why is that?” And then, with a twist of irony, traditional peer review makes it impossible to publish a paper saying that “Smith et al. got it wrong, we can’t reproduce their work” (you hardly ever see such papers as they are tremendously difficult to get beyond peer review).
Personally, I have seen many papers saying that article of X is wrong. I written that a few time myself.
Peer review is not the cause of the lack of reproducibility. Research practice is.
Peer review essentially stopped me dead in the waters. I have a Ph.D. in theoretical physics but have been blocked by peer review on every attempt. I went into physics because physicists were the only teachers who would make a serious attempt to explain their views (my desire was to understand the universe I found myself in). One of the responses I often received was “you will have to find someone more educated than I as I really don’t understand the explanation.”
So I went to graduate school to learn the foundations of their beliefs. I am now approaching 80 years and I have never met a person with an advanced education who had any interest in thinking about their beliefs. They are all interested in “being authorities”. I have come to be convinced that modern science is a religion and not a science.
I would love to encounter someone interested in understanding the universe. I could show them things they have never even considered which provide answers beyond common belief.
Einstein’s theories of “space-time” are as erroneous as belief in a flat earth!
I agree with the spirit of your comment.