I’ve worked in clusters of incompetency before – but by the same token, I’m no genius. How to encourage young scientists? (I feel a little patronising already). The academic culture sucks big time, imho – student has loads of pressure to turn out some script Just So. Not that I can see a way through to something better.
Given a US-style setup, maybe you could allow the student to follow through their research into the enterprise world, without gaps. My (very English) gut says that sounds horrible. (Pure Maths? Enron.)
Peer review isn’t designed to promote genius, but rather to support incremental progress and keep out cranks. Unfortunately, the establishment has difficulty spotting the occasional genius in an sea of cranks. Other marketplaces work differently–looks at the successful sales of “alternative” medicine hucksters who are unencumbered by peer review.
You’ve pointed out that we as researchers should not depend on peer review for validation of our work. That strikes me as the right approach: trust our own instincts as to what is valuable, and then participate in the available marketplaces of ideas, each having its own rules of engagement.
The first is that we cannot possibly expect everyone make a unique judgment on the capabilities of every person they come across. This is why people rely on the proxies variables, such as where a person graduated from, who they have co-published with, the grants they have secured, and indeed which university are they working at. This is the reality for today’s time-pressed academic.
However, having said that, it would be an exercise in laziness to accept the point I made in the previous paragraph. We owe a duty to ourselves and to others to work with our colleagues to help them improve their work (be it by suggesting avenues of work, giving feedback or pointing out shortcoming [which academics HAVE to do, and so often that it would break other form of social relationships].
Dan,
I am a peer reviewer for JOLT.I chose to do so becasue I would be exposed to new ideas. I sometimes have personal research or experience that I sometimes use to make a point when I ask for clarification. I usually use the opportunity to expose the article author to other people that I am aware of that may assist them with their arguments. In other words, I am acting as a connector between two networks in the George Siemens and Stephen Downes theory of Connectivism and Connected Learning.
People that criticize peer reviewed journals though usually do not offer up an alternative to the model. It would appear that blogs are the only viable alternative where comments can provide a feedback loop. While feedback is useful, how wide can the readership be. If all of readers used Google Alerts for the tags that interested us, we would become much more aware of what is being published.
I wonder if Journals shouldn’t encourage authors to set up blogs or even better yet wikis sites to write their article and get others to contribute. When the article no longer generates any more commentary, it is “fit” to be published as part of a hard copy or collection of other articles. In any event,the article is available to the public throughout the initial review cycles.
@Bradley I do not advocate we stop reviewing papers. You learn a lot by writing critical reviews. And the authors benefit tremendously when it is done right.
WHAT EXACTLY IS THE PROBLEM AND THE PURPORTED SOLUTION?
EWD: “Not only does the mechanism of peer review fail to protect us from disasters, in a certain way it guarantees mediocrity: the genius has no peers.”
(1) The question is not whether peer review fails to protect us from all disasters, just whether it protects as significantly better than an alternative.
(2) Yes, there is always regression on the mean with human judgment. That is why there is a hierarchy of quality levels among journals, and why editors (try to) pick same-level peers rather than drones to evaluate a given piece of research.
(3) Evaluating the work of a field’s handful of peerless geniuses in a generation are not the problem: evaluating everything else is.
EWD: “At the time it is done, truly original work… is very hard to publish as it takes at least another ten years for the appropriate journal to be founded.
It would have been nice to see the evidence supporting this assertion.
EWD: “The moral is that we cannot delegate our responsibility to judge ourself”
Since EWD has never had to deal with an unrefereed research corpus, this assertion too seems to have no empirical (or practical) basis.
EWD: “A good young scientist is able to explain what he is trying to achieve, why he is tackling this in the way he is, why he believes he can do it, the criterion by which he will decide whether he has succeeded or failed”
Yes, and the point it… ?
There are too many scientists, good, bad, indifferent and variable, to assume that all are good a priori.
And the judgment of whether or not a piece of research is sound, clear and new is what qualified experts are needed for (so that every researcher doesn’t have to try to judge every piece of research for himself).
EWD: “If we are too lazy or too stupid to follow such an explanation, we should resign.”
I can’t follow this explanation…
EWD: “By urging young scientists to submit papers for publication and to apply for grants so that we can rely on the judgements of others we make ourselves ridiculous.”
So researchers should be posting whatever they like (and unfunded? or funded on the assumption that everyone is “a good young scientist”).
It is not clear what EWD’s is complaining about, or trying to fix, or how…
arnad, S. (1998/2000/2004) The invisible hand of peer review. Nature [online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B. (2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242. http://cogprints.org/1646/
I’ve worked in clusters of incompetency before – but by the same token, I’m no genius. How to encourage young scientists? (I feel a little patronising already). The academic culture sucks big time, imho – student has loads of pressure to turn out some script Just So. Not that I can see a way through to something better.
Given a US-style setup, maybe you could allow the student to follow through their research into the enterprise world, without gaps. My (very English) gut says that sounds horrible. (Pure Maths? Enron.)
Good points anyhow, dude.
Peer review isn’t designed to promote genius, but rather to support incremental progress and keep out cranks. Unfortunately, the establishment has difficulty spotting the occasional genius in an sea of cranks. Other marketplaces work differently–looks at the successful sales of “alternative” medicine hucksters who are unencumbered by peer review.
You’ve pointed out that we as researchers should not depend on peer review for validation of our work. That strikes me as the right approach: trust our own instincts as to what is valuable, and then participate in the available marketplaces of ideas, each having its own rules of engagement.
There are two elements to this.
The first is that we cannot possibly expect everyone make a unique judgment on the capabilities of every person they come across. This is why people rely on the proxies variables, such as where a person graduated from, who they have co-published with, the grants they have secured, and indeed which university are they working at. This is the reality for today’s time-pressed academic.
However, having said that, it would be an exercise in laziness to accept the point I made in the previous paragraph. We owe a duty to ourselves and to others to work with our colleagues to help them improve their work (be it by suggesting avenues of work, giving feedback or pointing out shortcoming [which academics HAVE to do, and so often that it would break other form of social relationships].
We should live up to Dijkstr’s high expectations.
Dan,
I am a peer reviewer for JOLT.I chose to do so becasue I would be exposed to new ideas. I sometimes have personal research or experience that I sometimes use to make a point when I ask for clarification. I usually use the opportunity to expose the article author to other people that I am aware of that may assist them with their arguments. In other words, I am acting as a connector between two networks in the George Siemens and Stephen Downes theory of Connectivism and Connected Learning.
People that criticize peer reviewed journals though usually do not offer up an alternative to the model. It would appear that blogs are the only viable alternative where comments can provide a feedback loop. While feedback is useful, how wide can the readership be. If all of readers used Google Alerts for the tags that interested us, we would become much more aware of what is being published.
I wonder if Journals shouldn’t encourage authors to set up blogs or even better yet wikis sites to write their article and get others to contribute. When the article no longer generates any more commentary, it is “fit” to be published as part of a hard copy or collection of other articles. In any event,the article is available to the public throughout the initial review cycles.
@Bradley I do not advocate we stop reviewing papers. You learn a lot by writing critical reviews. And the authors benefit tremendously when it is done right.
WHAT EXACTLY IS THE PROBLEM AND THE PURPORTED SOLUTION?
EWD: “Not only does the mechanism of peer review fail to protect us from disasters, in a certain way it guarantees mediocrity: the genius has no peers.”
(1) The question is not whether peer review fails to protect us from all disasters, just whether it protects as significantly better than an alternative.
(2) Yes, there is always regression on the mean with human judgment. That is why there is a hierarchy of quality levels among journals, and why editors (try to) pick same-level peers rather than drones to evaluate a given piece of research.
(3) Evaluating the work of a field’s handful of peerless geniuses in a generation are not the problem: evaluating everything else is.
EWD: “At the time it is done, truly original work… is very hard to publish as it takes at least another ten years for the appropriate journal to be founded.
It would have been nice to see the evidence supporting this assertion.
EWD: “The moral is that we cannot delegate our responsibility to judge ourself”
Since EWD has never had to deal with an unrefereed research corpus, this assertion too seems to have no empirical (or practical) basis.
EWD: “A good young scientist is able to explain what he is trying to achieve, why he is tackling this in the way he is, why he believes he can do it, the criterion by which he will decide whether he has succeeded or failed”
Yes, and the point it… ?
There are too many scientists, good, bad, indifferent and variable, to assume that all are good a priori.
And the judgment of whether or not a piece of research is sound, clear and new is what qualified experts are needed for (so that every researcher doesn’t have to try to judge every piece of research for himself).
EWD: “If we are too lazy or too stupid to follow such an explanation, we should resign.”
I can’t follow this explanation…
EWD: “By urging young scientists to submit papers for publication and to apply for grants so that we can rely on the judgements of others we make ourselves ridiculous.”
So researchers should be posting whatever they like (and unfunded? or funded on the assumption that everyone is “a good young scientist”).
It is not clear what EWD’s is complaining about, or trying to fix, or how…
arnad, S. (1998/2000/2004) The invisible hand of peer review. Nature [online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B. (2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242. http://cogprints.org/1646/