While the case put forward against double blind review looks good, one case where your argument may fail is submission by a new author OR a publication which may be a part of a new story. I am aware of researchers changing areas and working on different projects, problems and hence, putting it into context isn’t possible. I do concur on the “transparency” aspect, however, I would vouch for half transparency viz. make the comments public and if any discrepancy found let the program committee decide on the fate of the reviewer. This may mitigate various risk scenarios arising out of the current review system.
Sebastian H.says:
“Research papers are often only interesting when put in a greater context.”
That is true but what that’s why papers should cite prior work and how the particular paper relates to it. It shouldn’t matter if it was your own prior work or someone else’s to make it worth publishing.
I reviewed papers for ICDAR recently, and I found its single-blind system to be very helpful. For one, knowing authors tells me how carefully I need to check their derivations. In ideal world I’d rederive every formula in every paper I review, but who has this kind of time?
In another example, I googled authors publications and found essentially identical paper published in earlier year. Author failed to mention this in their submission.
Recently there’s an increasing amount of research that is happening outside of traditional publishing system. What would you do if the submission is almost identical to a work you’ve seen on someone’s research blog/mathoverflow/etc? With double blind you have to guess if it’s the author finally packaging up their work for publication, or an impostor with copy/paste.
We recognise that women remain poorly represented at senior levels across the sciences, including ecology and evolution. Given the lack of evidence that double-blind review favours female authors, we suggest that efforts to address this imbalance should be directed into supporting innovative schemes aiming to change working conditions, including initiatives such as increased flexibility of working hours, support for scientists returning to research after career breaks and mentoring schemes. We see no reason to direct time and resources into overcoming the acknowledged resistance of editors and the scepticism of many in the field regarding the alleged benefits of double-blind review.
Reference:
Webb, T.J. and O’Hara, B. and Freckleton, R.P., Does double-blind review benefit female authors?, Heredity 77, 2008.
I think we agree that a more female-friendly science would be better for everyone.
I’m actually more interested in the actual perception of female/minority researchers in my field.
Fair enough. Let us accept as a fact that women feel that they are treated more fairly with double-blind peer review… as opposed to, say, open peer review.
Still. We have to watch out for unintended consequences.
Do consider however that double-blind peer reviews are reportedly harsher (and this makes intuitive sense to me). What impact do these strongly worded reviews have on women over time? Do men and women cope equally well with “macho” reviewing?
Cyrilsays:
I used to question double-blind review like you, but the best argument in its favour I have heard is that there is apparently a verifiable bias against women and minorities in “transparent” reviewing. I am not aware of references, but this point was made by prominent female researchers in my field (they are few and far between!), which I would tend to trust.
Technically this doesn’t mean we could not use a reverse single-blind system where authors are hidden and reviewers names are published. They could be listed (without their reviews) with a paper without jeopardizing the confidentiality, if there are enough of them, which there should be anyway.
Point is: it’s easy to come up with examples where keeping names hidden causes problems, but the trade-off is real and arguably worse.
Cyrilsays:
Daniel,
Note sure what the point of that excerpt is, especially in light of the fact that the paper right after that one (last page of the pdf you link to), as well as the paper they are replying to, seem to put things very clearly:
“[…] the study is a compelling indication that changes in review policy [to double blind] increase female representation through editor, reviewer or author behaviour.”
As it’s the nature of such contentious issues to be argued either way, I’m actually more interested in the actual perception of female/minority researchers in my field. Yours and others may be different!
This is a point that should at least be addressed as it seems to be more serious than potential recycling/plagiarism.
Kris Jacksays:
I’m interested in your opinions on this one, In practice, how far does double blind review got in retaining author anonymity?
Given that authors tend to cite their own previous works, as Daniel indicates, and that experts in a field tend to know each others work very well, how often are you able to guess who is the author of a paper that you are reviewing?
Cyrilsays:
@Kris Jack:
I’d say that out of all double-blind papers I have reviewed over the last 15 or so years, I’ve suspected who the author was many times, but mostly in a non relevant way (ie from references to past papers and work, but I did not know these people anyway). I have been dead certain only once or twice because I knew the authors, how they write and the field well enough.
Epizdochiesays:
I am emphatically FOR double-blind reviews, and reading the above comments re-inforce my conviction. Reviewers “suspect” who the author is, “google” his/her prior work, fail to check derivations based on the authors’ reputation, and spend their otherwise precious time on inconsequential aspects – focus on the paper given to you for review, and do not concern yourself with the author and his affiliation. Look, if you don’t have time to do a proper review, THEN DON’T DO IT, and say so from the start – there are probably other people in the whole wide world ready to step up and do a proper job (but you wouldn’t want that, would you, you want to be part of the game, don’t you, you want to be able to make or break other people, right?). The absence of a double-blind review perpetuates the rigging of the system and is deeply unfair to the authors – if you don’t want to do that, fairer would be to disclose the identity/affiliation of the reviewers to the author, so that he/she could have a better idea about how competent they really are to judge his/her work. How comfortable would reviewers feel in that situation?
Reviewers “suspect†who the author is, “google†his/her prior work, fail to check derivations based on the authors’ reputation, and spend their otherwise precious time on inconsequential aspects – focus on the paper given to you for review, and do not concern yourself with the author and his affiliation. Look, if you don’t have time to do a proper review, THEN DON’T DO IT
The person taking the decision to accept or reject the paper, the editor, knows your reputation and your affiliation. So, sadly, there is no way to take this out of the equation. In fact, in a double-blind system, the editor has even more power.
To properly review a paper, I need to compare it with the state-of-the-art and previous work. Typically, this means reviewing (even unknowingly) what the author worked on previously. So, you cannot just focus on the paper as a separate entity.
(…) fairer would be to disclose the identity/affiliation of the reviewers to the author, so that he/she could have a better idea about how competent they really are to judge his/her work. How comfortable would reviewers feel in that situation?
I have been advocating precisely this scheme for years. If you are going to criticize someone, you should be able to do it openly, in front of witnesses, not under the cover of anonymity.
But it works both ways… if you are going to ask me to review your work, through a journal or otherwise, please let me know who you are.
Once everything is public, the focus will naturally be on the science. If people unfairly review your work (because you lack a prestigious affiliation) then you’ll be able to point to the evidence, as it will all be out there. It also takes power away from the editors, and it puts it back in the community at large.
Epizdochiesays:
You’re very prompt, Daniel, I commend you for that (I also have an appreciation for the situation at hand, I am anonymous in this conversation, but you’re not, sorry).
I feel that the editor would have no ammo once he receives the reviews from the reviewers, so a double-blind would hurt no one, Also, the rejection of a paper based on a perceived lack of prestigious affiliation would be hard to prove even if the reviewers are known (after all, the author is a nobody to start with, AND, there is no published work to speak of, so who would pay any attention to him/her?) – a double-blind probably seems a better choice there, at least no one can cry “foul”.
But anyway, I’m glad that turning the tables on the reviewers (i.e., making them non-anonymous) seems like a good idea to you, too, so let’s work on that, it looks like a good start,
I feel that the editor would have no ammo once he receives the reviews from the reviewers
The objectivity of the editor is important especially because reviewers are rarely unanimous. Editors effectively act as super-reviewers. An editor can stop a paper from going to peer review. He can pull the paper out of the process after receiving even a hint of criticism, or he can ignore much of the criticism and push the paper forward.
The decision of the editor is typically final and without appeal. The recommendation of a reviewer is always only a recommendation, it can always be ignored by the editor.
So, really, if you feel that it is important that your identity be hidden, it should be hidden from the editor.
@Denzil
As a member of program committees, I often find it useful to look at who the authors are and what they have done. It seems almost essential.
Daniel,
While the case put forward against double blind review looks good, one case where your argument may fail is submission by a new author OR a publication which may be a part of a new story. I am aware of researchers changing areas and working on different projects, problems and hence, putting it into context isn’t possible. I do concur on the “transparency” aspect, however, I would vouch for half transparency viz. make the comments public and if any discrepancy found let the program committee decide on the fate of the reviewer. This may mitigate various risk scenarios arising out of the current review system.
“Research papers are often only interesting when put in a greater context.”
That is true but what that’s why papers should cite prior work and how the particular paper relates to it. It shouldn’t matter if it was your own prior work or someone else’s to make it worth publishing.
@Sebastian
I disagree. See my following blog post:
http://lemire.me/blog/archives/2011/04/29/is-science-more-art-or-industry/
I reviewed papers for ICDAR recently, and I found its single-blind system to be very helpful. For one, knowing authors tells me how carefully I need to check their derivations. In ideal world I’d rederive every formula in every paper I review, but who has this kind of time?
In another example, I googled authors publications and found essentially identical paper published in earlier year. Author failed to mention this in their submission.
Recently there’s an increasing amount of research that is happening outside of traditional publishing system. What would you do if the submission is almost identical to a work you’ve seen on someone’s research blog/mathoverflow/etc? With double blind you have to guess if it’s the author finally packaging up their work for publication, or an impostor with copy/paste.
@Cyril
We recognise that women remain poorly represented at senior levels across the sciences, including ecology and evolution. Given the lack of evidence that double-blind review favours female authors, we suggest that efforts to address this imbalance should be directed into supporting innovative schemes aiming to change working conditions, including initiatives such as increased flexibility of working hours, support for scientists returning to research after career breaks and mentoring schemes. We see no reason to direct time and resources into overcoming the acknowledged resistance of editors and the scepticism of many in the field regarding the alleged benefits of double-blind review.
Reference:
Webb, T.J. and O’Hara, B. and Freckleton, R.P., Does double-blind review benefit female authors?, Heredity 77, 2008.
@Cyril
I think we agree that a more female-friendly science would be better for everyone.
I’m actually more interested in the actual perception of female/minority researchers in my field.
Fair enough. Let us accept as a fact that women feel that they are treated more fairly with double-blind peer review… as opposed to, say, open peer review.
Still. We have to watch out for unintended consequences.
Do consider however that double-blind peer reviews are reportedly harsher (and this makes intuitive sense to me). What impact do these strongly worded reviews have on women over time? Do men and women cope equally well with “macho” reviewing?
I used to question double-blind review like you, but the best argument in its favour I have heard is that there is apparently a verifiable bias against women and minorities in “transparent” reviewing. I am not aware of references, but this point was made by prominent female researchers in my field (they are few and far between!), which I would tend to trust.
Technically this doesn’t mean we could not use a reverse single-blind system where authors are hidden and reviewers names are published. They could be listed (without their reviews) with a paper without jeopardizing the confidentiality, if there are enough of them, which there should be anyway.
Point is: it’s easy to come up with examples where keeping names hidden causes problems, but the trade-off is real and arguably worse.
Daniel,
Note sure what the point of that excerpt is, especially in light of the fact that the paper right after that one (last page of the pdf you link to), as well as the paper they are replying to, seem to put things very clearly:
“[…] the study is a compelling indication that changes in review policy [to double blind] increase female representation through editor, reviewer or author behaviour.”
As it’s the nature of such contentious issues to be argued either way, I’m actually more interested in the actual perception of female/minority researchers in my field. Yours and others may be different!
This is a point that should at least be addressed as it seems to be more serious than potential recycling/plagiarism.
I’m interested in your opinions on this one, In practice, how far does double blind review got in retaining author anonymity?
Given that authors tend to cite their own previous works, as Daniel indicates, and that experts in a field tend to know each others work very well, how often are you able to guess who is the author of a paper that you are reviewing?
@Kris Jack:
I’d say that out of all double-blind papers I have reviewed over the last 15 or so years, I’ve suspected who the author was many times, but mostly in a non relevant way (ie from references to past papers and work, but I did not know these people anyway). I have been dead certain only once or twice because I knew the authors, how they write and the field well enough.
I am emphatically FOR double-blind reviews, and reading the above comments re-inforce my conviction. Reviewers “suspect” who the author is, “google” his/her prior work, fail to check derivations based on the authors’ reputation, and spend their otherwise precious time on inconsequential aspects – focus on the paper given to you for review, and do not concern yourself with the author and his affiliation. Look, if you don’t have time to do a proper review, THEN DON’T DO IT, and say so from the start – there are probably other people in the whole wide world ready to step up and do a proper job (but you wouldn’t want that, would you, you want to be part of the game, don’t you, you want to be able to make or break other people, right?). The absence of a double-blind review perpetuates the rigging of the system and is deeply unfair to the authors – if you don’t want to do that, fairer would be to disclose the identity/affiliation of the reviewers to the author, so that he/she could have a better idea about how competent they really are to judge his/her work. How comfortable would reviewers feel in that situation?
@Epizdochie
The person taking the decision to accept or reject the paper, the editor, knows your reputation and your affiliation. So, sadly, there is no way to take this out of the equation. In fact, in a double-blind system, the editor has even more power.
To properly review a paper, I need to compare it with the state-of-the-art and previous work. Typically, this means reviewing (even unknowingly) what the author worked on previously. So, you cannot just focus on the paper as a separate entity.
I have been advocating precisely this scheme for years. If you are going to criticize someone, you should be able to do it openly, in front of witnesses, not under the cover of anonymity.
But it works both ways… if you are going to ask me to review your work, through a journal or otherwise, please let me know who you are.
Once everything is public, the focus will naturally be on the science. If people unfairly review your work (because you lack a prestigious affiliation) then you’ll be able to point to the evidence, as it will all be out there. It also takes power away from the editors, and it puts it back in the community at large.
You’re very prompt, Daniel, I commend you for that (I also have an appreciation for the situation at hand, I am anonymous in this conversation, but you’re not, sorry).
I feel that the editor would have no ammo once he receives the reviews from the reviewers, so a double-blind would hurt no one, Also, the rejection of a paper based on a perceived lack of prestigious affiliation would be hard to prove even if the reviewers are known (after all, the author is a nobody to start with, AND, there is no published work to speak of, so who would pay any attention to him/her?) – a double-blind probably seems a better choice there, at least no one can cry “foul”.
But anyway, I’m glad that turning the tables on the reviewers (i.e., making them non-anonymous) seems like a good idea to you, too, so let’s work on that, it looks like a good start,
Again, thanks for replying.
@Epizdochie
I feel that the editor would have no ammo once he receives the reviews from the reviewers
The objectivity of the editor is important especially because reviewers are rarely unanimous. Editors effectively act as super-reviewers. An editor can stop a paper from going to peer review. He can pull the paper out of the process after receiving even a hint of criticism, or he can ignore much of the criticism and push the paper forward.
The decision of the editor is typically final and without appeal. The recommendation of a reviewer is always only a recommendation, it can always be ignored by the editor.
So, really, if you feel that it is important that your identity be hidden, it should be hidden from the editor.