If you believe that bias in favor of Stanford is justified, that means that absence of bias agains women is equivalent to effective bias against men.
If explanation is needed: men have slightly higher average IQ, and significantly higher standard deviation of IQ (just look at the special education classes, or whatever euphemism they use for very low IQ kids these days – many more boys than girls).
Then, you would expect that among the most intelligent, there are more men than women, with ratio getting more lopsided the further you move towards the extreme values. Thus, among researchers, you would assume that male researcher is significantly smarter than the female one, just like a Stanford professor is on average smarter than one from Pudunk University.
JFsays:
We could also point to this research:
A 2014 meta-analysis of sex differences in scholastic achievement published in the journal of Psychological Bulletin found females outperformed males in teacher-assigned school marks throughout elementary, junior/middle, high school and at both undergraduate and graduate university level.[117] The meta-analysis, done by researchers Daniel Voyer and Susan D. Voyer from the University of New Brunswick, drew from 97 years of 502 effect sizes and 369 samples stemming from the year 1914 to 2011. Another 2015 study by researchers Gijsbert Stoet and David C. Geary in Intelligence found that girl’s overall education achievement is better in 70 percent of all the 47–75 countries that participated in PISA.[118] The study consisting of 1.5 million 15-year-olds found higher overall female achievement across reading, mathematics, and science literacy and better performance across 70% of participating countries including many with negative gaps in socioeconomic and gender equality, and they fell behind in only 4% of countries.[118] Stoet and Geary concluded that sex differences in educational achievement are not reliably linked to gender equality.[118]
Which found that females outperformed males at academic work, and as such finding no bias is a bias against women. The IQ curves thing is anything but solid, and even if it was, it’s anything but a good indicator of the quality of a research paper.
Women have been systemically discriminated against for the major part of human History, it’s a question worth exploring, and those billions of human beings deserve a lot more than the baseless rebuttal you gave.
Jakubsays:
I think that you misunderstood Jamie’s point. He is not saying that a bias toward women is justified, but rather pointing out that bias in favor toward prestigious institutions should not be justified.
Anonsays:
Tell me something: if women really are the equal of men, then why did they allow men to take over?
Mesays:
They are obviously not physically equal. Men on average are taller, for example.
You may not be aware of this, but Patriarchat dates back to a time when men would try to smash their heads with clubs and swords.
It may be indicative of intelligence to not participate in head smashing contests…
Dominic Amannsays:
You are assuming higher IQ translates into better research papers. I have no idea why you think that would be true.
It is widely suggested that women are better communicators (I don’t know if this is true), but that type of intelligence might also create better (or more readable) research papers.
Jakubsays:
Do you really think that a publication bias toward prestigious institutions is justifiable? Are the researchers from such institutions really “better” or are they simply made to be by the system (self-fulfilling prophecy)? If the reviewers are more lenient for them than for researchers from less prestigious universities then obviously they can publish more. Not only that, but they can also make bolder statements since they receive less scrutiny than their peers. This bias means that they have higher chances of publishing at more prestigious journals and also that readers will also choose their work to read and cite as more trustworthy. If the reviewers have such attitude, it is hard to imagine that funding agencies will also not give them an easier pass, which means more funding for them. Of course on the way it just boosts their self-confidence to challenge the limits of the system. A snowball effect with more funding leading to more papers and then more citations just based on one’s affiliation will result in creating a more recognized researcher. However, none of this is really his/her own merit, but an outcome of a bias towards them.
One could even argue that we should trust less researchers from prestigious universities since they tend to be caught more often on unethical practices. Of course this is due to some of the same reasons why the systems makes them to succeed – they publish in venues with more readers and their work grabs more attention, so they have higher chances of getting caught and have media reporting it. My point is that since we know nothing of the particular piece of work or researcher, any bias towards or against them diminishes the value of science. There are mediocre, average and good researchers in prestigious and less prestigious institutions. Considering the average quality of researchers in an institution when reviewing a paper is just unfair and meaningless for the proposed arguments validity.
You may or may not know this, but please consider that I am from a definitively non-prestigious university. You probably never heard of it.
Do you really think that a publication bias toward prestigious institutions is justifiable? Are the researchers from such institutions really “better†or are they simply made to be by the system (self-fulfilling prophecy)? If the reviewers are more lenient for them than for researchers from less prestigious universities then obviously they can publish more. Not only that, but they can also make bolder statements since they receive less scrutiny than their peers. This bias means that they have higher chances of publishing at more prestigious journals and also that readers will also choose their work to read and cite as more trustworthy. If the reviewers have such attitude, it is hard to imagine that funding agencies will also not give them an easier pass, which means more funding for them. Of course on the way it just boosts their self-confidence to challenge the limits of the system. A snowball effect with more funding leading to more papers and then more citations just based on one’s affiliation will result in creating a more recognized researcher. However, none of this is really his/her own merit, but an outcome of a bias towards them.
It is really hard to doubt that researchers at Stanford are better than researchers at any random school. It is simply the case that Stanford is a more desirable employer, so they can more easily recruit the very best. Of course, it is a virtuous circle.
Is this good or bad? What I think is maybe not as important as the social norm. It is simply a fact that we accept biases in favor of prestigious institutions. It is not only true in academia, by the way. It is perfectly fine to quote a researcher and to stress that he is from “Harvard”. It is fine to recruit a programmer and say that “he worked at Google”. It is not fine to quote a scientist and to add “and he is a man”. Research papers start with the affiliation on the front page. There is typically no explicit mention of the gender. We care a lot about the affiliation, and we stress them when they are prestigious.
There are mediocre, average and good researchers in prestigious and less prestigious institutions.
That is definitively true, but not in the same ratio.
Considering the average quality of researchers in an institution when reviewing a paper is just unfair and meaningless for the proposed arguments validity.
When you are reviewing a research paper, you are working with incomplete information. You do not have the software, the data, you do not know what the lab is like, you do not know how skillful the researchers are. Sometimes it comes down to beliefs. Do you believe that they did the work properly?
Note that affiliation is just one variable being used.
Suppose that I submit a paper where I claim that I cured cancer in 20 patients using some chemical. I have never done cancer research, and I am not at an institution with a medical school. How believable is this?
Suppose that someone from Harvard’s medical school makes the same claim. How believable is it?
It is much more believable in the second case.
But suppose that, in the first case, the author is from an unknown affiliation and he has never worked on cancer… but the previous year, he cured type 1 diabetes. The story changes all of a sudden.
So affiliation is one of many signals, maybe not the most important one.
It is simply the case that we need to read signals to assess people’s work. There is no other way.
Jakubsays:
Thank you for a thorough response. I completely agree with you when you say that affiliation plays a role in a review or recruitment processes. I will say even more. Even though I do not find bias toward prestigious universities justifiable, it is still possible that an author’s affiliation unconsciously affects my decisions during the review process. What surprised me is that you find it consciously justifiable.
I can also agree with you that on average researchers at Stanford are better than researchers at any random school. However, I still believe that it should not play a role during a review process.
There are way less available places at prestigious institutions than candidates. Therefore, it is inevitable that several equally qualified applicants (as the one offered a position) will get rejected purely due to the lack of luck and will end up in mid-tier universities. The quality of their work is not going to be any worse than the work done by the accepted applicant, yet over the course of their careers, due to the bias in favor of prestigious universities, the gap between their achievements and recognition is going to be disproportional in favor of the accepted applicant.
This bias makes it really hard to get rid of poor researchers who worked or graduated from a prestigious institution, i.e. once you have that fancy affiliation on your CV you will get a job or at least an interview. There are cases of people in my field who got their PhDs from a famous university, but constantly make experimental and statistical mistakes in their publications that even a honors student should be able to spot. One of them even was involved in some ethically questionable practices and got fired from a job, yet found no problem getting a new position elsewhere. Of course these are single cases, but I am afraid that it is an inevitable outcome of such bias.
I am afraid that bias in favor of some universities creates also a bias against other. It was an eye opening experience when I have done a short project in a country and university not recognized for its research qualities. Normally, my affiliations are not from prestigious, but respectable institutions, but for that article I have used the affiliation of that university. I have never received before or after such a patronizing review from an editor who implied inadequacy of the statistical analysis without even pointing at any single problem with it. To make it worse, the analysis was one of the paper strengths compared with the related papers. If this was not a pure coincidence, I cannot see how folks having less than glorious affiliations can learn anything from the peer review process. Moreover, not all great researchers will want to work at Stanford. For various personal or other reasons they may want to live in a country that does not have prestigious universities. I find it highly problematic if for their entire career, their affiliation will be used against them in case of any doubts, due to their colleagues being worse researchers.
My perception of this topic could be field specific. Speed performance of software is an objective measure so maybe the affiliation bias does not play such a big role. On the other hand, my field that involves research with human subjects is much more subjective. I am just tired of people who are famous for being famous (experts for being perceived as experts despite the lack of actual knowledge/publication record), which is more common if you have a right affiliation.
BTW, industry (or at least some major IT companies) seems to be finally moving away from hiring based on the university affiliation and focusing purely on skills. I believe that it would be healthy for academia to go in the same direction.
Jakubsays:
I forgot to answer the questions that you posed regarding a cancer cure discovery. What would be wrong if we applied the same skepticism and put work of Harvard medical school researchers under the same scrutiny as if it was done by an author from an unknown institution? If the particular authors really have the knowledge and have done their work properly, they should be able to address reviewers’ concerns. If they cannot, then I wouldn’t trust their study any more than if their affiliation was from a random other institution. It would be still much easier to run a replication of their work, requested by a skeptic reviewer, for Harvard researchers than a random university researchers since they have more resources and manpower. Considering how many other perks they gain based on their affiliation, I do not see any benefit of giving them a leeway in the review/recruitment process.
Mesays:
We don’t actually know there is no bias in peer review!
The research just indicates that the peer review bias is not worse than the selection bias before. We have much fewer female PhD students, much fewer female professors, etc.
So the conclusions to draw from this are:
We likely cannot improve gender bias in peer review much now
We first need to fix bias in education, then likely will need to revisit bias in peer review, as it may become more observable afterwards.
There are many reasons for this. IIRC you blogged before there sometimes is even a positive bias for women. In some cases, a bias in one part can mask an opposite bias in another step. There is always a selection bias, etc.
For example, the female authors may be much tougher selected. If we select the top 1% smartest women, but the top 10% smartest men, this should make the average female author smarter on average, and that could lead to higher quality submissions. Similarly, we may be putting tougher pre-submission thresholds, which would also lead to female submissions having a higher quality. But if they only fare “as good” as male authors, that can mean that either there is a bias against women (and thus, it cancels out the higher quality), or that just out peer review is so badly broken that quality does not matter that much. And there is strong evidence for this, if you consider the recent NIPS experiment. If acceptance is largely random, you’d expect it to be unbiased…
If you believe that bias in favor of Stanford is justified, that means that absence of bias agains women is equivalent to effective bias against men.
If explanation is needed: men have slightly higher average IQ, and significantly higher standard deviation of IQ (just look at the special education classes, or whatever euphemism they use for very low IQ kids these days – many more boys than girls).
Then, you would expect that among the most intelligent, there are more men than women, with ratio getting more lopsided the further you move towards the extreme values. Thus, among researchers, you would assume that male researcher is significantly smarter than the female one, just like a Stanford professor is on average smarter than one from Pudunk University.
We could also point to this research:
A 2014 meta-analysis of sex differences in scholastic achievement published in the journal of Psychological Bulletin found females outperformed males in teacher-assigned school marks throughout elementary, junior/middle, high school and at both undergraduate and graduate university level.[117] The meta-analysis, done by researchers Daniel Voyer and Susan D. Voyer from the University of New Brunswick, drew from 97 years of 502 effect sizes and 369 samples stemming from the year 1914 to 2011. Another 2015 study by researchers Gijsbert Stoet and David C. Geary in Intelligence found that girl’s overall education achievement is better in 70 percent of all the 47–75 countries that participated in PISA.[118] The study consisting of 1.5 million 15-year-olds found higher overall female achievement across reading, mathematics, and science literacy and better performance across 70% of participating countries including many with negative gaps in socioeconomic and gender equality, and they fell behind in only 4% of countries.[118] Stoet and Geary concluded that sex differences in educational achievement are not reliably linked to gender equality.[118]
https://en.wikipedia.org/wiki/Sex_differences_in_intelligence#Sex_differences_in_academics
Which found that females outperformed males at academic work, and as such finding no bias is a bias against women. The IQ curves thing is anything but solid, and even if it was, it’s anything but a good indicator of the quality of a research paper.
Women have been systemically discriminated against for the major part of human History, it’s a question worth exploring, and those billions of human beings deserve a lot more than the baseless rebuttal you gave.
I think that you misunderstood Jamie’s point. He is not saying that a bias toward women is justified, but rather pointing out that bias in favor toward prestigious institutions should not be justified.
Tell me something: if women really are the equal of men, then why did they allow men to take over?
They are obviously not physically equal. Men on average are taller, for example.
You may not be aware of this, but Patriarchat dates back to a time when men would try to smash their heads with clubs and swords.
It may be indicative of intelligence to not participate in head smashing contests…
You are assuming higher IQ translates into better research papers. I have no idea why you think that would be true.
It is widely suggested that women are better communicators (I don’t know if this is true), but that type of intelligence might also create better (or more readable) research papers.
Do you really think that a publication bias toward prestigious institutions is justifiable? Are the researchers from such institutions really “better” or are they simply made to be by the system (self-fulfilling prophecy)? If the reviewers are more lenient for them than for researchers from less prestigious universities then obviously they can publish more. Not only that, but they can also make bolder statements since they receive less scrutiny than their peers. This bias means that they have higher chances of publishing at more prestigious journals and also that readers will also choose their work to read and cite as more trustworthy. If the reviewers have such attitude, it is hard to imagine that funding agencies will also not give them an easier pass, which means more funding for them. Of course on the way it just boosts their self-confidence to challenge the limits of the system. A snowball effect with more funding leading to more papers and then more citations just based on one’s affiliation will result in creating a more recognized researcher. However, none of this is really his/her own merit, but an outcome of a bias towards them.
One could even argue that we should trust less researchers from prestigious universities since they tend to be caught more often on unethical practices. Of course this is due to some of the same reasons why the systems makes them to succeed – they publish in venues with more readers and their work grabs more attention, so they have higher chances of getting caught and have media reporting it. My point is that since we know nothing of the particular piece of work or researcher, any bias towards or against them diminishes the value of science. There are mediocre, average and good researchers in prestigious and less prestigious institutions. Considering the average quality of researchers in an institution when reviewing a paper is just unfair and meaningless for the proposed arguments validity.
You may or may not know this, but please consider that I am from a definitively non-prestigious university. You probably never heard of it.
Do you really think that a publication bias toward prestigious institutions is justifiable? Are the researchers from such institutions really “better†or are they simply made to be by the system (self-fulfilling prophecy)? If the reviewers are more lenient for them than for researchers from less prestigious universities then obviously they can publish more. Not only that, but they can also make bolder statements since they receive less scrutiny than their peers. This bias means that they have higher chances of publishing at more prestigious journals and also that readers will also choose their work to read and cite as more trustworthy. If the reviewers have such attitude, it is hard to imagine that funding agencies will also not give them an easier pass, which means more funding for them. Of course on the way it just boosts their self-confidence to challenge the limits of the system. A snowball effect with more funding leading to more papers and then more citations just based on one’s affiliation will result in creating a more recognized researcher. However, none of this is really his/her own merit, but an outcome of a bias towards them.
It is really hard to doubt that researchers at Stanford are better than researchers at any random school. It is simply the case that Stanford is a more desirable employer, so they can more easily recruit the very best. Of course, it is a virtuous circle.
Is this good or bad? What I think is maybe not as important as the social norm. It is simply a fact that we accept biases in favor of prestigious institutions. It is not only true in academia, by the way. It is perfectly fine to quote a researcher and to stress that he is from “Harvard”. It is fine to recruit a programmer and say that “he worked at Google”. It is not fine to quote a scientist and to add “and he is a man”. Research papers start with the affiliation on the front page. There is typically no explicit mention of the gender. We care a lot about the affiliation, and we stress them when they are prestigious.
There are mediocre, average and good researchers in prestigious and less prestigious institutions.
That is definitively true, but not in the same ratio.
Considering the average quality of researchers in an institution when reviewing a paper is just unfair and meaningless for the proposed arguments validity.
When you are reviewing a research paper, you are working with incomplete information. You do not have the software, the data, you do not know what the lab is like, you do not know how skillful the researchers are. Sometimes it comes down to beliefs. Do you believe that they did the work properly?
Note that affiliation is just one variable being used.
Suppose that I submit a paper where I claim that I cured cancer in 20 patients using some chemical. I have never done cancer research, and I am not at an institution with a medical school. How believable is this?
Suppose that someone from Harvard’s medical school makes the same claim. How believable is it?
It is much more believable in the second case.
But suppose that, in the first case, the author is from an unknown affiliation and he has never worked on cancer… but the previous year, he cured type 1 diabetes. The story changes all of a sudden.
So affiliation is one of many signals, maybe not the most important one.
It is simply the case that we need to read signals to assess people’s work. There is no other way.
Thank you for a thorough response. I completely agree with you when you say that affiliation plays a role in a review or recruitment processes. I will say even more. Even though I do not find bias toward prestigious universities justifiable, it is still possible that an author’s affiliation unconsciously affects my decisions during the review process. What surprised me is that you find it consciously justifiable.
I can also agree with you that on average researchers at Stanford are better than researchers at any random school. However, I still believe that it should not play a role during a review process.
There are way less available places at prestigious institutions than candidates. Therefore, it is inevitable that several equally qualified applicants (as the one offered a position) will get rejected purely due to the lack of luck and will end up in mid-tier universities. The quality of their work is not going to be any worse than the work done by the accepted applicant, yet over the course of their careers, due to the bias in favor of prestigious universities, the gap between their achievements and recognition is going to be disproportional in favor of the accepted applicant.
This bias makes it really hard to get rid of poor researchers who worked or graduated from a prestigious institution, i.e. once you have that fancy affiliation on your CV you will get a job or at least an interview. There are cases of people in my field who got their PhDs from a famous university, but constantly make experimental and statistical mistakes in their publications that even a honors student should be able to spot. One of them even was involved in some ethically questionable practices and got fired from a job, yet found no problem getting a new position elsewhere. Of course these are single cases, but I am afraid that it is an inevitable outcome of such bias.
I am afraid that bias in favor of some universities creates also a bias against other. It was an eye opening experience when I have done a short project in a country and university not recognized for its research qualities. Normally, my affiliations are not from prestigious, but respectable institutions, but for that article I have used the affiliation of that university. I have never received before or after such a patronizing review from an editor who implied inadequacy of the statistical analysis without even pointing at any single problem with it. To make it worse, the analysis was one of the paper strengths compared with the related papers. If this was not a pure coincidence, I cannot see how folks having less than glorious affiliations can learn anything from the peer review process. Moreover, not all great researchers will want to work at Stanford. For various personal or other reasons they may want to live in a country that does not have prestigious universities. I find it highly problematic if for their entire career, their affiliation will be used against them in case of any doubts, due to their colleagues being worse researchers.
My perception of this topic could be field specific. Speed performance of software is an objective measure so maybe the affiliation bias does not play such a big role. On the other hand, my field that involves research with human subjects is much more subjective. I am just tired of people who are famous for being famous (experts for being perceived as experts despite the lack of actual knowledge/publication record), which is more common if you have a right affiliation.
BTW, industry (or at least some major IT companies) seems to be finally moving away from hiring based on the university affiliation and focusing purely on skills. I believe that it would be healthy for academia to go in the same direction.
I forgot to answer the questions that you posed regarding a cancer cure discovery. What would be wrong if we applied the same skepticism and put work of Harvard medical school researchers under the same scrutiny as if it was done by an author from an unknown institution? If the particular authors really have the knowledge and have done their work properly, they should be able to address reviewers’ concerns. If they cannot, then I wouldn’t trust their study any more than if their affiliation was from a random other institution. It would be still much easier to run a replication of their work, requested by a skeptic reviewer, for Harvard researchers than a random university researchers since they have more resources and manpower. Considering how many other perks they gain based on their affiliation, I do not see any benefit of giving them a leeway in the review/recruitment process.
We don’t actually know there is no bias in peer review!
The research just indicates that the peer review bias is not worse than the selection bias before. We have much fewer female PhD students, much fewer female professors, etc.
So the conclusions to draw from this are:
We likely cannot improve gender bias in peer review much now
We first need to fix bias in education, then likely will need to revisit bias in peer review, as it may become more observable afterwards.
There are many reasons for this. IIRC you blogged before there sometimes is even a positive bias for women. In some cases, a bias in one part can mask an opposite bias in another step. There is always a selection bias, etc.
For example, the female authors may be much tougher selected. If we select the top 1% smartest women, but the top 10% smartest men, this should make the average female author smarter on average, and that could lead to higher quality submissions. Similarly, we may be putting tougher pre-submission thresholds, which would also lead to female submissions having a higher quality. But if they only fare “as good” as male authors, that can mean that either there is a bias against women (and thus, it cancels out the higher quality), or that just out peer review is so badly broken that quality does not matter that much. And there is strong evidence for this, if you consider the recent NIPS experiment. If acceptance is largely random, you’d expect it to be unbiased…