Number of papers is a poor measure of excellence. Some researchers churn out lots of worthless papers. Some publish a few gems. Some are more skilled than others at the politics of getting papers published. Some are more skilled than others at getting their name added to papers.
But do you think people in the United Kingdom or Canada are so incredibly different from Americans? The main difference that I see is that research is funded differently.
(As a proud Canadian, I claim that we match the Americans quality-wise. But other than my Canadian flag, I have no other evidence.)
If anything, I suspect that the bias is on the side of the Americans. They control the best journals and conferences. Due to inertia, they still attract the best students and professors. And I bet that the Americans scientists are overall richer.
“What this means in concrete terms is that few professors get most of the resources whereas the bulk of University professors are left with a pittance or nothing.”
I think, Chinese academic evaluation system nowadays does the same as in the USA. Actually, the “publish or perish”-situation is even more dramatic in China. “Counting the number of publications becomes the norm.” Here is a Nature news article on this issue: http://www.nature.com/news/2010/100112/full/463142a.html
@Jon The USA has stood still while UK, Canada, France, Germany and so on, have progressed in the number of Computer Science papers per year.
Why is that?
I submit to you, for discussion, that maybe now is not a good time to emulate the Americans.
As for my stunning ability to draw unscientific conclusions, well… are you suggesting that I cannot publish my conjectures, labelling them as such, on my blog?
@Daniel: I believe what you and Suresh are saying is that number of papers is a convenient indirect measure of achievement, albeit a flawed one. Or to put it another way, number of papers may not be highly correlated with excellence but it is positively correlated with excellence. I’m not convinced the correlation is positive. It probably is slightly positive, but I wouldn’t rule out negative values.
Countries where more and more people are involved in research will produce more results, and more papers.
Here is something I’d like to measure. How many researchers in the USA are active right now in Computer Science. How many were active in 1998? (By active, I mean “doing research at least 30% of their time”.)
We can argue until we are blue in the face about the metrics. I totally agree that I do not have the right measures.
But some people, it seems, take as an axiom that the USA dominates and will dominate. Then they seek the metrics that fit with this hypothesis and reject other metrics. Would people have reacted so harshly if I had written that American researchers publish more? I think not.
Let’s be open minded about this. Could it be that in 2010, the science models that the USA has pursued are no longer so competitive? I’m just asking the question.
Sureshsays:
John,
all of what you’re saying is indeed true, but you’d have to argue for a systemic DIFFERENCE in how these factors play out in different countries in order to make your case.
The clear story here is the rise of China as a paper writing powerhouse. One can of course argue over impact, but since it’s hard to argue about impact on a short term frame, we’d have to wait a long time to measure that well.
@Daniel: I wasn’t writing to defend American scientific productivity; I have no opinion on that one. I was going off on a tangent, taking a shot at the practice of assessing researchers by the number of pages they publish. (I remember a professor sheepishly admitting that the tenure committee literally counts the number of pages.)
Nobody counts the number of blog posts I write (I hope!). It would be silly to measure the quality of my blogging by the number of posts. However, someone who hasn’t written a single blog post is clearly not much of a blogger.
It would be very dangerous to start measuring blogger by how many posts they write. We know this. The same is true in research, I think.
I wonder how readership compares for a blog versus an academic paper. I have no idea how many people have read the paper articles I’ve published, but I do know many people have downloaded my technical reports and it’s a couple orders of magnitude less than my blog on average.
@John If we did away with the barriers to entry, that is, if anyone could publish any research paper whenever he wants (think: arxiv.org), then research papers would be more like blog posts. People would write to be read because there would no longer be any credit in merely publishing.
Jonsays:
Canada spreads its money wider than the US, I think. This is very likely a terrible statistic you are quoting, though. A large fraction of Chinese papers are plagiarized or otherwise derivative or intellectually unsound. Of course there is also good research coming out of China. But this statistic tells you nothing about that.
Even if the statistic is meaningful, you are drawing the wrong conclusion. You write in bold, “the USA has stood still.” That is entirely because of resources and has nothing to do with how resources are distributed. Or are you seriously arguing that with a better distribution of funds, per-researcher paper output in the US would grow exponentially in time?
I am stunned at your ability to draw unscientific conclusions.
I don’t know, probably because the number of researchers hasn’t changed much. What is the connection to the funding model?
The number of Computer Science papers published in Canada, France, UK, and so on has doubled, tripled or quadrupled, but not in the USA. Why?
Why would the number of researchers presumably grow in other countries, but not in the USA?
You don’t even speculate on what the connection is
Yes I do: ” By leaving most University professors behind, you are wasting precious resources. ”
I even have a way to test my hypothesis: determine the number of active Computer Science researchers in various countries. I predict that there are more now in most countries, whereas the number is more or less constant in the USA.
Briansays:
It’s pointless to discuss quantity without discussing quality. Look at 2006 (a year when apparently China did publish more CS papers than USA) and the avg number of non-self citations per paper:
USA (3.02);
Canada (2.53);
UK (2.36);
Germany (2.08);
France (1.92);
S. Korea (0.99);
Japan (0.94);
China: (0.41)
Maybe USA and Canada are doing something right after all.
Jonsays:
“@Jon The USA has stood still while UK, Canada, France, Germany and so on, have progressed in the number of Computer Science papers per year.
Why is that?”
I don’t know, probably because the number of researchers hasn’t changed much. What is the connection to the funding model? Do you really think that by changing the funding model, we would get twice as many papers per researcher? That doesn’t seem likely.
You don’t even speculate on what the connection is, let alone try to find statistics to prove it. My own conjecture is that this is all due to George Bush. Or maybe crop circles.
“As for my stunning ability to draw unscientific conclusions, well… are you suggesting that I cannot publish my conjectures, labelling them as such, on my blog?”
It doesn’t look good. For a scientist, a conjecture should be more than a random guess.
Sureshsays:
I was drilling down into the CS stats and not surprisingly, saw that the biggest jump from “the asiatic region” is in information systems. this is not surprising to me because I lurk at the main database conferences and can see the vastly increased number of researchers from china, singapore, and taiwan.
Actually, European countries are still emulating the American model. For example, Germany recently announced millions of euros in extra funding for ten so-called “elite” universities. The goal is to improve Germany’s international standing.
In Flanders, the big universities also get more funding …
Jonsays:
“Why would the number of researchers presumably grow in other countries, but not in the USA?”
Maybe because they were catching up after the internet bubble? I don’t have any evidence either way. You also haven’t claimed that their funding models changed, though–you have put up a conjecture about statics (funding models) to explain a dynamic change (research paper growth.
you have put up a conjecture about statics (funding models) to explain a dynamic change (research paper growth).
Research changed between 1998 and 2008. For one thing, we have almost finished a move from paper libraries to electronic libraries. The latter are much more widely accessible.
Living outside of academia, I’m curious as to the connection between funding and publication in CS. What happens to the under-funded professors? If they lose their jobs, this seems more like an issue of professor quantity or churn. But if they don’t, do you need funding to do research in CS? Maybe in a few subfields, and having some postdocs can always help, but it seems to me that a CS paper just requires time and (usually) a computer.
It’s also important to consider the other options available to a top quality student. 1998 puts you right before the inflection point where dot-com mania really took off. My understanding is that that time period drew lots of people into industry who otherwise would have gone into academia. Assuming things normalized after that, we may be seeing that batch of researchers entering their publishing primes with a disproportionate % of top CS people not within academia. Even more generally, the US arguably has more attractive industry prospects for a top student, which would reduce publication quality.
You are entirely entitled to your opinion that the American model of funding research is flawed, and I am inclined to agree with you, but
I feel the tiny sampling of numbers presented are very poorly presented. A careful statistician would not say things like this.
Note especially the last paragraph:
I conjecture that an all-or-nothing approach, so common in the USA, may not be so efficient after all.
Which is a perfectly fine thing to conjecture, and you next give a reason for it:
By leaving most University professors behind, you are wasting precious resources.
Which is very reasonable, also. Then you follow up with what Canada stands to lose:
And I fear that by emulating this model, Canada might be losing out too.
And that is also a reasonable statement. But the issue is most obvious in that while you say “While I do not have enough evidence to conclude”, you actually did make a conclusion. This kind of bias seemed pretty obvious to me. I think this is what Jon (comment #14) was referring to – a scientific hyposthesis should be testable, and the scientist should be prepared to accept the possibility that he is wrong. If, instead, you believe something, and interpret figures to support your belief, then it’s not science – that’s just confirmation bias.
If you’re interested in # of papers published and the way money is distributed, why not at least present some figures on the amount of money given out to researchers, and see what kind of correlation they have?
Number of papers is a poor measure of excellence. Some researchers churn out lots of worthless papers. Some publish a few gems. Some are more skilled than others at the politics of getting papers published. Some are more skilled than others at getting their name added to papers.
@John Good point.
But do you think people in the United Kingdom or Canada are so incredibly different from Americans? The main difference that I see is that research is funded differently.
(As a proud Canadian, I claim that we match the Americans quality-wise. But other than my Canadian flag, I have no other evidence.)
If anything, I suspect that the bias is on the side of the Americans. They control the best journals and conferences. Due to inertia, they still attract the best students and professors. And I bet that the Americans scientists are overall richer.
“What this means in concrete terms is that few professors get most of the resources whereas the bulk of University professors are left with a pittance or nothing.”
I think, Chinese academic evaluation system nowadays does the same as in the USA. Actually, the “publish or perish”-situation is even more dramatic in China. “Counting the number of publications becomes the norm.” Here is a Nature news article on this issue: http://www.nature.com/news/2010/100112/full/463142a.html
@Jon The USA has stood still while UK, Canada, France, Germany and so on, have progressed in the number of Computer Science papers per year.
Why is that?
I submit to you, for discussion, that maybe now is not a good time to emulate the Americans.
As for my stunning ability to draw unscientific conclusions, well… are you suggesting that I cannot publish my conjectures, labelling them as such, on my blog?
@Daniel: I believe what you and Suresh are saying is that number of papers is a convenient indirect measure of achievement, albeit a flawed one. Or to put it another way, number of papers may not be highly correlated with excellence but it is positively correlated with excellence. I’m not convinced the correlation is positive. It probably is slightly positive, but I wouldn’t rule out negative values.
@John
Countries where more and more people are involved in research will produce more results, and more papers.
Here is something I’d like to measure. How many researchers in the USA are active right now in Computer Science. How many were active in 1998? (By active, I mean “doing research at least 30% of their time”.)
We can argue until we are blue in the face about the metrics. I totally agree that I do not have the right measures.
But some people, it seems, take as an axiom that the USA dominates and will dominate. Then they seek the metrics that fit with this hypothesis and reject other metrics. Would people have reacted so harshly if I had written that American researchers publish more? I think not.
Let’s be open minded about this. Could it be that in 2010, the science models that the USA has pursued are no longer so competitive? I’m just asking the question.
John,
all of what you’re saying is indeed true, but you’d have to argue for a systemic DIFFERENCE in how these factors play out in different countries in order to make your case.
The clear story here is the rise of China as a paper writing powerhouse. One can of course argue over impact, but since it’s hard to argue about impact on a short term frame, we’d have to wait a long time to measure that well.
@Daniel: I wasn’t writing to defend American scientific productivity; I have no opinion on that one. I was going off on a tangent, taking a shot at the practice of assessing researchers by the number of pages they publish. (I remember a professor sheepishly admitting that the tenure committee literally counts the number of pages.)
@John
Interestingly Weiwei’s link (http://www.nature.com/news/2010/100112/full/463142a.html) identifies the strong pressure to publish in China as an incentive for fraud.
Nobody counts the number of blog posts I write (I hope!). It would be silly to measure the quality of my blogging by the number of posts. However, someone who hasn’t written a single blog post is clearly not much of a blogger.
It would be very dangerous to start measuring blogger by how many posts they write. We know this. The same is true in research, I think.
I wonder how readership compares for a blog versus an academic paper. I have no idea how many people have read the paper articles I’ve published, but I do know many people have downloaded my technical reports and it’s a couple orders of magnitude less than my blog on average.
@John If we did away with the barriers to entry, that is, if anyone could publish any research paper whenever he wants (think: arxiv.org), then research papers would be more like blog posts. People would write to be read because there would no longer be any credit in merely publishing.
Canada spreads its money wider than the US, I think. This is very likely a terrible statistic you are quoting, though. A large fraction of Chinese papers are plagiarized or otherwise derivative or intellectually unsound. Of course there is also good research coming out of China. But this statistic tells you nothing about that.
Even if the statistic is meaningful, you are drawing the wrong conclusion. You write in bold, “the USA has stood still.” That is entirely because of resources and has nothing to do with how resources are distributed. Or are you seriously arguing that with a better distribution of funds, per-researcher paper output in the US would grow exponentially in time?
I am stunned at your ability to draw unscientific conclusions.
@Jon
I don’t know, probably because the number of researchers hasn’t changed much. What is the connection to the funding model?
The number of Computer Science papers published in Canada, France, UK, and so on has doubled, tripled or quadrupled, but not in the USA. Why?
Why would the number of researchers presumably grow in other countries, but not in the USA?
You don’t even speculate on what the connection is
Yes I do: ” By leaving most University professors behind, you are wasting precious resources. ”
I even have a way to test my hypothesis: determine the number of active Computer Science researchers in various countries. I predict that there are more now in most countries, whereas the number is more or less constant in the USA.
It’s pointless to discuss quantity without discussing quality. Look at 2006 (a year when apparently China did publish more CS papers than USA) and the avg number of non-self citations per paper:
USA (3.02);
Canada (2.53);
UK (2.36);
Germany (2.08);
France (1.92);
S. Korea (0.99);
Japan (0.94);
China: (0.41)
Maybe USA and Canada are doing something right after all.
“@Jon The USA has stood still while UK, Canada, France, Germany and so on, have progressed in the number of Computer Science papers per year.
Why is that?”
I don’t know, probably because the number of researchers hasn’t changed much. What is the connection to the funding model? Do you really think that by changing the funding model, we would get twice as many papers per researcher? That doesn’t seem likely.
You don’t even speculate on what the connection is, let alone try to find statistics to prove it. My own conjecture is that this is all due to George Bush. Or maybe crop circles.
“As for my stunning ability to draw unscientific conclusions, well… are you suggesting that I cannot publish my conjectures, labelling them as such, on my blog?”
It doesn’t look good. For a scientist, a conjecture should be more than a random guess.
I was drilling down into the CS stats and not surprisingly, saw that the biggest jump from “the asiatic region” is in information systems. this is not surprising to me because I lurk at the main database conferences and can see the vastly increased number of researchers from china, singapore, and taiwan.
Actually, European countries are still emulating the American model. For example, Germany recently announced millions of euros in extra funding for ten so-called “elite” universities. The goal is to improve Germany’s international standing.
In Flanders, the big universities also get more funding …
“Why would the number of researchers presumably grow in other countries, but not in the USA?”
Maybe because they were catching up after the internet bubble? I don’t have any evidence either way. You also haven’t claimed that their funding models changed, though–you have put up a conjecture about statics (funding models) to explain a dynamic change (research paper growth.
@Jon
you have put up a conjecture about statics (funding models) to explain a dynamic change (research paper growth).
Research changed between 1998 and 2008. For one thing, we have almost finished a move from paper libraries to electronic libraries. The latter are much more widely accessible.
@Paul
But if they don’t, do you need funding to do research in CS?
You can certainly do excellent research with just your salary. It has certainly become much easier in the last ten years.
(BTW I suspect that this explains how poorer countries are now publishing a lot more. Money is less of a factor than it ever was.)
Even more generally, the US arguably has more attractive industry prospects (…)
Absolutely. I like this explanation. The USA could be “losing” its best researchers to industry, where they publish much less.
Living outside of academia, I’m curious as to the connection between funding and publication in CS. What happens to the under-funded professors? If they lose their jobs, this seems more like an issue of professor quantity or churn. But if they don’t, do you need funding to do research in CS? Maybe in a few subfields, and having some postdocs can always help, but it seems to me that a CS paper just requires time and (usually) a computer.
It’s also important to consider the other options available to a top quality student. 1998 puts you right before the inflection point where dot-com mania really took off. My understanding is that that time period drew lots of people into industry who otherwise would have gone into academia. Assuming things normalized after that, we may be seeing that batch of researchers entering their publishing primes with a disproportionate % of top CS people not within academia. Even more generally, the US arguably has more attractive industry prospects for a top student, which would reduce publication quality.
Interesting stuff to speculate on…
@Shani
Thanks. Note that I am specifically addressing Computer Science.
According to this article, China will publish more papers than the US by 2020
http://www.telegraph.co.uk/news/worldnews/asia/china/7075698/China-to-lead-world-scientific-research-by-2020.html
Bwahahaha!
I just stumbled upon this, academia is kind of a jungle, survival of the greediest.
You are entirely entitled to your opinion that the American model of funding research is flawed, and I am inclined to agree with you, but
I feel the tiny sampling of numbers presented are very poorly presented. A careful statistician would not say things like this.
Note especially the last paragraph:
I conjecture that an all-or-nothing approach, so common in the USA, may not be so efficient after all.
Which is a perfectly fine thing to conjecture, and you next give a reason for it:
By leaving most University professors behind, you are wasting precious resources.
Which is very reasonable, also. Then you follow up with what Canada stands to lose:
And I fear that by emulating this model, Canada might be losing out too.
And that is also a reasonable statement. But the issue is most obvious in that while you say “While I do not have enough evidence to conclude”, you actually did make a conclusion. This kind of bias seemed pretty obvious to me. I think this is what Jon (comment #14) was referring to – a scientific hyposthesis should be testable, and the scientist should be prepared to accept the possibility that he is wrong. If, instead, you believe something, and interpret figures to support your belief, then it’s not science – that’s just confirmation bias.
If you’re interested in # of papers published and the way money is distributed, why not at least present some figures on the amount of money given out to researchers, and see what kind of correlation they have?