I agree that science does have its inconveniences. But I don’t think your idea is that good either.
A problem -and a big one- of blogs and social media is that the quality standards are sub-par. Most of the publishers lack a writing background and an editorial filter. Pieces are ill-written. And -like Zinserr puts it in his book- modern writing is just rubbish. We can accept people blogging about their dogs like that. But can we accept people writing scientific ideas like that.
If you look at some Chinese Journals, authors get accepted for friendship. The papers have terrible errors. Ideas are often mixed and ill posed and the authors have little idea on the concepts they’re using. Not to mention their English terrible.
On the other hand, a good reviewer may be your best friend. I have found much help from kind reviewers. Who propose reading references and point out writing errors. My papers are usually enhanced by their ideas. (One would be incredibly arrogant to think no one could give ideas to improve their papers)
And on the markets,
The QWERTY keyboard exists because the markets decided it should. Yet, it is far from being the ideal keyboard layout. Few people could argue that windows is the best OS, yet it rules the market. The best product don’t rule the markets. Is the most popular products the ones that does.
The other huge problem is that peer-review is a justification for putting academic research behind pay-walls. (Despite the fact that reviewers aren’t paid.)
Good academic research is now one of the least accessible kinds of information in the world, while paid propaganda and dumb speculations are rife.
The publishers with their captive market have no incentive to make it more visible. They know the universities have to keep subscribing to the journals.
The academics are usually decent enough, but as their objective is to please the reviewers and not piss off the publishers, and they have no incentive to gain a mass readership, they don’t systematically make their work visible as a matter of course.
And, of course, the work isn’t popular enough for anyone to make available on bittorrent.
The result is that academic writing and the conversation in the academy is kept artificially distinct from popular / internet discourse.
In an ideal world, good academics, who were identified and credited by the system, would also become influential public intellectuals by virtue of being *linked*.
As it is, their most important work is almost never visible to the web, never linked from wikipedia or discussions. And so has far less influence than it deserves.
Philippe Beaudoinsays:
Still lots of friction to adoption of a publish-then-filter model in academia. The arguments I hear most often
– – – –
1) Publish-then-filter makes it impossible to design a much-needed metric to allocate research funds.
This argument holds some merit, but IMHO it’s backwards. The reward system should be designed AFTER the publication system, not the other way around. If we seriously move to a publish-then-filter model (and if there is a real need for a metric) then I’m convinced we are creative enough to find one. Sure, it’s frightening, sure poor researchers who are able to game the current metric risk doing poorly, but I’m convinced we can design one that would reward researchers who truly work for the greater good of our society — what the reward system should have been about in the first place.
– – – –
2) Blogs are crappy, so any content following a publish-then-filter model will be crappy.
Wrong in so many ways:
– A lot of blogs are not crappy, those I run into given my online social filter tend to be much better (to me, at least) than a random journal paper.
– Journal paper hunting also requires filtering (your colleage, your advisor, a conference).
– If good writers start to publish-then-filter quality will increase.
If you insist on using this argument, please at least point out a structural reason why blogs would be of lower quality. Personally, I can’t think of any.
– – – –
I’d be really interested in hearing any other arguments against this model. I think we need to start advocating it seriously in academia, and a list of counter-arguments would be a good place to start.
I think all systems will have some flaws.
Here are some thoughts that come to mind for me:
(a) The “publish and filter” system already exists. Anyone can publish their work on their web page, in technical reports, workshops, in arXiv, and other fora, and many do.
(b) The branding of well-known journals and conferences can be hugely useful. If you are an unknown author and you make a grand break-through, publishing in a well known reviewed venue potentially gives you a huge leg up, where otherwise your article may languish in obscurity or be adopted by others without due credit.
(c) Social filtering mechanisms introduce biases of their own. If famous scientist XYZ and unknown author xyz each come up with the same idea at the same time, a formal reviewing process can at least strive to put them on an equal footing. In a social filtering model, ‘branding’ exists in any case, only it happens at the level of the individual. Formal reviewing can at least strive to separate fame from content.
There are of course good aspects to publish-then-filter. But network effects introduce their own (huge) biases — I would argue that evaluation of research under this model depends too much on connectivity rather than content.
Philippe Beaudoinsays:
Michiel, very interesting input. I beg to disagree though, for a couple of reasons.
– The current academic system, with regard to impact, is filter-then-publish-then-filter again. That’s the only way I can explain how some Siggraph papers have impact while others don’t. As such, they are still pretty much driven by network effects.
– The scenario you make up (one famous and one less famous person publishing the same paper) can occur in a F-t-P model and 9 times out of 10 the famous person will win and the other wont get anything. That’s because reviewers are not only the idea, but the ability to express it well in a fashion that is pleasing to reviewers. In a P-t-F model at least your stuff is up there and it has all chances of sparking a discussion.
– The network effects I see on the web today tend to reward people who contribute original content, people who cite their sources, people who connect with others. To me, the idea that fame is enough to gain you a lasting audience is deeply rooted in the dynamics of pre-web media. In a web community fame is important to make you a hub node, but if you fail to give credit where credit is due, the network’s backlash tend to be pretty harsh.
– Finally, it’s not enough that you can already do publish-then-filter today. If your community doesn’t adopt it, you have no one to connect with. You can’t be alone in your separate model, especially since it’s largely incompatible with today’s model, as we’ve often discussed.
Anyway, I believe most of academia is in too deep… What will happen is that parallel communities of researchers build on a P-t-F model will slowly emerge. Then we’ll be able to verify if you’re right or if I am, using the good old experimental scientific method. 😉
Donnysays:
Hi Daniel,
Thank you very much for a candid account of the “over-complication” of papers. It is really tiring to sort through the mass of over-complicated statements intended to elucidate a pretty straightforward point.
It is really cathartic to know that other people face this frustration too!
In my experience, peer reviewing is a useful thing, which may significantly improve the quality of the paper.
I entirely agree.
You cannot create the system that lacks favoritism unless you improve the people themselves!
People behave differently depending on the system they are in. An over-competitive, top-down system will generate one type of behavior. An open, bottom-up system will generate another type of behavior.
in the publish-then-filter model, nobody will point out to your errors. Nobody will improve your writings! Nobody will suggest relevant references!
On my blog there are many people pointing out my mistakes (you!), there are many people pointing out my poor writing (I get nasty emails at almost every typo) and people suggest a lot of references to me.
(…) by arguing that they were inappropriate (…)
The traditional peer review process leaves very little room for the author to argue and authors have a strong incentive to give to the reviewers exactly what they expect. This is even more true in conferences where you almost never have a chance to defend your paper. With acceptance rates at 10% or lower, every bit matters.
(1) If the paper is rejected, you will never get a chance to argue, period.
(2) You can write back to the reviewers, to explain that you disagree, but it is a lot of effort and there is a good chance that the reviewers will dismiss your arguments. It is a lot easier to go along.
Itmansays:
Phillipe,
commmon, the web fame is deeply flamed. You cannot create the system that lacks favoritism unless you improve the people themselves!
All models have drawbacks: in the publish-then-filter model, nobody will point out to your errors. Nobody will improve your writings! Nobody will suggest relevant references!
Itmansays:
Meant: the web fame is deeply flawed, sorry for the typo.
Itmansays:
Daniel,
First of all: yohohoho, they want everybody to use R-trees or similar methods. I could have gotten around this, by arguing that they were inappropriate in my case (which they probably were).
As to the system themselves, I am pretty sure that
1) Pure publish-then-filter system wont work;
2) Scientists should have blogs now, where they promote their work. In particular, explain it to the public;
3) Publish-then-filter and filter-then-publish should co-exist. The balance, however, matters. It will probably shift slightly towards to publish-than-filter.
In my experience, peer reviewing is a useful thing, which may significantly improve the quality of the paper. My experience also says that peers would suggest to simplify the paper rather than to make it more complex.
I agree that the system may foster incrementalism. Yet, I also believe that truly big ideas will find their way.
Itmansays:
O yes, conferences are bad. In regard to pointing out to the errors: yes, some people will highlight your mistakes, but not necessarily experts. To fix errors, you have to attract a lot of experts. Some authors will succeed at this, but most – will not. I am afraid that experienced and well-connected authors will get most experts.
Hey Daniel,
I agree that science does have its inconveniences. But I don’t think your idea is that good either.
A problem -and a big one- of blogs and social media is that the quality standards are sub-par. Most of the publishers lack a writing background and an editorial filter. Pieces are ill-written. And -like Zinserr puts it in his book- modern writing is just rubbish. We can accept people blogging about their dogs like that. But can we accept people writing scientific ideas like that.
If you look at some Chinese Journals, authors get accepted for friendship. The papers have terrible errors. Ideas are often mixed and ill posed and the authors have little idea on the concepts they’re using. Not to mention their English terrible.
On the other hand, a good reviewer may be your best friend. I have found much help from kind reviewers. Who propose reading references and point out writing errors. My papers are usually enhanced by their ideas. (One would be incredibly arrogant to think no one could give ideas to improve their papers)
And on the markets,
The QWERTY keyboard exists because the markets decided it should. Yet, it is far from being the ideal keyboard layout. Few people could argue that windows is the best OS, yet it rules the market. The best product don’t rule the markets. Is the most popular products the ones that does.
I just think Leon missed the point of your post.
Anyways, While what you say sounds decent and interesting a blog post outlining how the “publish-and-then-filter” approach would be implemented.
The blog was excellent and made for a very interesting read. It’s refreshing to hear such perspectives.
The other huge problem is that peer-review is a justification for putting academic research behind pay-walls. (Despite the fact that reviewers aren’t paid.)
Good academic research is now one of the least accessible kinds of information in the world, while paid propaganda and dumb speculations are rife.
The publishers with their captive market have no incentive to make it more visible. They know the universities have to keep subscribing to the journals.
The academics are usually decent enough, but as their objective is to please the reviewers and not piss off the publishers, and they have no incentive to gain a mass readership, they don’t systematically make their work visible as a matter of course.
And, of course, the work isn’t popular enough for anyone to make available on bittorrent.
The result is that academic writing and the conversation in the academy is kept artificially distinct from popular / internet discourse.
In an ideal world, good academics, who were identified and credited by the system, would also become influential public intellectuals by virtue of being *linked*.
As it is, their most important work is almost never visible to the web, never linked from wikipedia or discussions. And so has far less influence than it deserves.
Still lots of friction to adoption of a publish-then-filter model in academia. The arguments I hear most often
– – – –
1) Publish-then-filter makes it impossible to design a much-needed metric to allocate research funds.
This argument holds some merit, but IMHO it’s backwards. The reward system should be designed AFTER the publication system, not the other way around. If we seriously move to a publish-then-filter model (and if there is a real need for a metric) then I’m convinced we are creative enough to find one. Sure, it’s frightening, sure poor researchers who are able to game the current metric risk doing poorly, but I’m convinced we can design one that would reward researchers who truly work for the greater good of our society — what the reward system should have been about in the first place.
– – – –
2) Blogs are crappy, so any content following a publish-then-filter model will be crappy.
Wrong in so many ways:
– A lot of blogs are not crappy, those I run into given my online social filter tend to be much better (to me, at least) than a random journal paper.
– Journal paper hunting also requires filtering (your colleage, your advisor, a conference).
– If good writers start to publish-then-filter quality will increase.
If you insist on using this argument, please at least point out a structural reason why blogs would be of lower quality. Personally, I can’t think of any.
– – – –
I’d be really interested in hearing any other arguments against this model. I think we need to start advocating it seriously in academia, and a list of counter-arguments would be a good place to start.
I think all systems will have some flaws.
Here are some thoughts that come to mind for me:
(a) The “publish and filter” system already exists. Anyone can publish their work on their web page, in technical reports, workshops, in arXiv, and other fora, and many do.
(b) The branding of well-known journals and conferences can be hugely useful. If you are an unknown author and you make a grand break-through, publishing in a well known reviewed venue potentially gives you a huge leg up, where otherwise your article may languish in obscurity or be adopted by others without due credit.
(c) Social filtering mechanisms introduce biases of their own. If famous scientist XYZ and unknown author xyz each come up with the same idea at the same time, a formal reviewing process can at least strive to put them on an equal footing. In a social filtering model, ‘branding’ exists in any case, only it happens at the level of the individual. Formal reviewing can at least strive to separate fame from content.
There are of course good aspects to publish-then-filter. But network effects introduce their own (huge) biases — I would argue that evaluation of research under this model depends too much on connectivity rather than content.
Michiel, very interesting input. I beg to disagree though, for a couple of reasons.
– The current academic system, with regard to impact, is filter-then-publish-then-filter again. That’s the only way I can explain how some Siggraph papers have impact while others don’t. As such, they are still pretty much driven by network effects.
– The scenario you make up (one famous and one less famous person publishing the same paper) can occur in a F-t-P model and 9 times out of 10 the famous person will win and the other wont get anything. That’s because reviewers are not only the idea, but the ability to express it well in a fashion that is pleasing to reviewers. In a P-t-F model at least your stuff is up there and it has all chances of sparking a discussion.
– The network effects I see on the web today tend to reward people who contribute original content, people who cite their sources, people who connect with others. To me, the idea that fame is enough to gain you a lasting audience is deeply rooted in the dynamics of pre-web media. In a web community fame is important to make you a hub node, but if you fail to give credit where credit is due, the network’s backlash tend to be pretty harsh.
– Finally, it’s not enough that you can already do publish-then-filter today. If your community doesn’t adopt it, you have no one to connect with. You can’t be alone in your separate model, especially since it’s largely incompatible with today’s model, as we’ve often discussed.
Anyway, I believe most of academia is in too deep… What will happen is that parallel communities of researchers build on a P-t-F model will slowly emerge. Then we’ll be able to verify if you’re right or if I am, using the good old experimental scientific method. 😉
Hi Daniel,
Thank you very much for a candid account of the “over-complication” of papers. It is really tiring to sort through the mass of over-complicated statements intended to elucidate a pretty straightforward point.
It is really cathartic to know that other people face this frustration too!
@Itman
In my experience, peer reviewing is a useful thing, which may significantly improve the quality of the paper.
I entirely agree.
You cannot create the system that lacks favoritism unless you improve the people themselves!
People behave differently depending on the system they are in. An over-competitive, top-down system will generate one type of behavior. An open, bottom-up system will generate another type of behavior.
in the publish-then-filter model, nobody will point out to your errors. Nobody will improve your writings! Nobody will suggest relevant references!
On my blog there are many people pointing out my mistakes (you!), there are many people pointing out my poor writing (I get nasty emails at almost every typo) and people suggest a lot of references to me.
(…) by arguing that they were inappropriate (…)
The traditional peer review process leaves very little room for the author to argue and authors have a strong incentive to give to the reviewers exactly what they expect. This is even more true in conferences where you almost never have a chance to defend your paper. With acceptance rates at 10% or lower, every bit matters.
(1) If the paper is rejected, you will never get a chance to argue, period.
(2) You can write back to the reviewers, to explain that you disagree, but it is a lot of effort and there is a good chance that the reviewers will dismiss your arguments. It is a lot easier to go along.
Phillipe,
commmon, the web fame is deeply flamed. You cannot create the system that lacks favoritism unless you improve the people themselves!
All models have drawbacks: in the publish-then-filter model, nobody will point out to your errors. Nobody will improve your writings! Nobody will suggest relevant references!
Meant: the web fame is deeply flawed, sorry for the typo.
Daniel,
First of all: yohohoho, they want everybody to use R-trees or similar methods. I could have gotten around this, by arguing that they were inappropriate in my case (which they probably were).
As to the system themselves, I am pretty sure that
1) Pure publish-then-filter system wont work;
2) Scientists should have blogs now, where they promote their work. In particular, explain it to the public;
3) Publish-then-filter and filter-then-publish should co-exist. The balance, however, matters. It will probably shift slightly towards to publish-than-filter.
In my experience, peer reviewing is a useful thing, which may significantly improve the quality of the paper. My experience also says that peers would suggest to simplify the paper rather than to make it more complex.
I agree that the system may foster incrementalism. Yet, I also believe that truly big ideas will find their way.
O yes, conferences are bad. In regard to pointing out to the errors: yes, some people will highlight your mistakes, but not necessarily experts. To fix errors, you have to attract a lot of experts. Some authors will succeed at this, but most – will not. I am afraid that experienced and well-connected authors will get most experts.
Too bad for you you cannot apply to the The Howard Hughes Medical Institute.