It takes years for a theoretical idea to mature into quality code. The first implementations of an algorithm (and the first papers about it) tend to be bad, as you probably had no clear understanding of the algorithm when writing them. Yet if you continue working, you will eventually understand the algorithm well enough, so that you can describe it in simple terms and write a proper implementation of it.
In bioinformatics, new tools tend to use algorithms from 5 to 10 years ago. This is the time it takes first for the original authors to get the algorithm right, and then for the others to notice the algorithm, understand it, and see what it is good for.
John Regehrsays:
It sounds like you’re mixing up “high quality software” with “software that effectively solves a real problem.” These are different and I doubt that very much of Facebook’s software is gorgeous.
Back when I studied computer science, there really was very little training in terms of programming. People assumed that you already knew how to program. Later on there were classes on software engineering, but these were not mandatory (and pretty theoretical, too). There was also a strong feeling that programming is not “academic” enough. I doubt that that has changed in the mean-time, unfortunately.
In Academy, there are not only professors and students. There are also post-doc researchers, research assistants, collaborators. You could even hire developers for academy-related projects.
robbisgsays:
Well, the purposes of software development is different: software house build programs to sell it to a third part, thus it should be usable for final users. Researchers/PhD students, as explained, develop only for their purposes: the paper (or more romantically to grow human knowledge!). In my todo list there is always review code, but then other software modules have to be written and the priority of review drops!
The solution is to hire in academia software engineers that could coordinate research group coders and teach how to efficiently code.
I’ve read code that is less understandable than hieroglyphs, in my career at university!!
A great, but all too rare counter-example is Maple. It was released commercially at the end of the eighties (and is still popular these days) after I think 10 years of being built by the professors and students in the Symbolic Computation Lab in the University of Waterloo.
I worked with the code twenty years ago to port it to many different platforms. It was a grand example of abstraction and elegance, I really do wish that they’d publish it someday.
The article is generalizing a bit but there is value in what was written. Some universities do not prepare students for practical, modern day software development. Students exit with their degree with an excellent theoretical background and have little or no hands-on practice to producing excellent software. You cannot exclusively blame the universities. You’ll see a wide variety of curricula available that are highly theoretical, mix of theory and practical, and little theory + lots of hands-on. Students need to investigate before committing to a university.
It is a little unfair to paint a wide brush over professors saying they don’t code either. Some of the best “coders” I know are professors. Many universities own the content created by professors and thus cannot be published publicly. Similar to working for a closed-source software company. In the same vein, there is a competitive advantage for a stellar professor to keep her curricula available to paying students. I agree, however, if there is anything that could be shared then please do stick it up on Github!! 🙂
Furthermore, many universities make attempts to connect students with software companies to give them some practical skills via internships, co-ops and work experience. I encourage students to look for higher education institutions that provide those as part of the degree.
Finally, the point of the article could be used as an incentive to light a flame on existing universities and professors that don’t care. That is the sad part. I’ve worked with amazing institutions and professors that are super talented and nobody sees their awesomeness! But there some who do not care and do not want to progress where necessary. I think those are the types of people the author is referring to.
You have a point, of course, but there are some very promising counter-examples, where a community has changed the incentives: if cleaning up and documenting that ‘throw-away code’ will land you a high-impact publication, even academics will do an effort. And then the paper review process improves it further.
After working 12 years in computer science research, I can only wholeheartedly agree that academic software is bad for the most part. Students have too little experience and time. Academic teams are composed entirely differently than industrial teams. Profs feel they need implementation to establish their ideas and naively believe they are producing software industry can just take over.
Myrosiasays:
There is another problem with poor quality academic software. In fields where efficiency or accuracy of algorithms is compared between different methods, the software that is not released, or that cannot be easily run by other people, leads to non-replicable results and therefore may inhibit scientific progress.
Ted Pedersen had a great short piece on it in Computational Linguistics
He proposes that releasing code as opensource should become a requirement. Sadly, I think this proposal would be unrealistic now because the time and effort that has to go into making code usable by other people is not appreciated in academia. And yes, we could hire software engineers. We do sometime. But there is rarely money for it, because funders value publications above anything else.
Alessandrosays:
Daniel, what’s the incentive for researchers and professors to spend time writing good software instead of studying the new big idea that might make them publish another paper?
As others have pointed out, there are two main reasons why a professor might be interested in good software: to contribute to an open source project or to build a prototype that can be marketed in the outside world (either for commercial purposes or for disseminating the results of one’s research).
None of these reasons are valued/supported by universities.
Braden Talbotsays:
Sounds like academic anything. Academic marketers, advertisers, and entrepreneurs are no different. They don’t test ads, start legit business, our are accountable to a customer.
As John noted quality software and software that someone uses is a difference.
But yes, at least some unit tests here in there would be nice – especially when new students come into the team.
But another reason that professors don’t code “openly” is that they need to show that they are the first when publishing a paper. Or they fear that someone could steal their ideas or their code base and will outperform himself.
Also I can see a minor movement that researchers are on GitHub …
Edwardsays:
Great points Daniel. Here’s a favorite quote of mine from Donald Knuth on developing the typesetting program TeX:
“The most important lesson for me, was that software is hard; and it takes a long time. From now on I shall have significantly greater respect for every successful software tool that I encounter. My original plan was to spend one year working on algorithms for typography; but that was the spring of 1977, so you can see that it has taken more than eight years to finish the job.
Software creation not only takes time, it’s also much more difficult than I thought it would be. Why is this so? I think the main reason is that a longer attention span is needed when working on a large computer program than when doing other intellectual tasks. A great deal of technical information must be kept in one’s head, all at once, in high-speed random access memory somewhere in the brain. I found to my dismay that I could not be writing large computer programs while teaching my regular classes; I simply couldn’t do justice to activities simultaneously, nor could I be happy if the programs were left unwritten. So, I reluctantly took occasional leaves of absence from university teaching. In this sense I believe that program writing is substantially more demanding than book-writing.”
One note of optimism though. Software matters a lot in our world, and open source and collaboration are having an accelerating impact on culture and innovation in many, many ways. The next big ideas and advances are often possible only by leveraging and contributing to well crafted open source tools and codes. Proper academic credit for software craftsmanship seems inevitable to me, because the academy becomes less relevant as it is sidelined. Eventually, as universities get left in the dust in this respect and funding agencies make software quality and availability more central to grant review sections, the institutions will grudgingly admit that software craftsmanship is as important as other intellectual tasks, sufficiently codify criteria by which to measure and teach it (and then probably begin to take credit for it). Computer science itself was dismissed as a lesser intellectual activity in its early years. Silliness!
Ankursays:
As a PhD student, the burden of writing good software is on the student themselves. If the student doesn’t care about writing good software, the academic software is hopeless. But the goal of a researcher is publications and finding commercial/industry partners, software efficiency and design does not matter. The theoretical knowledge in your publications and new avenues of development of product matters the most. At the end of the day, researchers answer “Does this concept/theory work?” and the question “Is this software efficient and fast?” is highly irrelevant. Researchers work under the assumption that if they make their research into a product — they will follow software engineering principles. Personally, I hate looking at poorly written code but you do deal with a lot in Academia.
This might be true in the practice of the business of computer science, but it does raise some questions:
If your code is ugly, untested and undocumented… how do you know that it does what you claim that it does?
Many papers and thesis say something like this “we have improved the speed by a factor of X”. If your code is all around inefficient, how can this “X” be meaningful? What does it mean that one inefficient and poorly tested piece of code is faster than another inefficient and poorly tested piece of code?
If you don’t know how to code properly, and you can’t find an academic position, what kind of job can you hope for?
Ankursays:
@Dan
I don’t think PhD Students are software engineers but having software engineering skills do make them stand out. PhD student’s purpose is to do research and not come up with brilliant scalability paradigms that software engineers deal with. They are just performing concept/theory checks.
1. The code can surely be ugly and undocumented, but not untested. If the results are not right based on simple use-cases and model-based cases (such as phantoms and datasets that you know the truth for), the code is wrong or your concept for your algorithm is wrong. The code always does what it is suppose to do i.e. produce the result consistent with the theory in the publication. If the code is untested, your results will not be well understood in the peer-review process and thus, no publication. Academia works based on reputation and no one is willing to harm their reputation by publishing suboptimal results based on incorrect code.
2. If a paper claims a “speedup of a factor X,” then this entails that you do care about software efficiency and thus, you have made coding choices to gain that efficiency. Inefficient code can almost never lead to a speedup, and researchers will tarnish their reputation if they claim something that is just not true.
3. If you don’t know how to code properly, and but your research is great, you can still land an academic position as the typical Professor who has lost their touch with software. If you do know how to code and are bad at research, you can land a job in any software firm. If you are bad at both, become a consultant.
Academia works based on reputation and no one is willing to harm their reputation by publishing (…) based on incorrect code.
But they don’t publish their code! Who is going to tell that the code is incorrect?
Inefficient code can almost never lead to a speedup, (…)
Sure it can, because researchers typically implement both algorithm A and algorithm B. Even if they are both inefficient, one can be faster than the other.
If you don’t know how to code properly, and but your research is great, you can still land an academic position
Absolutely, but it is a lot easier to find a good job as a developer than as an academic researcher.
Alessandrosays:
“PhD student’s purpose is to do research and not come up with brilliant scalability paradigms that software engineers deal with”
which means no real-life implementation of your ideas and algorithms.
and that’s why the great majority of academic research, it may be brilliant, but won’t make any impact in the industry.
Mariesays:
Great article!
As PhD student myself, I have to admit that in the beginning of my grad career, I wrote pretty awful code that I thought would only be seen and run by myself. I’ve learned a great deal since then, after doing an internship and reading far better code from my fellow peers and seniors.
Based on my experience, I think that grad students in systems research such as architecture/distributed systems/low-level databases tend to be better programmers than those in machine learning/NLP , where prediction accuracy, precision, recall are the numbers to report, as opposed to efficiency metrics. Building big and efficient systems usually requires good architectural design, which means students must be planning well beforehand before they start to build anything. Added to that, students in systems research usually tend to work together on big umbrella projects funded by advisors, which means that they have to share their components/modules to avoid duplication; at the very least, their code has to be readable.
Overall, I think a grad student’s time is better spent on doing just research-related activities, i.e. reading/writing papers, discussing ideas and collaborating, rather than on improving prototype code. For most students, we only have 5-6 years of being in academic environment, and at least on my part, I’d prefer to spend that time improving skills that will make me a better researcher rather than engineer. This is really not to stay that programming skills are not important (because I know I will be a future engineer), just secondary in my priorities at the moment.
Itmansays:
I totally agree that research coode should have production-like quality from the get-go. In particular, all kind of testing is a must?
Why? Because it directly relates to the quality of research. I don’t wanna read result based on faulty software. In addition, building good-enough software saves time when research code is reused in other projects.
I’d note however that most so called industry code is ugly and often times way uglier than research prototypes. It is not easy to beautify this crap, because it is fragile.
It takes years for a theoretical idea to mature into quality code. The first implementations of an algorithm (and the first papers about it) tend to be bad, as you probably had no clear understanding of the algorithm when writing them. Yet if you continue working, you will eventually understand the algorithm well enough, so that you can describe it in simple terms and write a proper implementation of it.
In bioinformatics, new tools tend to use algorithms from 5 to 10 years ago. This is the time it takes first for the original authors to get the algorithm right, and then for the others to notice the algorithm, understand it, and see what it is good for.
It sounds like you’re mixing up “high quality software” with “software that effectively solves a real problem.” These are different and I doubt that very much of Facebook’s software is gorgeous.
Back when I studied computer science, there really was very little training in terms of programming. People assumed that you already knew how to program. Later on there were classes on software engineering, but these were not mandatory (and pretty theoretical, too). There was also a strong feeling that programming is not “academic” enough. I doubt that that has changed in the mean-time, unfortunately.
Aren’t you overgeneralising?
In Academy, there are not only professors and students. There are also post-doc researchers, research assistants, collaborators. You could even hire developers for academy-related projects.
Well, the purposes of software development is different: software house build programs to sell it to a third part, thus it should be usable for final users. Researchers/PhD students, as explained, develop only for their purposes: the paper (or more romantically to grow human knowledge!). In my todo list there is always review code, but then other software modules have to be written and the priority of review drops!
The solution is to hire in academia software engineers that could coordinate research group coders and teach how to efficiently code.
I’ve read code that is less understandable than hieroglyphs, in my career at university!!
@robbisg
A great, but all too rare counter-example is Maple. It was released commercially at the end of the eighties (and is still popular these days) after I think 10 years of being built by the professors and students in the Symbolic Computation Lab in the University of Waterloo.
I worked with the code twenty years ago to port it to many different platforms. It was a grand example of abstraction and elegance, I really do wish that they’d publish it someday.
Paul.
@robbisg
“I’ve read code that is less understandable than hieroglyphs”
Great quote.
The article is generalizing a bit but there is value in what was written. Some universities do not prepare students for practical, modern day software development. Students exit with their degree with an excellent theoretical background and have little or no hands-on practice to producing excellent software. You cannot exclusively blame the universities. You’ll see a wide variety of curricula available that are highly theoretical, mix of theory and practical, and little theory + lots of hands-on. Students need to investigate before committing to a university.
It is a little unfair to paint a wide brush over professors saying they don’t code either. Some of the best “coders” I know are professors. Many universities own the content created by professors and thus cannot be published publicly. Similar to working for a closed-source software company. In the same vein, there is a competitive advantage for a stellar professor to keep her curricula available to paying students. I agree, however, if there is anything that could be shared then please do stick it up on Github!! 🙂
Furthermore, many universities make attempts to connect students with software companies to give them some practical skills via internships, co-ops and work experience. I encourage students to look for higher education institutions that provide those as part of the degree.
Finally, the point of the article could be used as an incentive to light a flame on existing universities and professors that don’t care. That is the sad part. I’ve worked with amazing institutions and professors that are super talented and nobody sees their awesomeness! But there some who do not care and do not want to progress where necessary. I think those are the types of people the author is referring to.
To read more on my thoughts about software in education you can read more on my blog: http://blog.dudeblake.com/2009/09/improving-traditional-software.html
You have a point, of course, but there are some very promising counter-examples, where a community has changed the incentives: if cleaning up and documenting that ‘throw-away code’ will land you a high-impact publication, even academics will do an effort. And then the paper review process improves it further.
This is what the open-source track of JMLR has done for the field of machine learning: http://jmlr.csail.mit.edu/mloss/.
After working 12 years in computer science research, I can only wholeheartedly agree that academic software is bad for the most part. Students have too little experience and time. Academic teams are composed entirely differently than industrial teams. Profs feel they need implementation to establish their ideas and naively believe they are producing software industry can just take over.
There is another problem with poor quality academic software. In fields where efficiency or accuracy of algorithms is compared between different methods, the software that is not released, or that cannot be easily run by other people, leads to non-replicable results and therefore may inhibit scientific progress.
Ted Pedersen had a great short piece on it in Computational Linguistics
Empiricism is not a matter of faith
Computational Linguistics,34(3), Pages 465-470
http://www.aclweb.org/anthology/J/J08/J08-3010.pdf
He proposes that releasing code as opensource should become a requirement. Sadly, I think this proposal would be unrealistic now because the time and effort that has to go into making code usable by other people is not appreciated in academia. And yes, we could hire software engineers. We do sometime. But there is rarely money for it, because funders value publications above anything else.
Daniel, what’s the incentive for researchers and professors to spend time writing good software instead of studying the new big idea that might make them publish another paper?
As others have pointed out, there are two main reasons why a professor might be interested in good software: to contribute to an open source project or to build a prototype that can be marketed in the outside world (either for commercial purposes or for disseminating the results of one’s research).
None of these reasons are valued/supported by universities.
Sounds like academic anything. Academic marketers, advertisers, and entrepreneurs are no different. They don’t test ads, start legit business, our are accountable to a customer.
As John noted quality software and software that someone uses is a difference.
But yes, at least some unit tests here in there would be nice – especially when new students come into the team.
But another reason that professors don’t code “openly” is that they need to show that they are the first when publishing a paper. Or they fear that someone could steal their ideas or their code base and will outperform himself.
Also I can see a minor movement that researchers are on GitHub …
Great points Daniel. Here’s a favorite quote of mine from Donald Knuth on developing the typesetting program TeX:
“The most important lesson for me, was that software is hard; and it takes a long time. From now on I shall have significantly greater respect for every successful software tool that I encounter. My original plan was to spend one year working on algorithms for typography; but that was the spring of 1977, so you can see that it has taken more than eight years to finish the job.
Software creation not only takes time, it’s also much more difficult than I thought it would be. Why is this so? I think the main reason is that a longer attention span is needed when working on a large computer program than when doing other intellectual tasks. A great deal of technical information must be kept in one’s head, all at once, in high-speed random access memory somewhere in the brain. I found to my dismay that I could not be writing large computer programs while teaching my regular classes; I simply couldn’t do justice to activities simultaneously, nor could I be happy if the programs were left unwritten. So, I reluctantly took occasional leaves of absence from university teaching. In this sense I believe that program writing is substantially more demanding than book-writing.”
One note of optimism though. Software matters a lot in our world, and open source and collaboration are having an accelerating impact on culture and innovation in many, many ways. The next big ideas and advances are often possible only by leveraging and contributing to well crafted open source tools and codes. Proper academic credit for software craftsmanship seems inevitable to me, because the academy becomes less relevant as it is sidelined. Eventually, as universities get left in the dust in this respect and funding agencies make software quality and availability more central to grant review sections, the institutions will grudgingly admit that software craftsmanship is as important as other intellectual tasks, sufficiently codify criteria by which to measure and teach it (and then probably begin to take credit for it). Computer science itself was dismissed as a lesser intellectual activity in its early years. Silliness!
As a PhD student, the burden of writing good software is on the student themselves. If the student doesn’t care about writing good software, the academic software is hopeless. But the goal of a researcher is publications and finding commercial/industry partners, software efficiency and design does not matter. The theoretical knowledge in your publications and new avenues of development of product matters the most. At the end of the day, researchers answer “Does this concept/theory work?” and the question “Is this software efficient and fast?” is highly irrelevant. Researchers work under the assumption that if they make their research into a product — they will follow software engineering principles. Personally, I hate looking at poorly written code but you do deal with a lot in Academia.
@Ankur
Software efficiency and design does not matter.
This might be true in the practice of the business of computer science, but it does raise some questions:
@Dan
I don’t think PhD Students are software engineers but having software engineering skills do make them stand out. PhD student’s purpose is to do research and not come up with brilliant scalability paradigms that software engineers deal with. They are just performing concept/theory checks.
1. The code can surely be ugly and undocumented, but not untested. If the results are not right based on simple use-cases and model-based cases (such as phantoms and datasets that you know the truth for), the code is wrong or your concept for your algorithm is wrong. The code always does what it is suppose to do i.e. produce the result consistent with the theory in the publication. If the code is untested, your results will not be well understood in the peer-review process and thus, no publication. Academia works based on reputation and no one is willing to harm their reputation by publishing suboptimal results based on incorrect code.
2. If a paper claims a “speedup of a factor X,” then this entails that you do care about software efficiency and thus, you have made coding choices to gain that efficiency. Inefficient code can almost never lead to a speedup, and researchers will tarnish their reputation if they claim something that is just not true.
3. If you don’t know how to code properly, and but your research is great, you can still land an academic position as the typical Professor who has lost their touch with software. If you do know how to code and are bad at research, you can land a job in any software firm. If you are bad at both, become a consultant.
Academia works based on reputation and no one is willing to harm their reputation by publishing (…) based on incorrect code.
But they don’t publish their code! Who is going to tell that the code is incorrect?
Inefficient code can almost never lead to a speedup, (…)
Sure it can, because researchers typically implement both algorithm A and algorithm B. Even if they are both inefficient, one can be faster than the other.
If you don’t know how to code properly, and but your research is great, you can still land an academic position
Absolutely, but it is a lot easier to find a good job as a developer than as an academic researcher.
“PhD student’s purpose is to do research and not come up with brilliant scalability paradigms that software engineers deal with”
which means no real-life implementation of your ideas and algorithms.
and that’s why the great majority of academic research, it may be brilliant, but won’t make any impact in the industry.
Great article!
As PhD student myself, I have to admit that in the beginning of my grad career, I wrote pretty awful code that I thought would only be seen and run by myself. I’ve learned a great deal since then, after doing an internship and reading far better code from my fellow peers and seniors.
Based on my experience, I think that grad students in systems research such as architecture/distributed systems/low-level databases tend to be better programmers than those in machine learning/NLP , where prediction accuracy, precision, recall are the numbers to report, as opposed to efficiency metrics. Building big and efficient systems usually requires good architectural design, which means students must be planning well beforehand before they start to build anything. Added to that, students in systems research usually tend to work together on big umbrella projects funded by advisors, which means that they have to share their components/modules to avoid duplication; at the very least, their code has to be readable.
Overall, I think a grad student’s time is better spent on doing just research-related activities, i.e. reading/writing papers, discussing ideas and collaborating, rather than on improving prototype code. For most students, we only have 5-6 years of being in academic environment, and at least on my part, I’d prefer to spend that time improving skills that will make me a better researcher rather than engineer. This is really not to stay that programming skills are not important (because I know I will be a future engineer), just secondary in my priorities at the moment.
I totally agree that research coode should have production-like quality from the get-go. In particular, all kind of testing is a must?
Why? Because it directly relates to the quality of research. I don’t wanna read result based on faulty software. In addition, building good-enough software saves time when research code is reused in other projects.
I’d note however that most so called industry code is ugly and often times way uglier than research prototypes. It is not easy to beautify this crap, because it is fragile.