, 3 min read
Share your software early: the Reinhart-Rogoff case
I like stories where prestigious professors screw up spectacularly. It reminds us that everybody gets it wrong some of the time. The Reinhart-Rogoff story is one such case.
Two very reputed professors at Harvard University, Carmen Reinhart and Kenneth Rogoff, published online a working paper in 2010 called Growth in a time of debt. The paper has since been cited in the scientific literature about 500 times, or about once every two days. They later wrote a best-seller book on the same topic. They built a web site to promote their ideas where they shared some of their data.
Their research paper is remarkably easy to understand: countries with large debt tend to have lesser economic growth. This claim is entirely credible. On the one hand, countries with poor economies may tend to grow large debts because they have weak revenues. On the other hand, large debts may lead to tight fiscal policies (e.g., high taxes) or lax monetary policies (e.g., printing money) that may reduce economic growth.
The part that is less credible is that Reinhart and Rogoff were able to find a magical threshold (debt-to-GDP ratio of 0.9) that triggered low growth. In effect, they implicitly claimed that any debt-to-GDP ratio in excess of 0.9 would reduce your economic growth. This gave politicians a scientific justification for austerity measures (e.g., raising taxes and cutting back on government services).
The story then becomes interesting. Thomas Herndon, a graduate student, grabbed data from the Reinhart-Rogoff web site and tried to reproduce their results. He couldn’t. He then asked the authors for help. Reinhart and Rogoff shared the Excel spreadsheet they used with Herndon. He then promptly found basic flaws in their data processing. For example, the two professors ran sums over the wrong cells. It seems that they made several odd choices when processing the data. His paper is freely available online.
Things took a turn for the worse when, faced with this opposition, Reinhart-Rogoff balked and asserted that these errors did not impact their work. Thankfully, they did not deny their mistakes.
This whole incident shows the importance of sharing data and software. Reinhart and Rogoff were almost exemplary regarding data in this case: they widely shared their data. Their mistake was in not widely distributing their software (in the form a spreadsheet) earlier. Had the spreadsheet been available from the start, they would be in a much better position. Of course, another version of the story could be that, had Reinhart and Rogoff not shared their data and software, they would have plausible deniability and their work could still be credible. But this only means that you, as a reader, should put more trust in work where the data and software are available. In any case, by keeping the software private, the best that Reinhart and Rogoff could have hoped for was to delay the inevitable: once your work is the basis for public policy, it is bound to come under intense scrutiny and significant mistakes will be discovered. By sharing your software, you establish your good faith.
There are other minor points that I find interesting in this story:
- All this work is posted online. To my knowledge, no journal has been directly involved.
- Usually, negative results are unpublishable: journals are not interested. It is unclear that the paper by Herndon et al. can be published in a conventional journal even though it is obviously important work.- It seems that Reinhart and Rogoff are credited for much of the suffering due to the austerity measures in Europe. This seems entirely ridiculous to me. I think that Reinhart and Rogoff acted in good faith.
Further reading: Influential Reinhart-Rogoff economics paper suffers spreadsheet error (via Scott Guthery) and My take on Reinhart and Rogoff by David Henderson.