Economist Paul Krugman commented in the New York Times on the Reinhart and Rogoff replication scandal. He wrote: only after R-R “allowed” others to see their own data set, the spreadsheet errors could be detected. I don’t think it should be a matter of “allowing” anyone to see replication data of published work. Here are
three four key points on what we can learn from this for the replication debate.
Economist Paul Krugman commented in the New York Times on the recent replication scandal involving the Reinhart and Rogoff paper “Growth in a Time of Debt”. They claimed that growth and debt are correlated, and that when debt exceeds 90 percent of GDP, economic growth drops off. Several researchers had tried to replicate their results using data on growth and debt that they collected themselves. They could not find it, as Krugman describes:
Reinhart-Rogoff faced substantial criticism from the start, and the controversy grew over time. (…) Other researchers, using seemingly comparable data on debt and growth, couldn’t replicate the Reinhart-Rogoff results. They typically found some correlation between high debt and slow growth — but nothing that looked like a tipping point at 90 percent or, indeed, any particular level of debt.
Replication based on newly collected, comparable, data, did not match with the R-R results. People were wondering why the results differed. But without the original data set by R-R, they actually couldn’t. Says Krugman:
Finally, Ms. Reinhart and Mr. Rogoff allowed researchers at the University of Massachusetts to look at their original spreadsheet — and the mystery of the irreproducible results was solved. First, they omitted some data; second, they used unusual and highly questionable statistical procedures; and finally, yes, they made an Excel coding error. (emphasis added)
This scandal does not just have implications for how politicians justify austerity measures. What do we learn for the replication debate? Three points:
1. It should not be a matter of “allowing” access
First, no author should be able to publish without submitting their replication data set. It’s the journal’s job to make sure this happens. The R-R case shows: Had the data set been available early on (instead of the authors “allowing” others to see it at some point), the errors might have been found much earlier. Unfortunately, in political science only few journals have a replication data policy, and it can’t be much better in economics.
Update: The replication data set needs to be available early on, ideally at the same time the article is published. This ensures that checks are possible earlier, rather than later. This avoids costly policy mistakes. [Thank you Jonathon Moses (Norwegian University of Science and Technology) for sharing this point with me on April 19.]
2. Good replication data includes code and a manual
Second, authors must submit their Rscript, STATA code and a description on data transformation and all models. No one can replicate a study just from the description in the text, which is often vague and does not include all steps (especially on data coding). In the R-R case we can see that coding errors can happen, and that we can find them if we have enough information.
3. We need to re-analyze & collect new data
Third, when replicating work, one needs to include a re-analysis on the original data set, and, ideally, on newly collected data. In the R-R case, newly collected data had shown different results, but this could not be compared to the original data and models (they were not available). Also, if someone had re-run the models on the original data, without collecting comparable data, the errors might not have been found. There is a controversy if replication means re-analysis or new data collection. The R-R case shows that you need both.
Update: 4. Replication should be part of graduate education
And there’s a fourth point that might well be the most important one: Replication should be part of graduate education. One of the authors of the replication is a grad student who did this as a class assignment. In an interview with nymag.com, we learn that Thomas Herndon, an economics grad student at UMass Amherst, only looked into the Reinhart and Rogoff paper as part of an assignment for a course he took in econometrics. He chose the R-R paper because it was one of the most politically influential papers in the last years. Replication has been part of grad student methods training in a few universities: Gary King at Harvard, Victoria Stodden at Columbia University, and we are running a Cambridge Replication Workshop based on King and Stodden’s example right now. We need more of these!
- Original paper: Reinhart, Carmen M., and Kenneth S. Rogoff. 2010. “Growth in a Time of Debt.” , American Economic Review Papers and Proceedings 100(2): 573-78. [pdf]
- Replication paper: Thomas Herndon, Michael Ash & Robert Pollin, Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff [pdf]
- Paul Krugman in the New York Times: The Excel Depression
- Interview with the author of the replication paper: Meet the 28-Year-Old Grad Student Who Just Shook the Global Austerity Movement
- Radio interview with the authors of the replication
- University of Michigan: Nicely categorized list of links and articles on this topic titled “Microsoft Excel: The Ruiner of Global Economies?“