What Reinhart-Rogoff means for the replication debate

Screen Shot 2013-04-19 at 9.38.04 AM
Economist Paul Krugman commented in the New York Times on the Reinhart and Rogoff replication scandal. He wrote: only after R-R “allowed” others to see their own data set, the spreadsheet errors could be detected. I don’t think it should be a matter of “allowing” anyone to see replication data of published work. Here are three four key points on what we can learn from this for the replication debate.

Screen Shot 2013-04-19 at 9.15.44 AMEconomist Paul Krugman commented in the New York Times on the recent replication scandal involving the Reinhart and Rogoff paper “Growth in a Time of Debt”. They claimed that growth and debt are correlated, and that when debt exceeds 90 percent of GDP, economic growth drops off. Several researchers had tried to replicate their results using data on growth and debt that they collected themselves. They could not find it, as Krugman describes:

Reinhart-Rogoff faced substantial criticism from the start, and the controversy grew over time. (…) Other researchers, using seemingly comparable data on debt and growth, couldn’t replicate the Reinhart-Rogoff results. They typically found some correlation between high debt and slow growth — but nothing that looked like a tipping point at 90 percent or, indeed, any particular level of debt.

Replication based on newly collected, comparable, data, did not match with the R-R results. People were wondering why the results differed. But without the original data set by R-R, they actually couldn’t. Says Krugman:

Finally, Ms. Reinhart and Mr. Rogoff allowed researchers at the University of Massachusetts to look at their original spreadsheet — and the mystery of the irreproducible results was solved. First, they omitted some data; second, they used unusual and highly questionable statistical procedures; and finally, yes, they made an Excel coding error. (emphasis added)

This scandal does not just have implications for how politicians justify austerity measures. What do we learn for the replication debate? Three points:


1. It should not be a matter of “allowing” access

First, no author should be able to publish without submitting their replication data set. It’s the journal’s job to make sure this happens. The R-R case shows: Had the data set been available early on (instead of the authors “allowing” others to see it at some point), the errors might have been found much earlier. Unfortunately, in political science only few journals have a replication data policy, and it can’t be much better in economics.

Update: The replication data set needs to be available early on, ideally at the same time the article is published. This ensures that checks are possible earlier, rather than later. This avoids costly policy mistakes. [Thank you Jonathon Moses (Norwegian University of Science and Technology) for sharing this point with me on April 19.]


2. Good replication data includes code and a manual

Second, authors must submit their Rscript, STATA code and a description on data transformation and all models. No one can replicate a study just from the description in the text, which is often vague and does not include all steps (especially on data coding). In the R-R case we can see that coding errors can happen, and that we can find them if we have enough information.


3. We need to re-analyze & collect new data

Third, when replicating work, one needs to include a re-analysis on the original data set, and, ideally, on newly collected data. In the R-R case, newly collected data had shown different results, but this could not be compared to the original data and models (they were not available). Also, if someone had re-run the models on the original data, without collecting comparable data, the errors might not have been found. There is a controversy if replication means re-analysis or new data collection. The R-R case shows that you need both.


Update: 4. Replication should be part of graduate education

And there’s a fourth point that might well be the most important one: Replication should be part of graduate education. One of the authors of the replication is a grad student who did this as a class assignment. In an interview with nymag.com, we learn that Thomas Herndon, an economics grad student at UMass Amherst, only looked into the Reinhart and Rogoff paper as part of an assignment for a course he took in econometrics. He chose the R-R paper because it was one of the most politically influential papers in the last years. Replication has been part of grad student methods training in a few universities: Gary King at Harvard, Victoria Stodden at Columbia University, and we are running a Cambridge Replication Workshop based on King and Stodden’s example right now. We need more of these!


Tagged , ,

2 thoughts on “What Reinhart-Rogoff means for the replication debate

  1. […] key to a change are journals, as I have argued before here and here. Journals can enforce better reproducibility and enhance quality of scholarly work. Currently, the […]


  2. […] student showed in his replication of the work. The case was picked up by the mainstream media, e.g. Economist Paul Krugman in the New York Times. What we learned from the scandal: Journals should require authors to provide their data set when […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: