Nightmare after nightmare: students trying to replicate work

Trying to replicate a published paper can be challenging. Often, authors do not remember their variable codings, models specifications, or where their files are. My students of the Cambridge Replication Workshop just finished their final assignments – here are their ‘horror’ stories.

What Reinhart-Rogoff means for the replication debate

Economist Paul Krugman commented in the New York Times on the Reinhart and Rogoff replication scandal. He wrote: only after R-R “allowed” others to see their own data set, the spreadsheet errors could be detected. I don’t think it should be a matter of “allowing” anyone to see replication data of published work. Here are three four key points on what we can learn from this for the replication debate.

Replication scandal: We might not need austerity measures after all

Economists Carmen Reinhart and Kenneth Rogoff claim in “Growth in a Time of Debt” that countries with debt-to-GDP ratios above 90 percent have a slightly negative average growth rate. Their paper is one of the most cited by governments to justify austerity measures – and it might be wrong, as a replication paper shows.

Only 18 of 120 political science journals have a replication policy

A recent article in “European Political Science” found that out of 120 political science journals only 18 have replication policies. The authors found that the existence of a data policy was correlated with the impact factor, type of audience, and age of the journal. Their analysis confirms that the state of reproducibility and political science is substandard.

“My doctoral institution bought the data”

Nicole,

(…) The ____ dataset I am afraid I can’t give you, because my doctoral institution bought it and it is proprietary. (…)

Best,
___________

 

More replication correspondence

I collect more responses from original authors here.

“Can’t make data available because of contractual agreement”

Dear Nicole,

(…) We can’t make our ____ data available because of our contractual agreement with ___. (…)

Best wishes,

________

 

More replication correspondence

I collect more responses from original authors here.

Replication Frustration in Political Science

“I don’t have a ready-made dataset.” – “We don’t have the R code for our paper available.” – “I’m travelling. I will definitely send the replication data when I can clean it up a bit.” These are just some of the answers I received when asking authors for their replication data in political science. Only few sent me replication-ready data and almost no one sent me their code or .do file. This is a serious case of replication frustration!