Guest post: How to persuade journals to accept your replication paper

MIT students Mark Bell and Nicholas Miller recently got their replication paper “Questioning the Effect of Nuclear Weapons on Conflict” accepted by the Journal of Conflict Resolution. They wrote the paper as an assignment for Gary King‘s class on Advanced Quantitative Research Methodology. They first published their analysis on a dataverse, and then rewrote the paper for the journal. In this guest post they show how they persuaded the journal and its reviewers to publish the replication.

We are PhD candidates in political science at MIT, and our paper, “Questioning the Effect of Nuclear Weapons on Conflict,” is now forthcoming in the Journal of Conflict Resolution (JCR). This paper began as a replication of Robert Rauchhaus’ 2009 article “Evaluating the Nuclear Peace Hypothesis: A Quantitative Approach,” that we wrote up for Gary King’s Gov 2001 class at Harvard in Spring of 2011. Here are a few lessons we learnt along the way that may be useful to others thinking of trying to publish replication papers (for a more comprehensive set of pointers, we recommend Gary King’s “Publication, Publication”).

Software is not interchangeable

We were easily able to replicate Rauchhaus’ key findings in Stata, but couldn’t get it to work in R. It took us a long while to work out why, but the reason turned out to be an error in Stata: Stata was finding a solution when it shouldn’t have (because of separation in the data). This solution, as we show in the paper, was wrong – and led Rauchhaus’ paper to overestimate the effect of nuclear weapons on conflict by a factor of several million.

It’s very easy when you’re working through someone else’s code to be satisfied when you’ve successfully got your computer to produce the same numbers you see in the paper. But replicating a published result in one software package does not mean that you necessarily understand what that software package is doing, that the software is doing it correctly, or that doing it at all is the appropriate thing to do – in our case none of those were true and working out why was key to writing our paper.

Go beyond pointing out errors

When you replicate, don’t stop once you’ve found mistakes in the original paper. It’s much easier to tear down the work of others than it is to build your own, but the latter is much more likely to be published. There are mistakes or questionable methodological choices in almost all published work, and unless the original paper is very influential (eg, Reinhart and Rogoff), the audience for pure critiques is likely to be fairly narrow.

We initially wrote a very short critique of the original Rauchhaus paper (around 3 pages long) and submitted it to JCR, only to have it desk rejected for being too short. However, once we refocused the paper away from being solely critical and produced new substantive findings on nuclear weapons and conflict, we ultimately got accepted.

Don’t worry if your paper isn’t going to revolutionize the field

Although replication papers should go beyond criticism, they don’t have to represent revolutionary contributions to the field to get published. Incremental improvements upon previous work are still an improvement, are beneficial to the accumulation of knowledge, and can still be published. “A marginal, but nevertheless important, empirical contribution” (as one of our reviewers described our paper) can still be worthy of publication.

Replications are part of your methods training

As you get in to the process of replication, don’t just use the methods you have been taught as part of your methods sequence – use it as an opportunity to learn about methods you haven’t been taught. Neither of us would have thought a great deal about the issue of separation if we hadn’t come across it in the Rauchhaus paper, but since we did, we’re now both aware of the seriousness of the problem and potential ways to deal with it.

Learning new methods as you conduct the replication ensures that even if you don’t end up publishing your replication paper, you’ll have learnt new skills that will be valuable down the road.

Mark Bell and Nicholas Miller
 

About Mark Bell

Mark Bell is a PhD student in Political Science at MIT with interests in international relations, political methodology and security studies, with a focus on issues relating to nuclear proliferation. He holds an MPP from the Harvard Kennedy School, where he was a Frank Knox Fellow, and a BA in Politics, Philosophy and Economics from St Anne’s College, Oxford University. He previously worked for the British think-tank CentreForum and for Senator John Kerry. [website]

About Nicholas Miller

Nicholas Miller is a Ph.D. candidate at MIT in international relations. His interests include nuclear proliferation, foreign military intervention, and civil war. His research is forthcoming in International Organization, the Journal of Conflict Resolution, and Security Studies. Prior to MIT, he graduated Phi Beta Kappa with a degree in Government from Wesleyan University. [website]
 

About their paper

Abstract: We examine the effect of nuclear weapons on interstate conflict. Using more appropriate methodologies than have previously been used, we find that dyads in which both states possess nuclear weapons are not significantly less likely to fight wars, nor are they significantly more or less belligerent at low levels of conflict. This stands in contrast to previous work, which suggests nuclear dyads are some 2.7 million times less likely to fight wars. We additionally find that dyads in which one state possesses nuclear weapons are more prone to low-level conflict (but not more prone to war). This appears to be because nuclear-armed states expand their interests after nuclear acquisition rather than because nuclear weapons provide a shield behind which states can aggress against more powerful conventional-armed states. This calls into question conventional wisdom on the impact of nuclear weapons and has policy implications for the impact of nuclear proliferation. [Paper | Analysis and Data]

Read more…

You might also want to read “Getting published by replicating work“, “Publishing as a grad student: replication helps” and other articles on replication publication.