Leading journal verifies articles before publication – so far, all replications failed

The American Journal of Political Science recently announced that every article will be externally verified before it is published. Authors have to provide all data, code and materials which will be verified by an external statistician. I talked to the editor William G. Jacoby about the goals of the new policy, and why no author has provided ‘perfect’ files yet.

How did you come up with the idea?
We are really picking up something the previous editor started. Previously, the authors were required to deposit their data, but we did not specify anything more. So the replication materials really varied enormously on what people provided and how usable they were. That’s why the editorial staff and I were thinking about providing more guidance. Any paper accepted for publication must provide all data and code, and it will be verified.

Is this the first journal in political science with such a strict policy?
There are some other journals in political science with replication policies, but none of them take it to the extent that we do. Some journals come close but don’t think anyone is providing the guidance and enforcing the quality control. Political Analysis is close. International Interactions does in-house verification, but no journal uses external verification by a third party.

Are you implementing the new policy already?
15 articles so far were accepted where authors had to send in their replication files.

Did the papers replicate?
Not a single one replicated. But let me clarify. Most of them have replication issues in which the author does not include all information to reproduce everything shown in the article. For example, they forgot the code for figure 1, or forgot to specify an R package. The issues were pretty minor but not one has passed through without requesting the author to send in more information.

Who exactly at the Odum Institute will conduct the pre-analysis?
Interns and graduate students do it but the results are checked by the staff who are data analysts.

Will existing (old) papers in your journal be checked as well?
No. We had so many submissions that we can’t handle that. We did have several inquiries about articles that were published before, where somebody asked for materials and the author did not put up all files online – it took some convincing and the author eventually did it. In another request from an article published in 2005 the author never responded to me when I asked on behalf of a researcher for the data. This illustrates why we need the new policy.

Will you publish the verification reports from the Odum institute alongside the paper?
There’s a technical data verification report that the Odum instiute provides, which goes to the author. That will not be published. But, this is not problematic because the report merely identifies the issues the author needs to address before we can proceed with publication. By the time the article appears in the Early View queue online, those issues will have been resolved.. We are also looking into a better way of linking the online version with the data files in the future.

Who is paying for the replications?
The Midwest Political Science Association has resources to support us. Most journals will probably not have the resources to do this – this sets us apart from other journals.

Note

Political Analysis requires in its replication policy that authors upload replication materials to the Political Analysis Dataverse and cite the link in the final version of the manuscript. “We will formally accept and then forward your final manuscript to production only after we have verified your compliance with this requirement.”

The dataverse for International Interactions is here.