Leading journal verifies articles before publication – so far, all replications failed

The American Journal of Political Science recently announced that every article will be externally verified before it is published. Authors have to provide all data, code and materials which will be verified by an external statistician. I talked to the editor William G. Jacoby about the goals of the new policy, and why no author has provided ‘perfect’ files yet.

How did you come up with the idea?
We are really picking up something the previous editor started. Previously, the authors were required to deposit their data, but we did not specify anything more. So the replication materials really varied enormously on what people provided and how usable they were. That’s why the editorial staff and I were thinking about providing more guidance. Any paper accepted for publication must provide all data and code, and it will be verified.

Is this the first journal in political science with such a strict policy?
There are some other journals in political science with replication policies, but none of them take it to the extent that we do. Some journals come close but don’t think anyone is providing the guidance and enforcing the quality control. Political Analysis is close. International Interactions does in-house verification, but no journal uses external verification by a third party.

Are you implementing the new policy already?
15 articles so far were accepted where authors had to send in their replication files.

Did the papers replicate?
Not a single one replicated. But let me clarify. Most of them have replication issues in which the author does not include all information to reproduce everything shown in the article. For example, they forgot the code for figure 1, or forgot to specify an R package. The issues were pretty minor but not one has passed through without requesting the author to send in more information.

Who exactly at the Odum Institute will conduct the pre-analysis?
Interns and graduate students do it but the results are checked by the staff who are data analysts.

Will existing (old) papers in your journal be checked as well?
No. We had so many submissions that we can’t handle that. We did have several inquiries about articles that were published before, where somebody asked for materials and the author did not put up all files online – it took some convincing and the author eventually did it. In another request from an article published in 2005 the author never responded to me when I asked on behalf of a researcher for the data. This illustrates why we need the new policy.

Will you publish the verification reports from the Odum institute alongside the paper?
There’s a technical data verification report that the Odum instiute provides, which goes to the author. That will not be published. But, this is not problematic because the report merely identifies the issues the author needs to address before we can proceed with publication. By the time the article appears in the Early View queue online, those issues will have been resolved.. We are also looking into a better way of linking the online version with the data files in the future.

Who is paying for the replications?
The Midwest Political Science Association has resources to support us. Most journals will probably not have the resources to do this – this sets us apart from other journals.

Note

Political Analysis requires in its replication policy that authors upload replication materials to the Political Analysis Dataverse and cite the link in the final version of the manuscript. “We will formally accept and then forward your final manuscript to production only after we have verified your compliance with this requirement.”

The dataverse for International Interactions is here.

Advertisements
Tagged , , , ,

12 thoughts on “Leading journal verifies articles before publication – so far, all replications failed

  1. George says:

    Shocking, but not surprising. To be a publishing social science quant, doctored, concocted data is a must. That was Stapel’s experience. (Stapel was a well-known quant sociologist who made-up his data for years.)

    Like

  2. I’m pretty sure QJPS has had a policy like this for a decade or so.

    http://www.nowpublishers.com/Journal/AuthorInstructions/QJPS

    Like

  3. Requin says:

    Political Science Research & Methods, the journal of the new(ish) European Political Science Association, also requires replication files prior to publication, and they won’t publish anything that isn’t replicable.

    Like

  4. Michael says:

    The link for the International Interactions dataverse is incomplete. The current dataverse that covers 2010-current is: https://dataverse.harvard.edu/dataverse/internationalinteractions

    Like

  5. “The issues were pretty minor but not one has passed through without requesting the author to send in more information.”

    I’d be interested to learn, after requesting the author to send in more information,

    1. What proportion of authors of the 15 papers submitted additional information?
    2. What proportion of the additional information submitted enabled replication?

    Like

    • George says:

      More importantly, will the AJPS publicly name those authors who could not be replicated — even after being asked to reform their data, or will such authors be simply allowed to submit their questionable/dubious research to other journals (which do not replicate accepted papers).

      Like

  6. […] Presentation Paradigm – Software development skills for data scientists (via @fazol) – Leading journal verifies articles before publication – so far, all replications failed – Explained Visually – The Pseudo Third Dimension (h/t […]

    Like

  7. […] is getting more attention from top journals: the American Journal of Political Science announced all articles submitted for publication must pass a replication test by an external […]

    Like

  8. […] already cleaned data will only tell you so much. The American Journal of Political Science actually already conducts such pre-publication checks, but either this article was published before they started it, or they […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: