Journal introduces replication pre-check

Many political journals have no replication policy, or only implement it half-heartedly. Now a leading journal introduces innovative guidelines for authors that include a verification of the analysis before publication.

ajps replicationThe American Journal of Political Science has criticized that the discipline has “paid lip service” to reproducibility in the past. In order to “practice what we preach”, the journal now uses a replication policy adhering to the highest standards in research transparency.

How to prepare replication files

The “Guidelines for Preparing Replication Files” [pdf] state that replication data must contain:

  • dataset
  • detailed code to reproduce all tables, figures, and exhibits
  • information about the source dataset
  • instructions for extracting the dataset from the source data

Pre-check of the replication

Author Guidelines ReplicationIn addition, all provided replication materials will be verified by the journal to check if they actually reproduce the analysis. For that purpose, the editorial staff of the journal will work together with the University of North Carolina’s Odum Institute for Research in Social Science to conduct the verifications “as a regular component of the publication workflow.”

As soon as a manuscript is tentatively accepted for publication, the process of actual publication will only happen when the author provides all data, and the analysis is verified.

When problems arise…

If the replication fails, the author gets the cance to resolve the issue and improve the replication materials.

The journal decided to adopt this policy, which follows the APSA guidelines for research ethics, because they noticed that data upload and reproducibility varied greatliy between authors in the past. While the editors had asked authors to provide the data on the journal’s dataverse, it was voluntary.

What next?

This is a great step towards more reproducibility in the field, and hopefully other journals follow. Not every journal may have the resources to conduct pre-checks, but at least the strict policy of requiring all materials for the analysis and data collection is an important step any journal can take.

Next up would be to require authors to pre-register their whole analysis. At the moment, this is not common yet in political science, and only one journal has tried how this process could work for a special issue.

Ideally, the American Journal of Political Science should also check their previous submissions and provided replication data to ensure that already published work is reproducible (or at least all data are provided). International Studies Quarterley is trying something like that, but it is time consuming.

Advertisements

3 thoughts on “Journal introduces replication pre-check

  1. It is laudable that journals do a full check of the replication material. It is also a good idea to gather and check the material of past publications, but what happens when an author does not deliver the material or the results cannot be reproduced? Is the article retracted or flagged as non-reproducible? These are not reasons against doing this (though it could concern many publications), but one needs to have guidelines on how to deal with non-reproducible studies.

    Like

  2. […] Why wait to replicate? The American Journal of Political Science introduces pre-publication replication: You submit your paper with your data and code, and the journal reproduces your results before […]

    Like

  3. […] American Journal of Political Science recently announced that every article will be externally verified before it is published. Authors have to provide in […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: