Many political journals have no replication policy, or only implement it half-heartedly. Now a leading journal introduces innovative guidelines for authors that include a verification of the analysis before publication.
The American Journal of Political Science has criticized that the discipline has “paid lip service” to reproducibility in the past. In order to “practice what we preach”, the journal now uses a replication policy adhering to the highest standards in research transparency.
How to prepare replication files
The “Guidelines for Preparing Replication Files” [pdf] state that replication data must contain:
- detailed code to reproduce all tables, figures, and exhibits
- information about the source dataset
- instructions for extracting the dataset from the source data
Pre-check of the replication
In addition, all provided replication materials will be verified by the journal to check if they actually reproduce the analysis. For that purpose, the editorial staff of the journal will work together with the University of North Carolina’s Odum Institute for Research in Social Science to conduct the verifications “as a regular component of the publication workflow.”
As soon as a manuscript is tentatively accepted for publication, the process of actual publication will only happen when the author provides all data, and the analysis is verified.
When problems arise…
If the replication fails, the author gets the cance to resolve the issue and improve the replication materials.
The journal decided to adopt this policy, which follows the APSA guidelines for research ethics, because they noticed that data upload and reproducibility varied greatliy between authors in the past. While the editors had asked authors to provide the data on the journal’s dataverse, it was voluntary.
This is a great step towards more reproducibility in the field, and hopefully other journals follow. Not every journal may have the resources to conduct pre-checks, but at least the strict policy of requiring all materials for the analysis and data collection is an important step any journal can take.
Next up would be to require authors to pre-register their whole analysis. At the moment, this is not common yet in political science, and only one journal has tried how this process could work for a special issue.
Ideally, the American Journal of Political Science should also check their previous submissions and provided replication data to ensure that already published work is reproducible (or at least all data are provided). International Studies Quarterley is trying something like that, but it is time consuming.