Science deserves better

In a new paper, Allan Dafoe (Yale University), examines the availability of replication data in top political science journals – and comes to mixed conclusions.

In a note to this blog, also published here, political scientist Dafoe says that for most published articles in political science we simply have to “trust” the authors’ statistical analyses. Why? Replication files are often not available for download.

“The reality is that it is not uncommon for the key results of scientific research to be non-reproducible or to arise from errors.”

Replication in top journals

In his paper “Science Deserves Better: The Imperative to Share Complete Replication Files”, which will be part part of a special issue in PS: Political Science & Politics on Data Access and Research Transparency, Dafoe collected data on the state of replication in political science:

1. On the availability of replication files for publications in two leading journals (APSR and AJPS): Nearly half of the articles employing statistical analysis promise on the first page that the data are available. For 68% Dafoe actually found the data.
2. On scholars’ attempts to replicate publications: 52% of the scholars asked reported that they were able to precisely reproduce the main results. 20% said that they found major technical errors.

Dafoe concludes: “This is encouraging in how many results were found to be robust, while also reinforcing the value of strong transparency norms so that the many fragile results can be more easily uncovered and examined.”

Among other journals the state of replication is more problematic. An article in “European Political Science” found that out of 120 political science journals only 18 have replication policies requiring data sharing.

How to improve reproducibility

Therefore, Dafoe outlines recommendations for more reprodicbility in the field, such as:

• For authors: Do all data preparation and analysis in code. Build all analysis from primary data files. Fully describe your variables. Document every empirical claim. Archive your files.
• For journals: Require complete replication files before acceptance. Encourage high standards for replication files. Implement a Replication Audit to assess the replicability and robustness of a random subset of publications. Retract publications with non-replicable analyses.

 

Read more in Dafoe’s Paper

Abstract:
Scientific knowledge is only as reliable as the empirical analysis on which it is based. For the majority of published statistical analyses, readers have to trust that the scholars correctly implemented the many stages of analysis between primary data collection and the presentation of results—including data cleaning, merging, recoding and transforming, analysis, and output. I advocate the adoption of a simple transparency maxim: good research involves publishing complete replication files, making every step of research as explicit and reproducible as is practical… more

 

Listen to Dafoe’s talk at ISA 2014

Allan Dafoe’s paper will be part of a double panel on replication at the International Studies Association’s Annual Convention in Toronto, which is organized by me and Nils Petter Gleditsch. The replication double panel takes place Friday, March 28, from 8.15am-12.30pm.

 

Read more in Dafoe’s Yale Blog Post