The largest International Relations conference, ISA’s 55th Annual Convention (March 26-29, 2014), features two panels on replication and reproducibility this year. The first panel covers “Replication in International Relations: How Journal Data Policies and Replication in Teaching Can Improve Reproducibility Standards.”
In 2002, a panel on replication was held at the ISA convention, leading to a symposium in International Studies Perspectives in 2003, which included a survey of journal practices and a joint pledge by four leading IR journals to make the posting of data mandatory. Ten years on, International Relations journals still vary considerably in their commitment to replication and quality of replication data. At the same time, replication archive practices have become more sophisticated, and official bodies are strengthening requirements for transparency in publicly funded research.
We also have many more published examples of attempts to replicate well-cited studies – some successful, others failures. However, as far as we know, there has not been a major re-examination at the ISA convention of replication standards. In this first of two linked panels [Panel 2], we examine two main pillars of reproducibility in the field:
We provide a new survey of the replication requirements of international relations journals and discuss the use of replication material in teaching of international relations. How strict are journal policies and how can they be improved? How can we motivate authors, and what role can teaching at graduate schools play in creating a new cohort of doctoral students who engage in replication?
Michael P. Colaresi (Michigan State University): Improving Standards for Replication v2.0: Reducing Obstruction, Introducing Preplication and Increasing Incentives
Despite great strides, replication still entails significant costs and standards are unevenly practiced across international relations. Here I make a three pronged argument to improve this situation: (1) We need to throw light upon informal practices that get in the way of replication. These include several strategies that scholars use to try and ward off replications and keep them from being published, as well as informal threats and reprisals targeted particularly at junior scholars. (2) We should explore the feasibility of validating work before it appears in print: preplication. This could be akin to copy editing, but for the data analysis and not the prose. Checks for a publicly available list of common mistakes in coding, merging, analyzing and interpreting data should be undertaken first by the author, and then by the editorial team. (3) Journals, and particularly journal editors, should see the benefits of facilitating, publishing and promoting replications. I present a simple game-theoretic model that shows that cheaters are more likely to target journals with weaker replication standards than strong ones, as well as journals with better reputations. Further, I present a research design to test whether journals with weaker replications standards tend to suffer greater publication bias.
Allan Dafoe (Yale University): Science Deserves Better: The Imperative to Share Complete Replication Files
Scientific knowledge is only as reliable as the empirical analysis on which it is based. For the majority of published statistical analyses, readers have to trust that the scholars correctly implemented the many stages of analysis between primary data collection and the presentation of results–including data cleaning, merging, recoding and transforming, analysis, and output. Horror stories abound about non-reproducible results. Such infelicities could be exposed and deterred by requiring all stages of statistical analysis to be executed in code, which is then made publicly accessible. Other benefits of replication transparency include: raising the quality of analysis; dissemination of useful code and data; greater freedom for scientists to explore each others’ results and data, and a richer scientific conversation. Stronger replication norms are a scientific public good: the costs are small but born by the authors; the benefits are great and are shared by the broader scientific community and public. Strong norms of replication transparency, once established, are likely to be self-enforcing. Specific recommendations about replication standards for scholars and journals are proposed. I present data on actual replication practices, replicability of published works, and the level of support for stronger replication practices.
John Ishiyama (University of North Texas): Replication Studies and Journal Publication in Political Science and International Relations
This presentation will focus on the challenges facing existing general political science journals on the issue of the publication of replication studies for both quantitative and qualitative research. Although many journals in political science and international relations now require making available replication data for public display, few journals provide space for the publication of replication studies. While journals in the natural sciences have long had a tradition of publishing replication studies, this has not been the tradition in most political science or international relations journals. The presentation will discuss the variety of efforts that have been undertaken by journal editors to provide opportunities for the publication of replication studies in the discipline as well as options for the future. In particular, the presentation will also discuss some recent efforts by APSA to address the issue of the provision of a venue for the publication of replication studies.
Nicole Janz (University of Cambridge): Bringing the Gold Standard Into the Class Room: Replication in University Teaching
The ability to reproduce results is held as the gold standard for scientific research. However, motivation to replicate work is low among scholars in international relations. Replications are difficult to conduct, time consuming, and seldom rewarding in terms of publication. This paper argues that a solution to a lack of replication studies lies in teaching. Universities should invest in new stand-alone replication workshops or replications as class assignments in methods training. The paper will first discuss benefits of replication for students. Drawing from the authors’ own experience of instructing such a workshop, as well as examples from other courses, the main part of the paper will describe how to integrate replication in the classroom: selecting a paper, obtaining data, re-creating work, adding value to the re-analysis, publishing the data in a repository, and journal submission. Particular emphasis will lie on pitfalls and challenges in the process.
The replication panels
Friday, March 28
8:15 AM – 10:00 AM
FA10: Replication in International Relations: How Journal Data Policies and Replication in Teaching Can Improve Reproducibility Standards
Willow Centre, Sheraton Centre Toronto
Immediately followed by:
10:30 AM – 12:15 PM
FB10: Replication in International Relations: Successes and Failures in Practice
Read the full conference program
ISA’s 55th Annual Convention
Spaces and Places – Geopolitics in an Era of Globalization
March 26th – 29th, 2014, Toronto, Canada