Replication in political science: why implementation lags behind the ideal

Embedding reproducibility in political science teaching is challenging. How should it be done? Which courses integrate replication studies already? And can it do harm to let graduate students replicate published work?

Replication and reproducibility in university teaching have been discussed for many years in political science and other fields, such as psychology or economics. Early debates concentrated on the question if such practices could be useful or potentially harmful. In a symposium on replication in 1995, published by PS: Political Science & Politics (Vol. 28, No. 3), King (1995) stated that reproducing and extending existing research is an “extremely useful pedagogical tool, albeit one that political science students have been able to exploit only infrequently” because data were seldom available.

King therefore called for political scientists to meet a Replication Standard, which holds that “sufficient information exists with which to understand, evaluate, and build upon a prior work if a third party could replicate the results without any additional information from the author.”

To implement the standard as a pedagogical tool, he proposed new policies for graduate programs, such as requiring students to submit replication data for their Ph.D. dissertation, and having students replicate published articles if data sets in political science become more widely available.

Data vultures

Some scholars were skeptical of this proposal. Gibson (1995) cautioned that, following King’s call, generations of young scholars, would be raised to spend their time on mining existing data, instead of creating new, improved data sets. By engaging in the re-analysis of existing articles, they would become “data vultures”, focusing on small methodological errors in published work while ignoring the big picture.

Replication as a healthy trend

By 2003, the debate had shifted towards a wide agreement in the field, where scholars see reproducibility and replication as a “healthy trend” (de Mesquita 2003); it was now discussed how replications can be facilitated by journal data policies. In a `Symposium on Replication in International Studies Research‘, a survey showed that few international relations and political science journals published policies about data access on their webpages; even if some of them had a policy, the practice for replication standards varied, or showed little implementation or enforcement (Gleditsch 2003). For those engaging in replication, e.g. in university courses, this provided unnecessary obstacles because they could not even locate data before trying a re-analysis.

Still an urgent need for further guidance

Ten years later, a study of reproducibility and data transparency in political science still found that only 18 of 120 journals have a replication policy that requires authors to upload their datasets. This reflects that there is still an urgent need for further guidance on concrete steps to implement reproducibility and replication standards in the field.

While such implementation is still a work in progress, there is now at least widespread agreement among scholars on replication and reproducibility as crucial quality standards in political science. In a recent symposium in PS: Political Science Politics (Vol 47, Issue 1) the American Political Science Association published revised ethics guidelines emphasizing the need for full data access and research transparency. In one of the articles, Thomas M. Carsey stresses that the success of such guidelines depends, in part, on integrating principles of data access and transparency as “central elements” into graduate teaching, which is now a “growing trend”.

Where is the research gap?

This short overview shows that there is a need for more work on how to implement calls for better reproducibility in political science into practice. There is no systematic assessment of how university courses integrate replication at the moment, nor are there papers that describe how different types of courses and workshops could be set up. In addition, teachers might hesitate to assign replications to students due to typical challenges such as the lack of data access or potential frustration among students when the replication fails and original authors do not answer to emails. Therefore, we need research that examines the issue systematically, and provides examples for universities and teachers on how to integrate principles of data access and transparency into graduate teaching.

Discussing this at ISA in Toronto

At ISA’s 55th Annual Convention end of March in Toronto, the issue was discussed, and I will report about this in a separate post. At the panel discussion, there was some skepticism against replication studies in teaching, e.g. the danger that an army of students engages in a ‘witch hunt’ to bring down big name professors and publish failed replication studies; or the challenge that important tasks of cross-checking work in the field would be relegated to students instead of being done by senior scholars.

If you have thoughts about this, please comment below.

Advertisements
Tagged , , , , ,

3 thoughts on “Replication in political science: why implementation lags behind the ideal

  1. […] that the community agrees that data transparency is important, but still struggles to find ways to implement quality […]

    Like

  2. Hello, I presented on the panel and thought I would provide a link to the Prezi used: http://prezi.com/svhrblzca9oi/replicating-focus-group-research-design-winters-and-carvalho/

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: