Reproducibility Pioneers: Political Scientists Assigning Replications to Students

Here are innovative and exciting syllabi of methods courses assigning replication studies to students. The instructors ensure that we build up a cohort of students who value reproducibility, while learning statistics at the same time.

For the ISA 2014 double panel on reproducibility and replication I had collected syllabi that require students to replicate published work. If you know more courses assigning the task of replication, please tweet [@polscireplicate], email or comment below.

“Advanced Quantitative Political Methodology: Government 2001“, taught by Gary King, Harvard University

According to the syllabus, the main assignment of the course is to submit a research paper “that replicates an existing piece of scholarship.” The students team up in small groups and conduct a replication study, with the “aim to produce a publishable article, and, in fact, most students do publish their final paper in a scholarly journal.” In order to encourage students to follow a reproducible workflow, students are required to hand over all data and information necessary to replicate their own results to another student team, which will then conduct a cross-check and give detailed feedback before the final submission of the class project. This feedback among students is part of each student’s evaluation.
(See also King’s papers on teaching replication: King 1995,King 2006)

PCS 531: Intermediate Statistics for the Social Sciences, taught by Carlisle Rainey, University at Buffalo

From the syllabus:
The most important thing we do in this class is a replication paper. Every assignment is designed to directly or indirectly improve your paper. I consider a wide range of ideas about what your paper will look like. I feel strongly, however, that your paper should be co-authored (with one or two other students), except in unusual circumstances. I expect this paper to be high-quality (think about it being ready for a conference at the very worst). Near the end of the semester, you will turn in to me the following:

1. A high-quality manuscript. It should a contribution to a political science literature. This paper should make a contribution to a political science literature, look fantastic, be well written, and have few (ideally no) typos. It should be shorter, rather than longer (think 10-15 pages; Bell and Miller is a good example). If I do not feel comfortable suggesting that you submit the paper to a conference, then you will not get a very good grade. However, I have structured the course so that you should do a good job if you put in the time.

2. A replication data set. Near the end of the semester, I will assign another student the task of replicating your tables and figures starting with the raw data set. If they cannot, you will not get a very good grade.

3. A conference-style presentation. It is part of my job to prepare you for a conference–you are training to be political scientists. Toward the end of the semester, I will give you the opportunity to make a conference-style presentation. I will fill in the details later, but for now, you should be thinking ten minutes, making every minute count.

PLS 501: Methods of Political Analysis (Research Design), taught by Christopher Fariss, Penn State University

From the syllabus:
Group Replication Project: In groups of 2-4 students, obtain the materials necessary to replicate a political science research paper published in the last 5 years. Describe the initial study and the ease with which the results replicate. Then identify any design flaws in the research and propose a new or improved design. Again, the write-up should be no more than 5-pages.

Political Science 582: Quantitative Analysis in Political Science II, Fall 2013, taught by Jeff Gill, Washington University in St Louis

From the syllabus:
Course Grade: The final grade will be based on three components: problem sets (40\%), a replication assignment (30\%), and an exam (30\%) on MLE theory and basic models. The exam covers material from the first 7 weeks of the course plus the assigned readings (Faraway and articles). Consequently, we will discuss the readings in as much detail as the class desires. The problem sets will be a combination of analytical and computational assignments and given in each meeting. See Alicia Uribe’s tips on success with the problem sets. For the replication assignment, find a published work in your field of interest, obtain the data, and exactly replicate the author’s model results. It is usually easier to find an article that uses the readily available datasets in the discipline (COW, ANES, GSS, etc.), but some authors are forthcoming about distributing their data if asked. The relevant model should be one of the nonlinear forms studied in this course.

POLI 784 – Intermediate Statistics, Spring 2014, taught by Thomas M. Carsey, University of North Carolina at Chapel Hill

From the syllabus:
For this class, you should pursue a paper that is a replication and extension of an existing published paper (or book chapter). This will make it easier for you to present the literature review and theory sections for your paper since they will be closely tied to the paper you replicate. (…) There is no formal page length, but for most of you I expect the paper will constitute 14-20 pages of text. You should model your paper after the quantitative papers you have seen in journals like APSR, AJPS or JOP, with the caveat that the front-end of your paper (everything up to the Data/Methods section) will be shorter than the typical journal article (because you are doing a replication), and that you will be asked to provide a bit more detail in the back half of your paper regarding the analyses, tests, etc. that you performed.

By “replication,” I mean that your first task will be to reproduce the findings exactly as shown in the published paper or chapter you are looking at. This DOES NOT mean simply contacting the author or using a so-called replication data set in which all of the coding, modeling, and estimation decisions/commands are already done for you. Rather, it means going back to the primary (electronic) data source if possible and proceeding from there. (…)

By “extension,” I mean that once you have replicated the results of an existing study, you will then build upon that analysis in some way. This might involve using a different coding of a variable, adding additional variables, considering different (maybe non-linear) model specifications, or adding additional data. Whatever extension you attempt, however, must be derived from a clear theoretical proposition and/or a clear methodological critique. (…) All students will read drafts of two other student papers in the course and provide written comments to them.
(See also Carsey’s paper on reproducibility: Carsey 2014)

Your help

If you know more courses assigning the task of replication, please tweet [@polscireplicate], email or comment below.