Is there a difference between replication, reproduction, and re-analysis?

In my replication workshop I’ve been discussing with grad students what replication actually is. Some students, from the field of Psychology, said that it involves re-doing data collection from scratch. In political science, I feel reproducing a paper is the same as replicating work: you take the data set, re-run the analysis, and potentially add to that. Here’s a selection of definitions.

Replication

“The replication standard holds that sufficient information exists with which to understand, evaluate, and build upon a prior work if a third party could replicate the results without any additional information from the author. (…) The process of reducing real-world phenomena to published work involves two phases: the representation of the real world by essentially descriptive quantitative and qualitative data, and the analysis of these data. Both phases are important components of the replication standard. Future scholars, with only your publication and other information you provide, ought to be able to start from the real world and arrive at the same substantive conclusions. In many types of research this is not possible, but it should always be attempted.” (King 1995, “Replication, Replication”)

Difference between replication & reproducibility

“A study is reproducible if there is a specific set of computational functions/analyses (usually specified in terms of code) that exactly reproduce all of the numbers in a published paper from raw data. It is now recognized that a critical component of the scientific process is that data analyses can be reproduced. (…) But just because a study is reproducible does not mean that it is replicable. Replicability is stronger than reproducibility. A study is only replicable if you perform the exact same experiment (at least) twice, collect data in the same way both times, perform the same data analysis, and arrive at the same conclusions. The difference with reproducibility is that to achieve replicability, you have to perform the experiment and collect the data again. This of course introduces all sorts of new potential sources of error in your experiment (new scientists, new materials, new lab, new thinking, different settings on the machines, etc.)”. (Simply Statistics 2012, “Replication, psychology, and big science”)

“Replication, using author-provided code and data, and independent reproduction work hand-in-hand. We can reserve the term “replicability” for the regeneration of published results from author-provided code and data. (…) Reproducibility is a more general term, implying both replication and the regeneration of findings with at least some independence from the code and/or data associated with the original publication. Both refer to the analysis that occurs after publication.” (Stodden 2011, “Trust Your Science? Open Your Data and Code”)

Terminology in my Replication Workshop

In the Cambridge Replication Workshop, we will talk about replication when we regenerate results from existing data and provided code. More specifically, we will regenerate tables, figures, and results as reported in the text of a paper. We follow Harvard Professor Gary King here, who encourages his students to “write a publishable article by beginning with the replication of a previously published article” (see King, “How to Write a Publishable Paper as a Class Project”)

Update: more definitions on replication and duplication

… are here and here.

About the Cambridge Replication Workshop

The pilot project “Cambridge Replication Workshop” introduces grad students in the social sciences to the process of reproducing published work. It is based on Gary King‘s course at Harvard which includes replication, and I also got advice and examples for a syllabus from Victoria Stodden (Columbia). The workshop stretches over 8 weeks, and students have three TAs to help them with their replications in practical parts of our meetings. The project is funded by the Social Sciences Research Methods Centre. [ Syllabus as pdf ]