Is there a difference between replication, reproduction, and re-analysis?

In my replication workshop I’ve been discussing with grad students what replication actually is. Some students, from the field of Psychology, said that it involves re-doing data collection from scratch. In political science, I feel reproducing a paper is the same as replicating work: you take the data set, re-run the analysis, and potentially add to that. Here’s a selection of definitions.

Replication

“The replication standard holds that sufficient information exists with which to understand, evaluate, and build upon a prior work if a third party could replicate the results without any additional information from the author. (…) The process of reducing real-world phenomena to published work involves two phases: the representation of the real world by essentially descriptive quantitative and qualitative data, and the analysis of these data. Both phases are important components of the replication standard. Future scholars, with only your publication and other information you provide, ought to be able to start from the real world and arrive at the same substantive conclusions. In many types of research this is not possible, but it should always be attempted.” (King 1995, “Replication, Replication”)

Difference between replication & reproducibility

“A study is reproducible if there is a specific set of computational functions/analyses (usually specified in terms of code) that exactly reproduce all of the numbers in a published paper from raw data. It is now recognized that a critical component of the scientific process is that data analyses can be reproduced. (…) But just because a study is reproducible does not mean that it is replicable. Replicability is stronger than reproducibility. A study is only replicable if you perform the exact same experiment (at least) twice, collect data in the same way both times, perform the same data analysis, and arrive at the same conclusions. The difference with reproducibility is that to achieve replicability, you have to perform the experiment and collect the data again. This of course introduces all sorts of new potential sources of error in your experiment (new scientists, new materials, new lab, new thinking, different settings on the machines, etc.)”. (Simply Statistics 2012, “Replication, psychology, and big science”)

“Replication, using author-provided code and data, and independent reproduction work hand-in-hand. We can reserve the term “replicability” for the regeneration of published results from author-provided code and data. (…) Reproducibility is a more general term, implying both replication and the regeneration of findings with at least some independence from the code and/or data associated with the original publication. Both refer to the analysis that occurs after publication.” (Stodden 2011, “Trust Your Science? Open Your Data and Code”)

Terminology in my Replication Workshop

In the Cambridge Replication Workshop, we will talk about replication when we regenerate results from existing data and provided code. More specifically, we will regenerate tables, figures, and results as reported in the text of a paper. We follow Harvard Professor Gary King here, who encourages his students to “write a publishable article by beginning with the replication of a previously published article” (see King, “How to Write a Publishable Paper as a Class Project”)

Update: more definitions on replication and duplication

… are here and here.

About the Cambridge Replication Workshop

Screen Shot 2013-02-24 at 10.01.41 AMThe pilot project “Cambridge Replication Workshop” introduces grad students in the social sciences to the process of reproducing published work. It is based on Gary King‘s course at Harvard which includes replication, and I also got advice and examples for a syllabus from Victoria Stodden (Columbia). The workshop stretches over 8 weeks, and students have three TAs to help them with their replications in practical parts of our meetings. The project is funded by the Social Sciences Research Methods Centre. [ Syllabus as pdf ]

Advertisements
Tagged , , , , , , ,

4 thoughts on “Is there a difference between replication, reproduction, and re-analysis?

  1. […] original data, without collecting comparable data, the errors might not have been found. There is a controversy if replication means re-analysis or new data collection. The R-R case shows that you need […]

    Like

  2. This is quite an interesting blog. I would suggest that difference between reanalysis and replication is the use of the original data (cf. Herrnson, 1995). A reanalysis is based on the original data; if the same research question is addressed it’s a verification – else it’s a secondary analysis. A replication, on the other hand, is built on data that has been collected separately. (An overlap with the original data may occur.)

    Herrnson, P. S. (1995). Replication, Verification, Secondary Analysis, and Data Collection in Political Science. PS: Political Science and Politics. Vol. 28, No. 3 (Sep., 1995), pp. 452-455. doi:10.2307/420302

    Like

  3. […] Replicating the work of another is established practice in some disciplines as a method of learning; for example, an apprentice may seek to reproduce an art work of their master. Deceit only arises if at a later date the ownership of the reproduction is mistakenly attributed. The difference between replication (seeking to imitate) and reproduction (following the same process) is discussed here: https://politicalsciencereplication.wordpress.com/2013/02/24/is-there-a-difference-between-replicatio… […]

    Like

  4. […] Is there a difference between replication, reproduction, and re-analysis? This short piece from 2013 quotes experts and gives definitions of key terms in the replication debate. The fact that it was read by so many people reflects the ongoing problem that there is still no agreed definition of “replication” out there that is accepted by social scientists. Each field – or ideally we should take an interdisciplinary appraoch – should try to define the key terms so that we know what we are talking about. […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: