Many social scientists agree that replication studies are necessary to provide quality standards in research. But how does a good replication study look like? Here is the checklist I will use in my Replication Workshop.
A replication is the process by which an article’s hypotheses and findings are re-analysed to confirm or challenge the findings. How exactly a replication study should be conducted, however, is still an “open question”, as political scientist Thomas M. Carsey has pointed out. Should we simply duplicate previous results, or go beyond that and conduct robustness checks? Here is a check-list for replicators.
A duplication verifies research results by attempting to produce the exact same results as the original author based on the exact same data and methods. This will uncover if the procedures described by the original author can produce the findings reported in the original article.
A challenge for duplication is that the data, codebook, readme file and software code need to be provided by the original author. While a duplication will probably not be publishable, it is a great tool to learn statistics, and to understand decisions each author faces in their analysis.
How to do a duplication:
|1. Download the replication data and conduct all variable transformations (e.g. logs) as specified in the original study.|
|2. Re-run all models as specified in the article; re-produce all figures and tables.|
|3. Compare your results with the original study. If the duplication was not successful, discuss possible errors in the data, variable transformation and coding procedures, or in the model specifications and software commands by the original author or by yourself.|
Gold Standard Replication
The goal of a replication study goes beyond duplication. It tests the robustness of the research results of the original study by employing newly collected data, and/or new variables, and/or new model specifications. An ideal ‘gold standard’ replication study would attempt to conduct most of these aspects, while ensuring that it is transparent and reproducible itself.
A replication study can take considerably longer than a duplication, because of the data collection and cleaning process. Also, a good replication study often adds knowledge that should be theoretically grounded, which takes a lot of literature research and extra work. On the upside, a replication study may be publishable.
How to do a full replication study:
|Full Replication Checklist|
|1. I would always start with a duplication because it clarifies what the original author did, and is often a first good check point about potential errors in the data, models or codings (that can later be improved.|
|2. Step two is to collect new data from the same and different sources. Data from the same sources can often be updated e.g. to cover the most recent years. Data from new sources can provide improved measurements or new variables such as potential confounders.|
|3. Re-run all models as specified in the original article but now using the new variables. Note how a change in the variables affects the results compared to the original study.|
|4. Then change the statistical models to fit the data and/or theory better (e.g. time lags, dealing with causality issues, handling missing data).|
|5. Discuss implications of the new data collection, new variable transformation, or new model specifications which might be responsible for the (very likely) change in results.|
|6. For a real gold standard replication, you should work transparently yourself: publish all data, software code for results tables and figures online. I would also consider pre-registration.|
- Brandt, Mark J. et al. (2014) The Replication Recipe: What makes for a convincing replication?, Journal of Experimental Social Psychology, Volume 50, January 2014: 217–224.
- Reporting Checklist For Life Sciences Articles
- King, Gary. (2006) Publication, publication. PS: Political Science and Politics, 39(1):119–125.
- Carsey, Thomas M. (2014) Making DA-RT a Reality. PS: Political Science and Politics, 47(1):72–77.