Checklist for a Gold Standard Replication

Screen Shot 2014-12-14 at 15.37.50Many social scientists agree that replication studies are necessary to provide quality standards in research. But how does a good replication study look like? Here is the checklist I will use in my Replication Workshop.

A replication is the process by which an article’s hypotheses and findings are re-analysed to confirm or challenge the findings. How exactly a replication study should be conducted, however, is still an “open question”, as political scientist Thomas M. Carsey has pointed out. Should we simply duplicate previous results, or go beyond that and conduct robustness checks? Here is a check-list for replicators.

Duplication

A duplication verifies research results by attempting to produce the exact same results as the original author based on the exact same data and methods. This will uncover if the procedures described by the original author can produce the findings reported in the original article.

A challenge for duplication is that the data, codebook, readme file and software code need to be provided by the original author. While a duplication will probably not be publishable, it is a great tool to learn statistics, and to understand decisions each author faces in their analysis.

How to do a duplication:

Duplication Checklist
1. Download the replication data and conduct all variable transformations (e.g. logs) as specified in the original study.
2. Re-run all models as specified in the article; re-produce all figures and tables.
3. Compare your results with the original study. If the duplication was not successful, discuss possible errors in the data, variable transformation and coding procedures, or in the model specifications and software commands by the original author or by yourself.

Gold Standard Replication

The goal of a replication study goes beyond duplication. It tests the robustness of the research results of the original study by employing newly collected data, and/or new variables, and/or new model specifications. An ideal ‘gold standard’ replication study would attempt to conduct most of these aspects, while ensuring that it is transparent and reproducible itself.

A replication study can take considerably longer than a duplication, because of the data collection and cleaning process. Also, a good replication study often adds knowledge that should be theoretically grounded, which takes a lot of literature research and extra work. On the upside, a replication study may be publishable.

How to do a full replication study:

Full Replication Checklist
1. I would always start with a duplication because it clarifies what the original author did, and is often a first good check point about potential errors in the data, models or codings (that can later be improved.
2. Step two is to collect new data from the same and different sources. Data from the same sources can often be updated e.g. to cover the most recent years. Data from new sources can provide improved measurements or new variables such as potential confounders.
3. Re-run all models as specified in the original article but now using the new variables. Note how a change in the variables affects the results compared to the original study.
4. Then change the statistical models to fit the data and/or theory better (e.g. time lags, dealing with causality issues, handling missing data).
5. Discuss implications of the new data collection, new variable transformation, or new model specifications which might be responsible for the (very likely) change in results.
6. For a real gold standard replication, you should work transparently yourself: publish all data, software code for results tables and figures online. I would also consider pre-registration.

Literature

Advertisements
Tagged , , , ,

One thought on “Checklist for a Gold Standard Replication

  1. […] Reacting to criticism of replicaiton, in 2014 more and more ideas were discussed on how to do replications ‘right’, for example in The Scientist, in this paper, at the World Bank blog on Development, and on the Political Science Replication blog. […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: