Guest Post: Six Crucial Steps to Strengthen a Replication, by Mallory Kidwell

Scientific research advances knowledge by making transparent the methodology and evidence supporting scientific claims. This way, others can evaluate the basis of evidence and repeat the research in order to confirm the conclusions. However, the current incentive structures emphasizing publication can cause a disconnect between the valued ideals and daily practices of individual researchers.

Mallory Kidwell ReplicationThe social sciences have begun a collective shift to better align these values and practices, and thereby strengthen the integrity of the research literature.

Where to start? Replication provides a means to confirm the presence and size of an effect. After all, how better to reinforce the evidence for a finding than to reproduce its original methodology? While arguably the best approach to substantiating research claims, replication is often more complicated than it seems. And, ineffective replications may increase uncertainty rather than reduce it.

There is always potential for error, but there are straightforward ways to maximize the quality and informativeness of a replication. As a project coordinator of a large-scale, community-based replication effort, the Reproducibility Project: Psychology, I have benefitted from guiding our replication teams through this standardized procedure for conducting good replications.

  1. Read, read, read the original paper. That’s no typo: read the paper you intend to replicate several times before you even begin to devise your replication protocol. Many publications have a word limit, forcing authors to pick and choose which pieces of information to include and exclude. Make a note of information you believe may be missing or any questions you’re left with after you’ve finished reading.
  2. Contact the original authors – before you start. Contact the original author(s) to request the original materials, clarify questions about the protocol, and see if they have any suggestions or insider tips regarding your replication attempt. Using the same materials as the original experiment can minimize sources of error, and often the original authors will draw your attention to important nuances in the protocol that you might have overlooked.
  3. Write a pre-data collection report. The pre-data collection report will be your guide as you start to put together your replication protocol. Include an introduction, power analysis, intended planned sample, procedure, analysis plan, and any known differences from the original protocol. It should include all procedural details so that someone else could evaluate and replicate your own replication.
  4. Share the pre-data collection report with the original authors. Ask for the original authors’ feedback and critique of your replication protocol. Incorporate their suggestions and note explicitly in the protocol when you make decisions that are different than the original authors’ recommendations. That way, it is clear that differences were noted in advance with explanation for why you did something different.
  5. Pre-register your research plan. When you’ve finalized your pre-data collection report, pre-register your plan. This way, you can be certain of the protocol you intended to follow beforehand and share it with others before and after the replication is complete for comparison.
  6. Follow through – in an open way. The strongest evidence is that which can be examined by others. Share your materials, data, analyses, and pre- and post-data collection reports so that others can review, critique, cite, and use your research. This can easily be achieved with the Open Science Framework, a free, open-source platform for registration, archiving, and project management that is maintained by the Center for Open Science.

About Mallory Kidwell

Mallory Kidwell is a Metascience Project Coordinator at the Center for Open Science and received her B.A. in Psychology and English from Wake Forest University. Mallory has served as an assistant and principal investigator to research on anxiety and posttraumatic growth, and currently coordinates the Reproducibility Project: Psychology. Find Mallory on Twitter at @kidwellmc.

Tagged ,

One thought on “Guest Post: Six Crucial Steps to Strengthen a Replication, by Mallory Kidwell

  1. Squeak Squeak says:

    The pre-registration part seems a bit unnecessary if the data collection and analysis are all exactly following the original paper, right? Sure, it would be nice to record every detail in advance, and to do so in even greater detail than the original investigators did. But as you say in your first paragraph, there is a problem of inadequate incentives to do replication. Won’t piling on additional burdens to those relatively few investigators doing replications exaccerbate that problem?

    Also, shouldn’t you have mentioned other registration vehicles besides OSF? You may have a bit of a transparency/ conflict of interest issue here. OSF, for which you work, benefits when people use OSF (sure, it’s free and nonprofit, but it wants usage to grow, right?). Those advocating making science more squeaky clean should themselves embody squeaky-cleanness, no?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: