Does the “gold standard” of reproducibility hinder promising research? Do we unfairly damage the reputations of scientist by declaring a finding irreproducible? How should we react when we cannot reproduce the work? A piece in Nature gives provoking arguments against thoughtless replication.
In her article “Reproducibility: The risks of the replication drive” (Nature 503, 333–334, 21 November 2013), Mina Bissell from the Lawrence Berkeley National Laboratory argues that in biology many experiments take years, while those repliating the work usually don’t (and can’t) invest so much time. When they declare that they failed to reproduce the results, they do might more harm then good.
“If a researcher spends six months, say, trying to replicate such work and reports that it is irreproducible, that can deter other scientists from pursuing a promising line of research, jeopardize the original scientists’ chances of obtaining funding to continue it themselves, and potentially damage their reputations.”
Bissell explains that biologists today use massive data sets and complex structure models. Tiny changes in the experiment conditions can influence if the same result can be reproduced. “Newcomers” trying to replicate work are not always aware of this. Even if they work closely with the lab that produced the original results, they still might not be able to reproduce the results – and that’s not because the original research was faulty, careless or not valid.
The slightest shift in their microenvironment can alter the results — something a newcomer might not spot. It is common for even a seasoned scientist to struggle with cell lines and culture conditions, and unknowingly introduce changes that will make it seem that a study cannot be reproduced.
Bissell therefore suggests to be very careful when replicating experiments. One should always consult the original authors before trying to publish a ‘failed replication paper’. It might well be that not enough time was invested into the replication, or the experiment conditions were slightly different.
If e-mails and phone calls don’t solve the problems in replication, ask either to go to the original lab to reproduce the data together, or invite someone from their lab to come to yours. Of course replicators must pay for all this, but it is a small price in relation to the time one will save, or the suffering one might otherwise cause by declaring a finding irreproducible.
Let’s talk about the newcomers
- First of all, I’m not sure I like the word ‘newcomer’. Although this might be a misinterpretation, it sounds as if those trying to replicate work are the ‘juniors’ who are not quite sure of what they are doing, while the ‘seniors’ worked for years on a topic and deserve special protection against reputational damage.
- It goes without saying that anyone trying to replicate works should try to cooperate with the original authors. I agree. However, I would like to point that original authors don’t always show the willingness or capacity to invest time into helping someone else reproducing the results. As Bissell says herself, experiments can take years – and once the paper is published and someone decides to try to replicate it, the authors might already work on a new, time-intensive topic. My students in the replication workshop were sometimes frustrated when original authors were not available to help with the replication. So I’d say, let’s not just postulate that ‘newcomers’ should try to cooperate, but that original authors should make time to help as well to produce the best possible outcome when validating work.
- It is in the interest of original authors to clearly report the experimental conditions, so that others are not thrown off track due to tiny differences. This goes for replication based on re-analysing data as well as for experiments. The responsibility of paying attention to details lies not only with those trying to replicate work. From my experience and that of my students trying to replicat work in the social sciences, papers do not always list all steps that led to the findings. My students are often surprised at how authors came up with a variable, recoded it, which model specifications they used etc. Sometimes STATA or R code is incomplete. Therefore, while those replicating work should try to be aware of such details, original authors need to work on this as well. Original research might take years, but it really should not take years to replicate them, just because not all information was given.
- Bissell states that replicators should bear the costs of visiting research labs and cooperating with the original authors (“Of course replicators must pay for all this, but it is a small price in relation to the time one will save, or the suffering one might otherwise cause by declaring a finding irreproducible.”). I’m not quite sure of this. As Bissell points out herself at the beginning of her article, it is often students who replicate work. Will their lab pay for expensive research trips? Will they get travel grants for a replication? And for the social sciences that often works without a lab structure, who will pay for the replication costs? I feel the statement that a replicator must bear all costs – even though original authors profit from cooperation as well, can be off-putting for many students engaging in replication.
On December 17, 2013 Andrew Gelman also reacted to Mina Bissell’s post on his blog. There’s an interesting discussion going on below his post.