Guest post by L.J Zigerell: Current practice in the social sciences places trust in researchers regarding their data collection, analysis, and reporting of results. That trust is sometimes unwarranted. Instead, we should increase trust in social science by encouraging tools of reproducibility: replication studies, pre-registration, third-party data collection, and open data.
Science is a method of learning about the world by testing claims with observations. But the results of scientific analyses can be communicated only through another way of learning about the world: testimony. We trust the testimony of researchers about their results. This testimony is sometimes flawed − because of, among other things, fraud, error, or selective reporting of results.
We should stop trusting researchers, and start using methods to trust data collection, analysis, and reporting of results.
In order to trust data, their analysis and results, replication and reproduction are crucial. The distinction between replication and reproduction can be summarized as:
- Replication of a scientific study is testing the same hypothesis with different observations: reporting a replication has the effect of adding testimony about the hypothesis of a replicated study.
- Reproduction of a scientific study is testing the same hypothesis with the same observations: reporting a reproduction has the effect of adding testimony about the reproduced study itself.
Both of these methods have value. Replication increases knowledge about what researchers should be most interested in: the presence of an effect, the direction of the effect, and the size of the effect. But knowledge about the presence, direction, and size of an effect is based on a formal or informal collective assessment of known studies regarding a hypothesis: reproduction is a method to assess the correctness or robustness of the studies that inform this collective assessment.
How to trust data instead of people
Concrete ways in which researchers, journal editors, and funding organizations can reduce trust in people (and replace it with trust in data and analysis) are:
- Pre-registration of research design protocols removes the trust that must be placed in a researcher’s explicit or implied testimony about whether model specifications and data analysis choices were planned before the outcome data were collected.
- Subcontracting data collection to an independent third party reduces the trust that must be placed in a researcher’s explicit or implied testimony regarding the method of data collection, such as stopping rules.
- Public posting of all collected data and the code necessary to reproduce research results reduces the trust that must be placed in a researcher’s explicit or implied testimony regarding the correctness, robustness, and representativeness of the reported analyses.
Note: This post reviews and extends thoughts first expressed in comments here.
About L.J Zigerell
L.J Zigerell is an assistant professor of politics and government at Illinois State University and received his Ph.D. from the University of Pittsburgh. L.J has researched Supreme Court nominations and public opinion, and his current research interests include racial politics and reproductions. You can follow L.J on Twitter at @LJZigerell.
Political Science Replication 
[…] Here. […]
LikeLike
I like the table and fully agree with the recommendations for enhancing trust. But I find it difficult to detach trust in the data, methods etc. from trust in the researcher because it is his/her data, methods etc. Furthermore, I read the post such that it implicitly raises the important question what the default attitude should be: should we have trust until we gather evidence undermining our trust, or should we have no trust and gradually become more trustful in the light of replications/reproductions? I am leaning toward trusting researchers until evidence to the contrary. The bar for “evidence” should not be that high; if it is not clear where the data comes from, for example, one should lower the level of trust.
LikeLiked by 1 person
Hi Ingo,
That’s a really good question, about the default level of trust that we should have. It might not matter *that* much, if we give thorough consideration to the specifics of a study.
I’d use the analogy of grading a student’s test: it does not matter whether the teacher grades with the presumption that the student starts with 100% and then loses points for each incorrect item, or whether the teacher grades with the presumption that the student starts with 0% and then gains points for each correct item; in both cases, the teacher should end up with the same grade for the student’s test.
It’s not a perfect analogy for grading research, because I’m not sure that readers of research “grade all items”, in the sense that a reader of research does not have a complete list of things for which to add or deduct points; therefore, it would make a difference if a reader of researcher started at 0% trust and added or 100% trust and subtracted. But the final level of trust should be in the same rough area, at least if the reader gives thorough consideration to the specifics of a study.
I think that, for evaluating the trust that can be placed in a specific study, I consider the three things mentioned in the post (preregistration, third-party data collection, and public posting of data and code), but I also consider things such as the reputation of the researcher and whether the research design has any unusual elements. For what it’s worth, I sometimes consider whether the results would or could have been published if the results were in a different direction or had a different strength.
LikeLike
[…] L. J. 2014. “Stop Trusting Other Researchers.” Political Science Replication Nov. 26th. Retrieved January 1, 2015 […]
LikeLike