Guest post by L.J Zigerell: Current practice in the social sciences places trust in researchers regarding their data collection, analysis, and reporting of results. That trust is sometimes unwarranted. Instead, we should increase trust in social science by encouraging tools of reproducibility: replication studies, pre-registration, third-party data collection, and open data.
Science is a method of learning about the world by testing claims with observations. But the results of scientific analyses can be communicated only through another way of learning about the world: testimony. We trust the testimony of researchers about their results. This testimony is sometimes flawed − because of, among other things, fraud, error, or selective reporting of results.
We should stop trusting researchers, and start using methods to trust data collection, analysis, and reporting of results.
In order to trust data, their analysis and results, replication and reproduction are crucial. The distinction between replication and reproduction can be summarized as:
- Replication of a scientific study is testing the same hypothesis with different observations: reporting a replication has the effect of adding testimony about the hypothesis of a replicated study.
- Reproduction of a scientific study is testing the same hypothesis with the same observations: reporting a reproduction has the effect of adding testimony about the reproduced study itself.
Both of these methods have value. Replication increases knowledge about what researchers should be most interested in: the presence of an effect, the direction of the effect, and the size of the effect. But knowledge about the presence, direction, and size of an effect is based on a formal or informal collective assessment of known studies regarding a hypothesis: reproduction is a method to assess the correctness or robustness of the studies that inform this collective assessment.
How to trust data instead of people
Concrete ways in which researchers, journal editors, and funding organizations can reduce trust in people (and replace it with trust in data and analysis) are:
- Pre-registration of research design protocols removes the trust that must be placed in a researcher’s explicit or implied testimony about whether model specifications and data analysis choices were planned before the outcome data were collected.
- Subcontracting data collection to an independent third party reduces the trust that must be placed in a researcher’s explicit or implied testimony regarding the method of data collection, such as stopping rules.
- Public posting of all collected data and the code necessary to reproduce research results reduces the trust that must be placed in a researcher’s explicit or implied testimony regarding the correctness, robustness, and representativeness of the reported analyses.
Note: This post reviews and extends thoughts first expressed in comments here.
About L.J Zigerell
L.J Zigerell is an assistant professor of politics and government at Illinois State University and received his Ph.D. from the University of Pittsburgh. L.J has researched Supreme Court nominations and public opinion, and his current research interests include racial politics and reproductions. You can follow L.J on Twitter at @LJZigerell.