I recently wrote about a student who was awarded a Masters degree for her replication study at the Norwegian University of Science and Technology. Political scientist Jonathon W. Moses, who supervised the thesis, explains why more students should dare to replicate articles for a degree.
How did you develop an idea about the value of replication in political science?
“As a doctoral student at UCLA, I studied statistics (mostly TSCS) extensively, and became well aware of the sensitivity of the results in many models. This experience, and in subsequent work with other statisticians, I came to realize the utility—nay, necessity—of replicating accepted results to check their robustness. Then, in 2004 a colleague and I tried to replicate an influential study on the effects of international migration on global economic activity (with great difficulty!). You can find the article here. In the process I learned how difficult it is to replicate when data are not published, and when the process of research is not adequately described in the published work. This experience led me to encourage students to conduct replication studies in their MA work. It is a wonderful way to become familiar with the techniques involved, and to learn of the strengths and shortcomings of the existing work.”
Do many of your students do a replication for their thesis?
“Unfortunately, most students shy away from this, as they don’t yet see the utility of replication, and want to strike out on their own.”
How valuable do you think is replication compared to ‘original research’?
“I think replication is a very valuable means for checking the robustness of existing research/results. As such, it is equally valuable to original research, as it allows us to be more certain about the findings upon which our discipline builds. The problem is that students (and journals) have yet to appreciate the important role that replication can and should play. These attitudes are beginning to change (we see it in the number of journals who are now publishing databases along with their articles), but it is taking time.”
What are the advantages of doing a replication for a degree?
“Replication is the easiest way to jump to the research frontier. In replicating an established argument, students learn by doing—they employ sophisticated models, get hands-on experience with datasets, and learn of the shortcuts and miss-representations that are often taken to secure publication results. The Øien thesis is a good example of this: by changing the time periods of a study, she was able to change the results, significantly. This is very important information that wouldn’t (didn’t) come out in the original published piece.”
Why should researchers in social and political sciences do replications?
“In short: our knowledge can be made more secure. As a profession, we would do good to remember Descartes’ First Meditation, where he describes the necessity of protecting against faulty beliefs. Descartes argues that we must first cleanse our mind of all former beliefs, because many of these are bound to be false. To do this, he uses an analogy of a basket of apples (the apples being representative of his beliefs). Descartes argues that he who is worried about rotten apples in a barrel will be well advised to tip out all the apples and then replace each one carefully, inspecting every single apple for damage and rot. Only when he is certain that an apple is sound should he put it back in the barrel. If he makes a single mistake, the entire barrel may be spoiled. Descartes’ point is that all claims should be treated as if they were false. We should only add a claim to our stock of knowledge if we are certain that it is true; if we are in the slightest doubt about a claim’s veracity, we should reject it (see “Ways of Knowing: Method and Methodology in Social Science“: chapter 2 for the background story). Replication helps us sort out the bad apples.”
What is your professional attitude about reproducibility?
“One of the defining features of scientific work is that it is public undertaking. The whole point of publishing and referencing our work is to allow subsequent researchers to follow in our footsteps, and replicate our results. It is our responsibility, as social scientists, to describe the research process in such a way as to facilitate that replication. This is true whether we are conducting small N or large N studies. Our findings are not reliable if a subsequent scholar cannot follow the same exact steps and arrive at the same exact outcome.”
Wouldn’t some researchers be worried that replication uncovers mistakes?
“It should be pointed out that we, as scholars, often make mistakes. The possibility of mistakes increases with the complexity of our models, datasets and arguments. Replication is not so much about smoking out dishonest scholars, as it is about ensuring that small mistakes are corrected, so that the knowledge we embrace is more certain and robust.”
Since 2004, Jonathon W. Moses is professor at the Norwegian University of Science and Technology (NTNU), Professor. He earned his PhD from the University of California in Los Angeles (UCLA). His research interests concern the impact of global economic integration on political sovereignty at various levels. In addition, he is a co-editor of European Political Science. He recently published “Ways of Knowing: Method and Methodology in Social Science” (co-authored with Torbjørn Knutsen; Palgrave, 2012). Contact: jonathon.moses (at) svt.ntnu.no [full CV].
The interview was conducted via email on January 7, 2013.