In the second part of our interview, political scientist Todd Landman (University of Essex) talks about how replication can be used in statistics training, how he administers his own data, why he is skeptical of replication as a dissertation, and how data sharing can work for qualitative researchers. (Read part I of the interview here)
How can replication and reproduction of published work be used in political science methods teaching?
Yes, we did that. I did it as a student in Colorado, where we replicated “Inequality and Insurgency” by Muller and Seligson in the American Political Science Review. There was a typo in one of the tables and the challenge for the students was to find the typo. That was a great exercise. We had to do the whole thing from scratch. We hated it when we did it but I’m really glad that I did it.
Do you let students download replication data and work with them?
At the Essex Summer School I use data sets for the students. I’m not giving statistics courses, but if I were, replication would be part of the curriculum. As a pedagogical tool I think it’s great.
When asking authors for their data, I often hear ‘I need to find the files’, or ‘I need to clean up the files’ first. How well do you think political scientists keep track of their own work, and how do you keep your files?
I’m publishing now for 19 years and I have two external hard drives and a bunch of memory sticks and haven’t actually sat down to consolidate all my data sets into one directory. Having said that, I know where all my data sets are, so that when a request comes in I can usually find it and send it off. It’s usually ordered by book or publication. I made an attempt a few years ago to organize everything but then ran out of time. The ones that are in published articles are in good shape. I have the codebooks and datasets to send out.
I’m going to start putting data sets on my webpage, on a data page. Then you can say here are all the data sets for my published articles.
I have recently asked an author for their data, and received the answer: ‘We can’t make our data available because of our contractual agreement with the data source.’ In which cases do you think it is acceptable to use data that cannot be shared?
Some data you have to pay for, and you can’t share it. Once I share it, and it’s in the public domain, then the source loses its market advantage. When we built the Human Rights Atlas data set, we consciously made a decision not to include any paid data, so that everything was publicly available. This is all stuff you can find in open sources. Any data that are created with public funds should be shared.
We’ve talked mostly about statistical data so far. You have also done qualitative case study work. How can we replicate such studies, if at all?
The problem is that if you get into the qualitative field like text analysis, search terms, semantics, narrative analysis, interpretivism – it’s down to that you have to trust the researcher that they are interpreting things. I have no idea if what they’re telling me is plausible or not. I’m not even sure about representative sampling in discourse; there could be sampling bias in everything they’re doing. I challenge many of our PhDs who do that kind of work on the case selection question. Among all of X’s speeches do you select speeches where they mentioned, for example, ‘immigration’, do you use all speeches, or were specific texts selected beforehand?
For qualitative work, do you think there is a way to make at least some of their data available, e.g. their documents, atlas.ti files?
There is, but the volume of information is gigantic, so there are trade off’s there. A lot of qualitative researchers will make interview transcripts available, others say they can’t share transcripts because it’s impossible to anonymize it in a form that allows you to look at it. The only way they got the information is because they promised anonymity, so there are issues about research ethics that come into play. The quantitative researchers usually have anonymous surveys or aggregate country data. Atlas.ti files of text codings would be great to see. (…) The ethos is important. When possible, one should strive to share data files as best as possible. (…) Be as public as you possible can, even if you’re qualitative. Be public about your case selection, be public about who you interview, even if you anonymize it. You can still say ‘I spoke to 15 village elders’, not ‘some village elders’.
I recently described how a political scientist was awarded a masters degree for a replication study. This still seems to be an exception because students might think it is not original research. What is your take on this?
I think there’s some problems with that. At the MA level it’s impressive that a student can learn enough in a short period of time to be able to replicate and then add something. If they found new things by doing so then they’ve added to the knowledge. If they only replicated I would not give a masters for that. I need them to add value. I’m always looking for students to add value. So it can be risky to do a replication as a dissertation – it depends on the balance between replication and added value.
How about replication for a PhD dissertation?
At PhD level – no way. I want you doing original research. You need to be creating the data sets and have original questions.
Todd Landman, BA (UPenn), MA (Georgetown), MA (Colorado), PhD (Essex) is Professor of Government and Director of the Institute for Democracy and Conflict Resolution at the University of Essex. He is author of Human Rights and Democracy: The Precarious Triumph of Ideals (forthcoming Bloomsbury), Protecting Human Rights (Georgetown 2005), Studying Human Rights (Routledge 2006), and Issues and Methods in Comparative Politics (Routledge 2000, 2003, 2008) and various other books. He has numerous articles published in, among others, International Studies Quarterly, The British Journal of Political Science, Human Rights Quarterly, The Journal of Human Rights. As a political methodologist, Professor Landman has carried out a large number of international consultancies in the areas of development, democracy and human rights with an emphasis on quantitative methods and analysis. www.todd-landman.com | @DrToddLandman | Blog
The interview was conducted via skype on January 22, 2013. Part I of the interview is here.