Human rights scholar Todd Landman on replication: “Be as public as you possibly can”

Political scientist Todd Landman (University of Essex) talks about data sharing ethos in the quantitative human rights field. In our interview he proposes that all journals should have a data depository, and explains how replication can improve methods and estimation techniques.


You work in quantitative human rights. Do you think there is a good ethic of data sharing in the community?
Yes, very much so.

Why?
I don’t think we’re threatened by replication. (…) There’s an advocacy element to human rights work, and whenever you deal with people who are advocates they want to share information. People who create terror scales and rights indices want to produce high quality data for the users. Linda Camp Keith is really generous with her data, Bethany Barratt and I share lots of data, David Cingranelli makes all his data available, he wants people to have his data. Neil Mitchell shares everything, Sabine Carey is very open. Although with tricky variables, like comprehensive legal codings, one usually waits until the work is published, and then people are happy to share.

We have recently seen a range of ‘confessions’ of natural and social scientists on twitter (#overlyhonestmethods). Some of them joke “you will never be able to reproduce our results”. Do you think these are common attitudes of researchers, or exceptions, or simple jokes?
First time I ever heard it. I have requested data from people and they’ve not supplied it. People request data from me and I supply it, if I remember. You know, they have to send me three emails because I’m so busy that I forget to get to that. But my principle is definitely to send out data. People have requested the ISQ data and I’ve always sent them out. The other one is the treaty ratification data that I’ve sent out. It’s outdated now, but people can add years to it. In my book I advocate the public nature of what we’re doing, and that we should be public about procedures, and that people should be able to replicate what we’ve done. (…) There’s no reason why you should not share your data. It’s a scientific process. Sharing code is another thing, because a lot of us don’t keep the code. Good practice is you keep it and share it.

Why data sharing in political science – whom does it benefit?
It benefits the original producer, because it’s nice to have someone cross-checking your work. You might have made a fundamental error! The funny thing is, when we make a mistake nothing happens. When a drug company makes a mistake, people die. So the pressure is a little lower. But I think it’s really cool if someone says, ‘can I have your data set, I want to replicate it’. Or, ‘we cannot replicate your data, what did you do’? Then you can go into a discussion with them. (…) It also helps you to get your data out there. You create a documentary trail and a lineage or chronology that you have worked on this topic, and that you created these data.

How does it benefit the user who replicates work?
They learn the steps another scholar has taken in terms of case selection, model specification, estimation. And they might have used a different version of OLS, so you can use different estimators and compare that to learn the difference. The war over pooled cross-section data is a good example. There are different estimation techniques for standard errors, Beck/Katz corrected errors that become code in STATA, and people start trying this and sometimes they find it does not work. Comparing results shows that there is often not a lot of difference, but replication is part of this story and very very useful for the conversation to go forward.

Do you think some people are afraid of someone else finding your errors?
Yes. Though I don’t mind if someone did that with my stuff, because you then enter into a conversation. Incremental gains only come from mistakes. And it seems to me that there’s a bias in the literature towards positive findings, and that really bothers me.

What role do journals play in data availability?
Journals should have a data depository. Problem is that journals are run by publishers, and publishers don’t want massive data sets sitting on their servers, I guess. There should be some attempt to collate data, but the problem is that someone has to check the quality of the data. Right now it’s completely anarchic – some journals require it, some don’t. Some authors put data sets online, like Gary King, others don’t. There’s no compulsion to do that, none whatsoever.

Do you ask for the data when you review a journal article?
Not for review. I’ve never done that. Partly because we are so busy with work, we look through an article and say: Have they done the job? Is it an interesting question? Did they specify the model properly, are there any methodological issues? Is there spuriousness, multicollinearity, are they using the data consistently? Then we pass a judgment on that.

Part II of the interview

Read more about how replication can be used in statistics training, how Todd Landman administers his own data, why he is sceptical of replication as a dissertation, and how data sharing can work for qualitative researchers in part II of the interview.

Bio

Todd Landman, BA (UPenn), MA (Georgetown), MA (Colorado), PhD (Essex) is Professor of Government and Director of the Institute for Democracy and Conflict Resolution at the University of Essex. He is author of Human Rights and Democracy: The Precarious Triumph of Ideals (forthcoming Bloomsbury), Protecting Human Rights (Georgetown 2005), Studying Human Rights (Routledge 2006), and Issues and Methods in Comparative Politics (Routledge 2000, 2003, 2008); co-author of Measuring Human Rights (Routledge 2009), Assessing the Quality of Democracy (International IDEA 2008); Governing Latin America (2003), and Citizenship Rights and Social Movements (Oxford 1997, 2000); editor of Human Rights Volumes I-IV (Sage 2009), and co-editor of the Sage Handbook of Comparative Politics (Sage 2009) and Real Social Science: Applied Phronesis (Cambridge 2012). He has numerous articles published in International Studies Quarterly, The British Journal of Political Science, Human Rights Quarterly, Democratization, Political Studies, The Journal of Human Rights, The British Journal of Politics and International Relations, Electoral Studies, Human Rights and Human Welfare, Public Law and The California Western International Law Journal. As a political methodologist, Professor Landman has carried out a large number of international consultancies over the last 15 years in the areas of development, democracy and human rights with an emphasis on quantitative methods and analysis. www.todd-landman.com | @DrToddLandman | Blog

Interview

The interview was conducted via skype on January 22, 2013.

Literature

• Todd Landman and Marco Larizza (2009) ‘Inequality and Human Rights: Who Controls What When and How’, International Studies Quarterly, 53 (3): 715-736.
• List of data sources for human rights, conflict and development research by the Institute for Democracy and Conflict Resolution.