Replication and transparency in political science – have we made any progress?

When a range of top political science journals signed a statement to enforce transparency in 2014 (JETS statement), there was an immediate backlash by qualitative researchers. Hundreds of scholars signed a petition against strict transparency rules asking for clarification. Then the LaCour scandal happened, where a political scientist fabricated a study and pretended to withhold his data because of confidentially. Another wake-up call. Where is the debate in political science now? 

Coding errors can be avoided

An article in the American Journal of Political Science was corrected after the coding of a political attitude variable was accidentally the wrong way around. Pre-publication cross-checks by the authors and the journal, as well as publication of the original data and variable transformations can avoid such problems.

Published articles understate the probability of type I errors

Can we trust published articles in political science? A recent paper suggests that we should be sceptic. When comparing the published results of survey experiments with the pre-registered plans for the same study, a lot of information gets lost. 80 percent of the studies failed to report all experimental conditions and planned outcomes.

Political Scientists Trying to Delay Research Transparency

A group of 625 political scientists signed a petition to delay the new APSA guidelines for transparency. They want to discuss the implications for qualitative data, hand-written field notes and confidential data first. I agree that practical discussions are necessary – but this should not be a reason to abandon the transparency guidelines.

Journal editor: It is up to the author to release their data

I received this email from a journal editor in political science after I asked about their replication policy. My original email is below.

Leading journal verifies articles before publication – so far, all replications failed

The American Journal of Political Science recently announced that every article will be externally verified before it is published. Authors have to provide all data, code and materials which will be verified by an external statistician. I talked to the editor William G. Jacoby about the goals of the new policy, and why no author has provided ‘perfect’ files yet.

Repost: Do Political Scientists Care About Effect Sizes – Replication and Type M Errors

Repost of an article by Christoper Gandrud: Reproducibility has come a long way in political science. Many major journals now require replication materials be made available either on their websites or some service such as the Dataverse Network. This is certainly progress. But what are political scientists actually supposed to do with this new information?

Share your replication: Political Science Replication Initiative

Earlier this year, we sent out a survey on replication in political science to the Political Methodology (PolMeth) mailing list. Our survey results, which we wrote up in a post on the Monkey Cage blog, indicated quite a bit of interest in a site to share replication studies.

Guest Post: Leading international studies journal takes replication seriously, by Joseph Young

Transparency and data access – these public goods are crucial principals for science. But why don’t researchers implement them? Political scientist Joseph K. Young discusses incentives for reproducibility, and how he is tracking down old replication data for the leading international studies journal ISQ.

Against publication bias: Politics journal invites pre-registered papers

After fields like Psychology have explored pre-registration of studies, now Political Science follows the trend for more transparency in research. A high ranking journal, Comparative Political Studies, plans a special issue in which all papers have to be pre-registered and are submitted without the results section. I asked Michael Findley, one of the editors behind the initative, about publication bias and transparency in the field.