Only 18 of 120 political science journals have a replication policy

A recent article in “European Political Science” found that out of 120 political science journals only 18 have replication policies. The authors found that the existence of a data policy was correlated with the impact factor, type of audience, and age of the journal. Their analysis confirms that the state of reproducibility and political science is substandard.

Sergiu Gherghina and Alexia Katsanidou stress in their article “Data Availability in Political Science Journals” that replication “is essential for the evaluation of the quality of a piece of work”. Replication also uncovers new paths in research, and improves methodology. The research question of their paper is therefore: What is the situation of data availability policies in political science journals?

Data availability policies on websites

Their exploratory study uses a sample of 120 international peer-reviewed political science journals. During three months in 2011, they searched websites for data availability policies, and sent emails to the editors when they could not find anything. “Following the content check of journal websites, we were able to identify data policies for eighteen journals.” Even when they sent emails to the other 102 publications asking for data policies, only half of the editors replied. The authors got no answer from 57 editors even though they sent several reminders. One journal, International Studies Quarterly, had a policy but it was not online at that point (update: it is now online).

This result is outrageous and embarrassing at the same time. There has been a long-standing debate in the field about the necessity of validation of published work. While it might be the case that some of the ‘bad’ journals still provide data sets, or the authors send out data on request, the lack of a clear policy seems to be a signal that journals don’t seem to care about the quality of the work they publish.

Why do journals not have a replication policy?

So, why do only few journals in political science have a replication policy? In a second step Gherghina and Katsanidou searched for explanations. They collected information about the journals they examined, which include: (1) the age of the journal, (2) the frequency of issues per year, (1) English language, (4) general interest or specialized audience, and (5) impact factor of each journal. Running bivariate correlation tests (Pearson/Spearman) between these variables and existence of a policy. They find that journal with a higher number of citations (impact factor) seem to be more likely to have a data availability policy. We do not know the direction of this correlation, or if there exists a causal link between impact factor and replication policy. I’m not completely convinced by the authors’ conclusion that journals with a policy are more popular than the rest and that the adoption of the policy could be a solution to increase publications visibility. It could also be the other way round: journals with a high impact factor have a reputation to lose, and they might therefore be more inclined to engage in high standard policies.

The authors also find that journals with a general audience tend to have a policy, compared to those with special interest topics. In addition, older journals are slightly more likely to have a replication policy. Very few of recently emerged journals adopted state of policies. This is also somewhat surprising since I would think that younger journals try to imitate successful, high impact journals and start their publication with the latest developments (i.e. having a replication policy) in mind.

The good ones: journals with a policy

[1] American Journal of Political Science

[2] American Political Science Review

[3] American Politics Research

[4] British Journal Of Political Science

[5] Communist And Post-Communist Studies

[6] Electoral Studies

[7] European Union Politics

[8] International Political Science Review

[9] Journal of Conflict Resolution

[10] Journal of Peace Research

[11] Journal of Politics

[12] Political Analysis

[13] Political Communication

[14] Political Research Quarterly

[15] Politische Vierteljahresschrift

[16] Public Choice

[17] Public Opinion Quarterly

[18] Quarterly Journal of Political Science

and:

[19] International Studies Quarterly (ISQ did have a policy, even though it was not online yet during the data collection)

The bad ones

101 journals did not have a data policy available at the point of data collection of the paper in 2011 (if you don’t count ISQ). Out of these, seven have a high impact factor so that I would have expected that they take care of this:

[1] European Journal Of Political Research

[2] Governance-An International Journal Of Policy Administration And Institutions

[3] Political Geography

[4] Political Psychology

[5] Annual Review Of Political Science

[6] Comparative Political Studies

[7] Global Environmental Politics

Replication of the analysis

The authors make their data set available here.

According to the data set, the list of journals with a data policy is 19, so they include ISQ. I quickly re-ran the correlation tests (Spearman and Pearson’s r), and the results matched except for one: the correlation between data policy and age was 0.19 instead of 0.20, but it was still significant, so this is is not really a problem.

cor.test(Data.Policy,age,use="pairwise.complete.obs",method="spearman") #0.1944095 p-value = 0.03412

[update: Will Lowe has now also started replicating the results on his blog.]

Future research

This exploratory study is an excellent basis for further research into reproducibility in political science. It would be interesting to see an update, adding other variables, maybe with more advanced statistics on the data.

One could also translate their study into sub-fields of political science or into economy, psychology, development or sociology to compare the ‘state’ of replication policies across disciplines. A similar study deals with data availability in economics.

Literature

• Gleditsch, N.P. (2002) ‘Double-blind but more transparent’, Journal of Peace Research 39(3): 259–262.
• Gleditsch, N.P. and Metelits, C. (2003) ‘Replication in international relations journals: Policies and practices’, International Studies Perspectives 4(1): 72–79.
• King, G. (2003) ‘The future of the replication movement’, International Studies Perspectives 4(1): 100–105.
• Piwowar, H.A., Day, R.S. and Fridsma, D.B. (2007) ‘Sharing detailed research data is associated with increased citation rate’, PLoS ONE 2(3): 1–5.

More on this

Read what one of the editors of European Political Science, the journal in which this study was published, thinks about replication.