Against publication bias: Politics journal invites pre-registered papers

After fields like Psychology have explored pre-registration of studies, now Political Science follows the trend for more transparency in research. A high ranking journal, Comparative Political Studies, plans a special issue in which all papers have to be pre-registered and are submitted without the results section. I asked Michael Findley, one of the editors behind the initative, about publication bias and transparency in the field.

cpsThe editors do not expect papers about reproducibility and how to embed it in research. Instead, the special issue of Comparative Political Studies will serve as an experiment in itself. It will publish papers on substantial political science topics, but for the first time via pre-registration. This way, the editors can learn what works, and what doesn’t. The call for papers [pdf] invites two kinds of submissions:

  1. a research design plan for a future research project: theory, main hypotheses, design, feasibility, and potential contributions of the results
  2. a research designs for a project that has already been conducted; but remove the results from the draft

Each paper or analysis plan will be sent to reviewers (without any results tables!). When the paper is accepted, all authors must make their work transparent. Qualitative researchers must provide e.g. coding notes and other documents; quantitative researchers should upload data sets and analysis code.

Interview: Goals of the special issue, and what to submit

The special issue “Research Transparency in the Social Sciences”, is edited by Michael Findley, Nathan Jensen, Edmund Malesky, and Thomas Pepinsky.

One of the editors, Michael Findley, explains the goals of their initiative: Findley

How did you get the idea for the special issue?

There has recently been much attention on pre-registration and results-blind review, but little systematic investigation of the process or implications. The debate is just beginning and we hope to shed light on best and worst practices.

Many scholars are moving forward with more transparency on their own and some norms may be developing in a less coordinated fashion – but journals need to be key players.

We should also mention that the editors of Comparative Political Studies, Ben Ansell and David Samuels, had approached some of us about possibilities for an innovative special issue on transparency and related issues. They deserve some of the credit for thinking about how a high-impact journal could lead the way in exploring how we produce knowledge.

Is pre-registration new among political science journals?

Individuals and organizations have advanced registration practices. But political science journals have not taken the initiative in this area. Actually, much of the discussion on transparency in social science research is carried out in abstract terms. Many scholars are left with questions about how it should ideally occur in practice.

How can pre-registered studies prevent publication bias (=publishing only positive, statistically significant results)?

A benefit of our approach is that it will encourage the publication of any null results that follow from faithfully carrying out pre-analysis plans. This is important because many of us worry that the lack of transparency in empirical social science research is one of the key factors leading to publication bias.

What kinds of studies are you looking for?

Ideally, contributors will propose a substantive research question for investigation. We hope researchers will tackle any of the interesting and important puzzles that would normally be submitted to Comparative Political Studies or another similar journal. As part of the proposal, they should detail the research design as thoroughly as possible to allow for peer review without any results. In other words, we are not actually looking for papers about research transparency; instead, we seek to understand the implications of being transparent.

Should prospective authors submit their papers or analysis plan, and then once they are ‘approved,’ they pre-register the study where ever they choose? Or does submitting it to the journal ‘count as’ a pre-registration with the journal editors?

Someone could preregister their design elsewhere, and then submit to us. They’d get feedback in the review process and could then update their registration document before carrying out the study. Or someone could submit without preregistering and then complete the registration after the peer review process. We expect that some/many will preregister with another outlet such as EGAP, but so long as they are transparent and public in a credible way then that’s fine.

For some useful resources on pre-registration, see:

What are you looking for in submissions using qualitative research? How can a pre-registered qualitative study work?

We believe that in principle, the benefits of transparency apply equally to qualitative and quantitative research. However, we do anticipate that there will be interesting differences in what qualitative and quantitative researchers do and how they respond to the strictures of pre-registration.

It is easier to imagine a pre-registration protocol for an experimental study than for deep comparative case studies. Moreover, claims that scholars have formulated the research protocol without knowledge of outcome measures may not be credible for case studies or quantitative analyses of observational data contained in publicly-available datasets.

In both cases, though, we expect researchers either to preregister what they intend to do in their study or submit their research results-free to allow peer-review on design only.

Efforts to increase transparency in qualitative research are moving forward and we hope that this special issue will complement those efforts.

Our hope is our special issue can shed some light on best practices for both quantitative and qualitative research papers and we hope that authors can bring in their own creative ideas.

How does the special issue support research transparency in the social sciences?

The special issue supports research transparency in the social sciences by encouraging authors to publicly declare their intentions for research ahead of data collection and analysis. The special issue will also encourage the publication of results that are not biased by the peer review process. That is, by making all decisions about publication prior to viewing any results, a wider range of possible results may make it into print.

More broadly, though, this special issue will give us a sense of the challenges that might emerge when we place greater emphasis on this particular understanding of transparency. We don’t yet have a good sense, for example, of just how common it is for research to fail to reject pre-registered null hypotheses.

We don’t know how often pre-registered research designs are forced to change in the course of conducting the research itself.

We don’t know how qualitative researchers will respond to calls for greater transparency in their own research. We may learn that there’s a lot more that we still need to learn about how to incentivize transparency in practice.

Regardless of what we learn, though, this will hopefully shed some light on pretty fundamental methodological debates where evidence is lacking.

The interview was conducted via email.

About Michael Findley

FindleyMichael G. Findley is a Political Scientist at the University of Texas at Austin. His research and teaching address civil wars, terrorism and development. He uses field experiments, statistical and computational models and some interviews, and conducts ongoing fieldwork in Uganda, South Africa and Malawi. Findley has pre-registered much of his work since 2010 and the results of the pre-registered work has appeared in Cambridge University Press, International Organization, American Journal of Political Science, Minnesota Law Review, and University of Pennsylvania Law Review.

About the special issue

The special issue in CPS, “Research Transparency in the Social Sciences”, is edited by Michael Findley, Nathan Jensen, Edmund Malesky, and Thomas Pepinsky. The deadline for submitted proposals is September 15, 2014 (will appear 2015/16 academic year). Read the full call for papers here or as pdf. Contact the editors at transparency@ipdutexas.org.

About the journal

Comparative Political Studies (CPS) is published fourteen times a year and covers work on comparative politics with important implications for the formation of domestic and foreign policies. To maintain low backlog, almost half of all submissions are not sent out for review, and then the average time from submission to first decision is, according to their website, about a month. CPS accepts submissions in Word or LaTeX (as .pdfs) and requires authors to explicitly note how they are making the data and code available.

Advertisements
Tagged , , , , , ,

3 thoughts on “Against publication bias: Politics journal invites pre-registered papers

  1. […] have to be pre-registered and are submitted without the results section. I look forward to hearing their experiences. Meanwhile, International Studies Quarterly, the leading IR journal, started tracking […]

    Like

  2. […] their whole analysis. At the moment, this is not common yet in political science, and only one journal has tried how this process could work for a special […]

    Like

  3. […] online. This practice is only slowly being taken up in political science (e.g. by the journal CPS), but there are some instances in which we can already examine the difference between an initial […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: