Guest Post: The case for pre-registration in social science, by Jamie Monogan

pre-registration replicationPolitical scientist Jamie Monogan, University of Georgia, discusses preregistration: Why should you announce your data analysis beforehand? Which journals support preregistration? And where can you register your study?

Political Scientists have placed increasing emphasis on transparency in research. The Political Science Replication blog itself is a testament to this trend, in that it serves to promote reproducibility and replication in research. As part of this trend in transparency, one notion that has emerged is the idea political scientists can preregister their research. Study registration is the idea that a researcher can publicly release a data analysis plan prior to observing a project’s outcome variable.

While there is controversy over the topic (see: Anderson 2013; Gelman 2013; Laitin 2013), several scholars have made a case for the benefits of preregistration in social research (Asendorpf et al. 2013; Casey, Glennerster, and Miguel 2012; Humphreys, de la Sierra, and van der Windt 2013; King et al. 2007; King et al. 2009; Monogan 2013).

Why do it?

The case for preregistration in social science research makes several points, but in brief a few of them are:

  1. First, preregistration can curb publication bias. It does this by tying the researcher’s hands a priori so that he or she cannot manipulate the sample or design to produce a positive result, and it reduces the “file-drawer problem” by creating a public record of projects regardless of whether a publication subsequently emerged.
  2. Second, preregistration clearly distinguishes inductive from deductive studies, as registering a design clarifies whether theory was developed first or whether data was studied first.
  3. Third, preregistration can be liberating to researchers who wish to obtain honest results even in the face of pressure from grant agencies or government officials who would like for a positive result to be reported.

Pioneer journals supporting preregistration

A number of journals have begun to incorporate optional preregistration into the editorial process. The instructions for authors at Political Analysis, for example, “encourages authors to consider preregistering their studies, when appropriate”. The guidelines go on to list a number of registries that are available. The neuroscience journal Cortex has even started a publication opportunity called “Registered Reports,” wherein an author submits a research design for review at the preregistration stage. If the editor and reviewers choose to accept the article at the design phase, then it will be published regardless of the results, provided that the preregistered design is implemented as stated (Chambers 2013). Hence, the practice of preregistration is starting to get some usage in the social sciences.

Where to register your analysis

Scholars interested in this new trend, or perhaps unsure of what to think of preregistration, are encouraged to preregister one of their own projects. By preregistering a design and completing the research process using these steps, a scholar can decide firsthand how well the process works. The Political Science Registered Studies Dataverse is a free resource that any scholar may take advantage of to register his or her own research designs and try preregistration in practice. Researchers may upload their own study using the automated platform, or they may contact the webmaster (monogan@uga.edu) about creating an anonymous version of their registered study. The latter process would allow authors to include preregistration information in a double-blind review, with the webmaster revealing the authors’ identity upon acceptance of a publication.

Replication PreregistrationThe Political Science Registered Studies Dataverse offers a variety of features that can be of use to registrants, including:

  • The Dataverse is hosted by an independent party, the IQSS at Harvard University. Any registered design is dated, archived, and made available for all interested parties to review.
  • Each study is given a unique global identifier that permanently identifies the study.
  • Public comments for suggestions can be recorded, along with researchers’ responses.
  • Any changes to the study will be recorded in a new version of the study, with prior versions preserved and publicly available.
  • Preliminary and pre-outcome data can be archived, and these will be identified with a universal numeric fingerprint. Further, complete replication data can also be added after the study’s completion.

The Political Science Registered Studies Dataverse is one platform for preregistration that is available, and the practice is getting increased traction in the social sciences. You are invited to use this resource in your own research to evaluate how useful this practice will be for the discipline.

About Jamie Monogan

Jamie Monogan ReplicationJamie Monogan is an assistant professor in the Department of Political Science, part of the School of Public and International Affairs, at the University of Georgia. Jamie’s substantive research focuses primarily on American state politics and policy, with published work that investigates the causes of policy choices in the areas of immigration, health care, the environment, and general policy liberalism. His methodological research has focused on geospatial data analysis, time series analysis, and study preregistration. Twitter: @jmonogan

Literature

Anderson, Richard G. 2013. “Registration and Replication: A Comment.” Political Analysis 21(1):38–39.

Asendorpf, Jens B., Mark Conner, Filip de Fruyt, Jan de Houwer, Jaap J. A. Denissen, Klaus Fiedler, Susann Fiedler, David C. Funder, Reinhold Kliegl, Brian A. Nosek, Marco Perugini, Brent W. Roberts, Manfred Schmitt, Marcel A. G. Vanaken, Hannelore Weber and Jelte M. Wicherts. 2013. “Recommendations for Increasing Replicability in Psychology.” European Journal of Personality 27(2):108–119.

Casey, Katherine, Rachel Glennerster and Edward Miguel. 2012. “Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan.” Quarterly Journal of Economics 127(4):1755-1812.

Chambers, Christopher D. 2013. “Registered Reports: A New Publishing Initiative at Cortex.” Cortex 49(3):609–610.

Gelman, Andrew. 2013. “Preregistration of Studies and Mock Reports.” Political Analysis 21(1):40–41.

Humphreys, Macartan, Raul Sanchez de la Sierra and Peter van der Windt. 2013. “Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration.” Political Analysis 21(1):1–20.

King, Gary, Emmanuela Gakidou, Kosuke Imai, Jason Lakin, Ryan T. Moore, Clayton Nall, Nirmala Ravishankar, Manett Vargas, Martha María Téllez-Rojo, Juan Eugenio Hernández Ávila, Maurico Hernández Ávila and Héctor Hernández Llamas. 2009. “Public Policy for the Poor? A Randomized Assessment of the Mexican Universal Health Insurance Programme.” Lancet 373(9673):1447–1454.

King, Gary, Emmanuela Gakidou, Nirmala Ravishankar, Ryan T. Moore, Jason Lakin, Manett Vargas, Martha María Téllez-Rojo, Juan Eugenio Hernández Ávila, Maurico Hernández Ávila and Héctor Hernández Llamas.2007. “A ‘PoliticallyRobust’ Exper- imental Design for Public Policy Evaluation, with Application to the Mexican Universal Health Insurance Program.” Journal of Policy Analysis and Management 26(3):479–506.

Laitin, David D. 2013. “Fisheries Management.” Political Analysis 21(1):42–47.

Monogan, III, James E. 2013. “A Case for Registering Studies of Political Outcomes: An Application in the 2010 House Elections.” Political Analysis 21(1):21–37.

Advertisements
Tagged , , ,

One thought on “Guest Post: The case for pre-registration in social science, by Jamie Monogan

  1. […] up would be to require authors to pre-register their whole analysis. At the moment, this is not common yet in political science, and only one […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: