In a blog post by Jesse Singal at the New York Magazine, one of the students who tried to replicate LaCour‘s findings on gay marriage said that young researchers are often actively discouraged from replicating work.
The criticism David Broockman (and his co-authors) faced was:
- young replicators become known as troublemakers
- young replicators should do original work instead
Here’s the paragraph:
Most important, in Broockman’s opinion, his experience highlights a failure on the part of political science to nurture and assist young researchers who have suspicions about other scientists’ data, but who can’t, or can’t yet, prove any sort of malfeasance. In fact, throughout the entire process, until the very last moment when multiple “smoking guns” finally appeared, Broockman was consistently told by friends and advisers to keep quiet about his concerns lest he earn a reputation as a troublemaker, or — perhaps worse — someone who merely replicates and investigates others’ research rather than plant a flag of his own. [read more]
This criticism reminds me of similar points that were made at the International Studies Association’s annual convention last year. I answered to this criticism in my paper on replication in university teaching [pdf]:
Criticism 3: There could be reputational repercussions for young scholars if their first appearance in the “journal arena” is a paper that aims to denigrate “big names.” Such criticism seems patronizing. Introducing replication in the classroom ensures that students learn to conduct replication studies professionally, using adequate methods and language. I am not sure that the community really punishes replicators in the job market; if it does, then it must change. The answer cannot be to stop students from checking existing work. [[read more]:]
Why young scholars should replicate
Instead, we should remember all the reasons why young researchers can benefit from doing replication studies, which I summarized at my talk for the BITSS summer institute on transparency and reproducibility. [By the way, Brookmann was at the summer institute to report his experiences with replication.]
I wrote about these reasons to replicate before, but it’s worth remembering them.
Here’s a few replication studies by students that show that replications are not a waste of time at all:
More literature on how to replicate and why it’s worth it
• Janz, N. (2015) Bringing the Gold Standard Into the Class Room: Replication in University Teaching, International Studies Perspectives, Article first published online: 9 March 2015. Copy at: https://osf.io/argeh/
• Brandt et al. (2014) The Replication Recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, Vol 50, pp. 217-224. Copy at: http://tinyurl.com/poe474k
• Porte, G. (2012) Replication Research in Applied Linguistics, Cambridge: Cambridge University Press.
• King, Gary. (2006). How to Write a Publishable Paper as a Class Project,copy at: http://gking.harvard.edu/papers (with updates)
• Michael Clemens. 2015. “The Meaning of Failed Replications: A Review and Proposal.” CGD Working Paper 399. Washington, DC: Center for Global Development. Copy at: http://tinyurl.com/peaclnl
Replication Studies by Students
• Bernard Fraga & Eitan Hersh (2010). Voting Costs and Voter Turnout in Competitive Elections. Quarterly Journal of Political Science, 2010, 5: 339–356.
• Alexander Coppock (2014). Information Spillovers: Another Look at Experimental Estimates of Legislator Responsiveness. Journal of Experimental Political Science, 1, pp 159-169.
• Yin Wu, Eric van Dijk, & Luke Clark (2015). Near-wins and near-losses in gambling: A behavioral and facial EMG study. Psychophysiology, 52, pp. 359-366.
• Saleuddin R, Coffman DD (2014). Can inflation expectations be measured using commodity futures prices? Cambridge Working Papers in Economic and Social History, copy at: http://tinyurl.com/ncdddf5
• Bell, M. & Nicholas L. Miller (2013) Questioning the Effect of Nuclear Weapons on Conflict. Journal of Conflict Resolution 59 (1): 74–92.
• Thomas Herndon, Michael Ash & Robert Pollin (2013). Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff. Working Paper Series 322. Political Economy Research Institute. Copy at: http://tinyurl.com/d8nbbuf
• David Broockman, Joshua Kalla, Peter Aronow, Irregularities in LaCour (2014). Copy at: http://tinyurl.com/lgzsr3t
Political Science Replication 