Academic slugfest: the wonderful world of replication chains

Screen Shot 2013-03-08 at 6.28.24 PMSometimes an author, when being replicated by someone, answers to that replication in a new paper. In that new paper (s)he again might replicate some of the disputed results. Most authors defend their earlier paper by claiming that the replication was: fundamentally flawed, contains statistical and reporting errors, is of trivial nature, or less realistic and of limited utility. Such replication chains are not just entertaining academic slugfest, but they are useful because they provide detailed discussions about data and methods in the field.

Replication chain on voter calls and turnout

Screen Shot 2013-03-13 at 4.24.33 PMStep 1 – Original article: The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment, by Alan S. Gerber and Donald P. Green, The American Political Science Review, Vol. 94, No. 3, Sep., 2000.

Abstract: We report the results of a randomized field experiment involving approximately 30,000 registered voters in New Haven, Connecticut. Nonpartisan get-out-the-vote messages were conveyed through personal canvassing, direct mail, and telephone calls shortly before the November 1998 election. A variety of substantive messages were used. Voter turnout was increased substantially by personal canvassing, slightly by direct mail, and not at all by telephone calls. These findings support our hypothesis that the long-term retrenchment in voter turnout is partly attributable to the decline in face-to-face political mobilization.

 

Screen Shot 2013-03-13 at 4.35.58 PMStep 2 – Replication: Do Get-Out-the-Vote Calls Reduce Turnout? The Importance of Statistical Methods for Field Experiments, by Kosuke Imai, The American Political Science Review, Vol. 99, No. 2, May, 2005.

“(…) Gerber and Green’s negative finding is caused by inadvertent deviations from their stated experimental protocol. The initial discovery led to revisions of the original data by the authors and retraction of the numerical results in their article. Analysis of their revised data, however, reveals new systematic patterns of implementation errors.”

 

Screen Shot 2013-03-13 at 4.36.09 PMStep 3 – Answer from original author: Correction to Gerber and Green (2000), Replication of Disputed Findings, and Reply to Imai (2005), by Alan S. Gerber and Donald P. Green, The American Political Science Review, Vol. 99, No. 2, May, 2005.

“This essay corrects the results reported in Gerber and Green 2000 and replies to Imai (2005). When data-processing errors are repaired, the original substantive findings from the New Haven experiment remain unchanged. (…) The “correction” that Imai (2005) offers, which purports to show that these phone calls produce large, significant, and robust increases in voter turnout, is shown to contain statistical, computational, and reporting errors that invalidate its conclusions about the effectiveness and cost-effectiveness of phone calls and mail.”

 

You might also want to read:

Replication chain on values and political tolerance

Screen Shot 2013-03-13 at 4.24.55 PM

 

Coming soon: Replication chain on international trade and democracies

Screen Shot 2013-03-13 at 4.25.12 PM

Advertisements
Tagged , , , , , , ,

2 thoughts on “Academic slugfest: the wonderful world of replication chains

  1. […] presented a “fundamentally flawed analysis”, and, in a third case, an original author said that the replicator published “statistical, computational, and reporting errors that […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: