Guest Post: I asked Nate Breznau, an empirical sociologist, to write about his experience replicating published work – the challenges, benefits, and how he got published.
My dissertation was inspired by two studies on social welfare – and my thesis supervisor asked me to replicate the articles to help me enter the field. The papers by Brooks and Manza (2006a,b) were the first ever to offer quantitative empirical support to the theory that public policy preferences shape social welfare spending across advanced democracies. All data came from secondary sources, which made this a reanalysis or reproduction of previous research (Zigerell 2014), thus the only replication taking place was of their statistical modeling procedure. So I thought.
I could not reproduce their findings.
I expected exact reproduction of the results, given that it was simply a reanalysis with the same methods. Before I discovered some quirky things about replication with secondary data, I contacted both Brooks and Manza via an email hoping that they might share their own data with me so I could find what I assumed was my mistake. After some non-response, I finally got a reply stating that research in this area was timely and that I should look into the sources myself to construct their secondary data. Fair enough I thought, as the data are publically available.
Data and methods are never really the same
The first discovery I made was that data and methods are not necessarily the same even when they are the ‘same’. This led me to the discovery of a phenomenon I call Secondary Observer Effects (Breznau 2015): the exact numerical values of publically available secondary data tend to change, depending on when and where a researcher downloads them. Furthermore, idiosyncratic aspects of the researchers doing reproductions mean that methods are unlikely to be identical. The default statistical procedures associated with various methods may differ across software platforms (Bell and Miller 2013) or across researchers (Hartgerink 2014), for example. Thus, sharing exact methods (i.e. syntax) is the only way to overcome these biases.
The second discovery I made challenged me to reflect on what science is.
Pointing out that something is ‘wrong’ is unworthy of publication
At the start of my replication I noticed that Brooks and Manza (2006a) used an interaction term in their empirical regression model. I learned in my linear regression course a few years earlier that an interaction has three coefficients: two main effects and one interaction of the main effects. It puzzled me that Brooks and Manza only used one main effect and one interaction effect. Although this practice is not necessarily wrong, in this case their theory and interpretation of results are based on a model with the main effect. I submitted a first version of my critique of their paper to the journal where the original article was published. I was told by the reviewers and the editor that simply pointing out that something is ‘wrong’ is unworthy of publication. Apparently leaving out a main effect is not interesting enough.
I learned that science as a profession is not filled with individuals purely in the pursuit of knowledge. Journal editors and researchers have reputations that give then status and income, and these are things that researchers have self-interest in defending. I met Jeff Manza at a cocktail party, and despite being very popular as the chair of the sociology department at NYU he took time to speak to me. He was very friendly I might add. After I told him that his work formed the basis for my dissertation he was interested. Unfortunately, I made a rookie mistake by commenting just one minute later that I actually replicated his research and showed that it is wrong. I do not recommend this strategy for making acquaintances or developing professional relationships with senior scholars.
My replication of Brooks and Manza – i.e. a reproduction of the same analysis on the same data that was actually done on different data and with very slight differences in methods which remain unknown as Brooks and Manza did not share how they conducted their factor analysis or their collapsing of the attitude data they used – found an outlet in Sociological Science (Breznau forthcoming). I recommend to young scholars just this type of journal, rather than submitting to the original journal. Sociological Science is committed to very high standards while reviewing articles in less than 30 days, giving copyright to the author(s), encouraging original research and argument style as opposed to an assembly line document that must pass up to five peer reviewers, and a platform for open online discussion of articles on their website.
Massive detour during my dissertation studies
The replication felt like a massive detour during my dissertation studies as it took nearly 4 years to provide me with any publications, but I have come to believe that in addition to patience, replication is itself a scientific destination from which we may learn a great deal. Replication and reproduction can provide what Hegel describes in the scientific process as thesis and antithesis. I never would have discovered the Secondary Observer Effect, learned about the replication crisis in social science, or come to value so highly the ethical practice of freely sharing all technical details of my research as advocated by the Transparency and Openness Promotion (TOP) guidelines had I not followed this reproduction and replication detour which itself turned out to be a destination.
[update: the paper is now published at Sociological Science.]
About Nate Breznau
Nate Breznau is an empirical sociologist with degrees from Bates College (BA), the University of Nevada, Reno (MA) and the University of Bremen (PhD). His research focuses on public opinion and social policy which spans disciplines of political science and psychology. He recently completed a dissertation in 2013 which highlights how public opinion – or what are known as social policy preferences – are shaped by self-interest, institutions, values and immigration. His current research is focused on understanding the reciprocal relationship of public opinion and social policy using nonrecursive path modeling. He is currently a postdoc at the Bremen International Graduate School of Social Sciences and will join the Mannheim Centre for European Social Research at the University of Mannheim in September 2015.
Literature
Bell, Mark and Nicholas Miller. 2013. “How to Persuade Journals to Accept Your Replication Paper.” Political Science Replication Sept. 11th. Retrieved January 1, 2015.
Breznau, Nate. 2015. “Secondary Observer Effects: Idiosyncratic Errors in Small-N Secondary Data Analysis.” International Journal of Social Research Methodology (online first):1–18.
Breznau, Nate. forthcoming. “The Missing Main Effect of Welfare State Regimes: A Replication of ‘Social Policy Responsiveness in Developed Democracies’ by Brooks and Manza.” Sociological Science. Pre-publication accepted version access.
Brooks, Clem and Jeff Manza. 2006a. “Social Policy Responsiveness in Developed Democracies.” American Sociological Review 71(3):474–94.
Brooks, Clem and Jeff Manza. 2006b. “Why Do Welfare States Persist?” The Journal of Politics 68(4):816–27.
Hartgerink, Chris. 2014. “Publishing a Replication? Definitely Worth Repeating.” Political Science Replication June 18th. Retrieved January 1, 2015.
Zigerell, L. J. 2014. “Stop Trusting Other Researchers.” Political Science Replication Nov. 26th. Retrieved January 1, 2015 .
Nosek, B. A. et al. 2015. “Promoting an Open Research Culture.” Science 348(6242):1422–25.
Political Science Replication 