One of the graduate students in my Replication Workshop writes about replication, the reproducibility crisis and validity in experimental psychology.
I was initiated to issues of replicability early in my academic career. As an undergraduate in Australia I conducted an adaptation of Stanley Milgram’s classic experiment, which found that participants administered potentially lethal shocks to other individuals when under instructions from an authoritative figure (Milgram, 1963). Specifically, I found evidence that in certain conditions, positive portrayals of the prospective victim could also motivate participants to administer more intense shocks. Excited, I was ready to begin drafting my manuscript.
My supervisor was equally excited, but told me that my study was underpowered and that I should add some more participants before publishing. At the time, I also had the opportunity to replicate my finding in Singapore. Lo and behold, my original effect disappeared and my cross-cultural replication failed. Upset, I turned to the literature. My original finding wasn’t atypically underpowered, and the design seemed as rigorous as most others in the journal I had originally targeted. This got me thinking – how valid was other research in experimental psychology? To begin to answer my question, I turned to both the academic literature, and the blogosphere.
Phase 1 of social psychology’s replication crisis
I did this in late 2012 – right at the apex of what I think of as ‘Phase 1’ of social psychology’s replication crisis. Perspectives on Psychological Science had just issued its “Special Section on Replicability in Psychological Science: A Crisis of Confidence,” which was a candid scrutiny of research in social psychology. Meanwhile, a leader in the field – John Bargh – had issued a number of incendiary blog posts criticising Ed Yong’s (2012) coverage of a failed replication of Bargh, Chen & Burrows (1996).
I emerged with my idealised vision of the academy shattered. For the first time, I thought that research in social psychology could be methodologically weak, and researchers reactionary. I was subsumed by existential terror. As an aspiring social psychologist, how on earth could I ensure that my research was based on well-validated theory? How could I know that my own methods were rigorous? What on earth was rigorousness?!
I emerged with my idealised vision of the academy shattered.
Fortunately, social psychology’s replication crisis soon entered ‘Phase 2‘. Cumming (2013) explained how to conduct robust analyses through estimation; Funder et al. (2014) provided a model of best practise; and, Murayama, Pekrun and Fiedler (2014) gave reassurances that the decades of empirical research in social psychology weren’t wholly invalid. I came out of Phase 2 with greater understanding of how empirical research can achieve both internal and external validity, and an appreciation for the value of convergent evidence.
As a result, I developed an admiration for John Haidt’s Moral Foundations Theory. Haidt (2007) derived his five moral domains from a cross-culturally robust questionnaire, and thousands of participants. Moreover, his theory was buttressed by convergent evidence from a range of disciplines, like social anthropology and political science, as well as psychology.
I didn’t care that it was purely descriptive. For a still nascent discipline like psychology, descriptive research is incredibly useful (see Rozin, 2001). In short, Phase 2 of the replication crisis pushed me to towards a new epistemology that emphasised extensive theory validation, and the simultaneous emphasis on both rigorous enquiry and convergent forms of flawed evidence. I also retained a lingering but healthy cynicism towards existing research in social psychology. I took this epistemology into my PhD.
In 2014, I began my PhD in the Prosociality and Well-Being Lab, at the University of Cambridge (UK). Like Haidt’s Moral Foundations Theory, the initial phases of my research are descriptive. Specifically, I am examining when others’ personalities enhance individuals’ well-being. My project is computationally and sample-size intensive (I use regressions with multiple levels of nested variables to detect moderate and sometimes weak effects). Thus, there isn’t a lot of previous research to guide my hypotheses. The one exception is Fulmer et al. (2010), who found that participants with high extroversion were happier when others in their country were also high in extroversion. Wary of the importance of this singular finding, I decided to replicate it using the author’s original data. The Cambridge Replication Workshop proved the ideal environment to do this.
Conviction to replicate
My replication experience was wholly positive. The original authors were forthcoming with their data, and with support from the instructor and a particularly dedicated Teaching Assistant named Sergio Cuesta, I replicated their effects with only minor and justifiable discrepancies. I also conducted an extension of the finding with additional controls. Although the extension was inconclusive, results provided impetus for my forthcoming project, which builds on my replication project by asking similar questions of some four-plus million American participants. I emerged from my experience both justified in my conviction to replicate, and reassured that others in social psychology were dedicated to the same pillars of empirical rigor that I have come to cherish.
About Matthew J. Samson
Matthew J. Samson is a graduate student at the Department of Psychology, University of Cambridge. He received his BA in Arts-Psychology from Macquarie University (Australia) and his MPhil in Social and Developmental Psychology from the University of Cambridge. During his BA, Matt spent one year studying and conducting research in the Division of Psychology at Nanyang Technological University (Singapore), and an additional six months studying at Michigan State University (USA). Matt’s backgrounds are in social cognitive, embodied and cross-cultural psychology. In his PhD, Matt hopes to ask questions of big data that help us to understand the predictors of psychological well-being. You can find his CV here. Twitter: @mjsamson05
Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behaviour: Direct effects of trait contrast and stereotype activation on action. Journal of Personality and Social Psychology, 71, 230-244.
Cumming, G. (2013). The new statistics: Why and how. Psychological Science, 25, 7-29.
Fulmer, C. A., Gelfand, M. J., Kruglanski, A. W., Kim-Prieto, C., Diener, E., Pierro, A., & Higgins, E. T. (2010). On “feeling right” in cultural contexts: How person-culture match affects self-esteem and subjective well-being. Psychological Science, 21, 1563-1569.
Funder, D. C., Levine, J. M., Mackie, D. M., Morf, C. C., Sansone, C., Vazire, S., & West, S. G. (2014). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practise. Personality and Social Psychology Review, 18, 3-12.
Haidt, J. (2007). The new synthesis in moral psychology. Science, 316, 998-1002.
Milgram, S. (1963). Behavioral study of obedience. The Journal of Abnormal Social Psychology, 67, 371-378.
Murayama, K., Pekrun, R., & Fiedler, C. (2014). Research practises that can prevent an inflation of false-positive rates. Personality and Social Psychology Review, 18, 107-118.
Rozin, P. (2001). Social psychology and science: Some lessons from Solomon Asch. Personality and Social Psychology Review, 5, 2-14.
Yong. (2012). Bad copy. Nature, 485, 298-300.