Setting up a replication workshop: Why it’s worth it & what to do better

More and more scholars integrate replication into their teaching, for example Gary King, Victoria Stodden, Chris Fariss, Jamie Monogan. With the second edition of the Replication Workshop starting today, here’s why it’s worth it, and how to improve teaching that includes replication – based on what my TAs said after the last Cambridge Replication Workshop.

Improving statistical skills

Basic stats knowledges from introductory models on bivariate association and regression, as well as R skills, were a prerequisite in our course. The TAs found that “it’s an excellent way of learning statistics. Replicating real, recently published work may be more motivating and enriching than using ‘perfect’ examples or than designing a study from scratch.”

Learning about the research process

It seems students from last year got a new feeling on what the research process entails, especially regarding keeping a log of all your code, keeping a clear filing system, and making data available. The TAs said:

• “I think that now we are all much more sensitive to how data needs to be presented and analysed in order to be a comprehensible contribution to scientific progress.”
• “[I]t also helps better understanding how research is done ‘in real’, beyond the ideal setting of statistical books.”
• Replicating instead of conducting their own research from the beginning was not seen as a time waste, but “they can learn about publishing without having to fully conduct their own study” yet.

How to improve the workshop

The new workshop started today. This is a perfect time to remember what went well and not so well last time. This is what TAs noticed could go better:

• “The biggest problem with the current format of the course is that actual research often times involves much more than simple or multiple regression, ANOVA, t-tests and the basic statistical techniques covered in introductions to statistics. So, even if students have done these basic introductions they might find that models they encounter are sometimes quite incomprehensible. This is also due to the fact that they will choose papers to replicate on the basis of interest rather than on the basis of whether they feel that they understand the statistics involved.”
• “The fact that many results weren’t the same as in the original papers has a risk of students getting frustrated. While this is inherently part of any replication process, I would perhaps try to address this issue: either making students aware beforehand that this is very frequent, or helping to select papers which database is well organised and ready for replication.”
• “[I]t would be good to have a set selection of papers that were selected by the lecturer and maybe the TA’s.”
• “About prerequisites, a minimum command of R is necessary.”
• “Definitely wouldn’t work for students who don’t have a background in quantitative methods. If students don’t have knowledge in R, then it would be difficult for them to keep up.”
• “Its probably possible for a TA to manage upto 3-4 students” during the practicals in class.
• “Of course, if individual departments had enough students interested in replicating studies that would be great. Having a more specialized group of students, TA’s and lecturers would most probably enhance the quality of the replications.”

[See also my other posts reporting on how the replication workshop went.]