The current issue of PS: Political Science & Politics (Vol 47, Issue 1) is devoted to reproducibility, replication and data access. In eight articles, political scientists discuss the need for better quality standards in qualitative and quantitative research. This blog will publish a series of posts on the main points, starting with observations on the current state of reproducibility by Arthur Lupia and Colin Elman.
In their article Openness in Political Science: Data Access and Research Transparency, Arthur Lupia (University of Michigan) and Colin Elman (Syracuse University) discuss new policies for data access and research transparency developed by the American Political Science Association (APSA) Council. The policies replace earlier guidelines after “an extended and broad consultation with a variety of APSA committees”.
Legitimacy and credibility of published work should not be “based solely on personality cults” around a famous professor or the fact it was produced in an ivy leage university.
[T]he reason to believe a scientist’s claim is not because he or she wears a lab coat, have a PhD, or have published a widely viewed paper in the past. Appeals to personality or faith, which facilitate information transmission in other domains, are not supposed to be required to access the content of a scientific claim. A claim’s perceived legitimacy is grounded in the fact that the results are the product of publicly described processes that in turn are based on a stable and shared set of beliefs about how knowledge is produced. Such open access to the origins of others’ claims is the hallmark of scientific ways of knowing.
Whenever scientists fail to provide a detailed documentation of their “assumptions, decisions, and actions”, the community cannot understand or assess the results. In fact, the credibility of the whole political science community is imperiled when failures to replicate, and scandals about reproducibility are frequent.
New ethics guidelines
Therefore, the new APSA ethics guidelines* now state that “researchers have an ethical obligation to facilitate the evaluation of their evidencebased knowledge claims through data access, production transparency, and analytic transparency so that their work can be tested or replicated”:
- Data access: Researchers making evidence-based knowledge claims should reference the data they used to make those claims. If these are data they themselves generated or collected,
researchers should provide access to those data or explain why they cannot.
- Production transparency: Researchers providing access to data they themselves generated or collected, should offer a full account of the procedures used to collect or generate the
- Analytic Transparency: Researchers making evidence-based knowledge claims should provide a full account of how they draw their analytic conclusions from the data, i.e., clearly
explicate the links connecting data to conclusions.
[* update 19th June, 2015: the link is now dead. The guidelines are in the Lupia and Elman 2014 paper and a screenshot of the relevant page is here.]
In their article, Lupia and Elman point out that the new guidelines are “more consistent
with current and emerging standards across the sciences.”
Where APSA’s previous language emphasized making data accessible only when findings were challenged, the new guidelines recognize data access and research transparency as an indispensable part of the research endeavor.
Replication and Data Access
While these guidelines are set for qualitative and quantitative researchers in the field, those using statistics have to translate this into: make replication possible; and provide data access for published work.
Through replication, quantitative researchers can “evaluate claims and form an evidentiary and logical basis for treating the claims as valid.” Access to data is not only crucial for replication. But researchers can make more use of existing data sets through secondary analysis.
Shared data can be a valuable public good. (…) In the best-case scenario, secondary data analyses allow authors to derive meaning from data that need not have occurred to the original researcher. When scholars can use research materials in these diverse ways, the data can become more valuable to science and society. Instead of a dataset producing one set of insights, data sharing gives other scholars the ability to multiply datas’ value.
Putting guidelines into practice
The following years will show how political scientists apply the new guidelines in practice. While many of the benefits of data sharing and transparency are already accepted in the field, many researchers are reluctant to implement them for various reasons:
- lack of clearly specified guidelines as to what kinds of data and research information should be shared
- lack of professional incentives for documenting the evidentiary and logical foundations of knowledge claims
- time and monetary expense involved in archiving research material
- potential for embarrassment that can come from having one’s work replicated
The way forward is therefore a change in graduate student training to establish a transparency culture; changes in journal policies for authors submitting work; data production and access being valued more for tenure and promotion.
Read the article by Lupia and Elman
Arthur Lupia and Colin Elman (2014). Openness in Political Science: Data Access and Research Transparency . PS: Political Science & Politics, 47, pp 19-42. doi:10.1017/S1049096513001716.