A group of 625 political scientists signed a petition to delay the new APSA guidelines for transparency. They want to discuss the implications for qualitative data, hand-written field notes and confidential data first. I agree that practical discussions are necessary – but this should not be a reason to abandon the transparency guidelines.
In 2014, the American Political Science Association issued the “Data Access and Research Transparency (DA-RT)” guidelines for more transparency in the discipline. The guidelines state that data and code should be made available when publishing in a journal. So far, the guidelines have been adopted by 27 journals [read the full guidelines here].
Why a transparency backlash?
Now there’s a transparency backlash. In a “Petition to Delay DA-RT Implementation” [see pdf], 625 political scientists fear that some researchers will not be able publish their work when they cannot provide the data – especially when they work with sensitive and qualitative information. The petition states:
[T]he decision about whether or not to make research materials publicly available should belong to scholars and not journal editors or reviewers.
The signatories worry that “many key questions remain unresolved”. For example:
- How to achieve transparency for non-statistical research
- Who covers costs involved with preparing data for archiving (especially hand-written data)
- Legal and ethical obligations to human subjects (e.g. in casese of research on political violence)
- Will a data repository provide graded access, and is it safe to hacking
In order to discuss these issues, the petition requests that journals abandon the transparency guidelines until at least after the 2016 annual meeting.
Why we should not delay the transparency guidelines
Of course, the concerns are important and valid. I agree that we should be very clear about what kind of information should be made transparent, and in which cases that is not possible. The DA-RT guidelines also acknowledge that:
6.4 Scholars may be exempted from Data Access and Production Transparency in order to
(A) address well-founded privacy and confidentiality concerns, including abiding by relevant human subjects regulation; and/or
(B) comply with relevant and applicable laws, including copyright.Decisions to withhold data and a full account of the procedures used to collect or generate them should be made in good faith and on reasonable grounds.
6.7 Nothing in this section shall require researchers to transfer ownership or other proprietary rights they may have.
6.8 As citizens, researchers have an obligation to cooperate with grand juries, other law enforcement agencies, and institutional officials. Conversely, researchers also have a professional duty not to divulge the identity of confidential sources of information or data developed in the course of research, whether to governmental or nongovernmental officials or bodies, even though in the present state of American law they run the risk of suffering an applicable penalty.
We should discuss the practicality of these points in detail. But it would be a grave mistake to delay the guidelines.
First, it is simply not necessary. The guidelines have inbuilt exemptions so that no journal editor will have reason to reject a paper that uses sensitive data.
Second, stopping the current guidelines can actually hinder the overall progress in reproducibility we have made in the discipline. It has taken years to agree on these guidelines. Many other disciplines have already made much more progressive suggestions in replication of published work, reproducibility and data transparency. Stopping the DA-RT guidelines now will not solve any of the issues discussed above, but it may prevent us from finding a solution to irreproducible articles and authors withholding data for no good reason.
Third, we would send a very questionable signal to the other disciplines if we abandon the new guidelines now. Interdisciplinary discussions (see Royal Society, TOP guidelines, Reproducibility Project) show that most fields agree that any scholar who wants his work to be legitimate must make the sources, procedures and data – as much as possible – available. Universities and funders in most disciplines have acknowledged that.
If details about research findings are not made available, scholars cannot evaluate and cross-check published work. Just have a look at the recent LaCour scandal in political science, where a scholar tried to cover up data fabrication in his highly influential paper.
What should we do?
We should keep the existing DA-RT guidelines intact, and work on creating further practical implementation guidelines, check-lists and ‘how-to’ manuals.
This could be a start:
- How to achieve transparency for non-statistical forms of inquiry: There are two good articles on qualitative data, “Data Access and Research Transparency in the Qualitative Tradition” and “Transparency: The Revolution in Qualitative Research“
- Who covers costs involved with preparing data for archiving (especially when raw data are not easily digitalized): Many funders now require transparency, and this can be written into the budget. Data archives are for free.
- Legal and ethical obligations to human subjects: These are exempt from the transparency rule.
- Will a data repository provide graded access, and is it safe to hacking: Data can be made available in repositories such as the Institution for Social and Policy Studies (ISPS) at Yale University, the Dataverse Network at Harvard University, and the Inter-university Consortium for Political and Social Research (ICPSR), as a Open Science Framework project page, on journal-specific Web sites and archives; or on the original author’s webpages. The Harvard Dataverse allows a setting where access to particular files can be blocked. The author can decide whom to allow to download the files. The OSF project pages also allow to distinguish between public and blocked files.
Update: Other reactions to the petition
Responding To A Petition To Nobody (Or Everybody)
Quick roundup on DA-RT, the “Data Access & Research Transparency” initiative that is shaking up Political Science
The DA-RT Petition
Political Science Replication 
[…] “petition”, so far signed by more than 600 colleagues (linked from the article) as “Political Scientists Trying to Delay Research Transparency”. John Patty, who has signed up DA-RT, gives the petition another beating: “Responding To A […]
LikeLike
[…] On November 7, 2015, the blog Political Science Replication published “Political Scientists Trying to Delay Research Transparency.” […]
LikeLike
The problem Isaac raises is not transparency. Its the neopositivism hiding behind the call for more transparency. Wake up! There is a causal relation between the rise of the DART philosophy and the current Disdain for the political science profession. See review of Clarke Primo at
https://independent.academia.edu/WilliamJKelleherPhD
William J. Kelleher, Ph.D.
LikeLike
Hi everyone –We write as some of those who first circulated the petition asking the editors for a delay. A quick clarification and then some answers to the specific concerns raised in this blog.
Before anything, the clarification. The blog reads, “…The petition states: “ [T]he decision about whether or not to make research materials publicly available should belong to scholars and not journal editors or reviewers.”
The quotation suggests that the signers of the petition took this position. But those words appear in the following context:
1) They are preceded by the introductory statement, “At this point, many key questions remain unresolved. Some of the issues raised at the Annual Meeting and in other venues include: …,” and are part of the last bullet point after that introduction. So the signers merely state that the issue has not been resolved.
2) That particular bullet point begins, “Finally, many scholars have expressed the view that the decision about whether or not… etc.” Those who signed the petition assert that many scholars have done so but not that they themselves hold that opinion.
3) The letter concludes, “The signatories below may or may not support any particular formulation of the issues among the examples we give, but all join in hoping that you will agree to this delay.”
This wording was crafted to make it clear that the signers of the petition/letter did not take a position on the highly controversial question of whether the decision to make research materials publicly available should belong to the scholars or the journal editors/reviewers.
It is a fact that many scholars have expressed this view. We believe that the existing reasons for and against each position (scholars should decide vs. journal editors and/or reviewers should decide) should be the subject of deliberation so that those who have not made up their minds on the issue can contemplate those reasons, perhaps come up with new reasons one way or another, and decide. We expect exactly this kind of reflection will take place as a result of the excellent proposal for a deliberative process being developed by Tim Büthe and Alan Jacobs at the request of the Qualitative and Multi-Method Research section.
To respond to the specific concerns raised in the blog:
First, to the idea that a delay “is simply not necessary,” we think it would be divisive to proceed when there is such consternation and puzzlement among several affected research communities about what the implementation of these as-yet-not-fully-articulated policies might mean in practice. An excellent start on a draft of “Guidelines for Data Access and Research Transparency for Qualitative Research in Political Science” was made in August 2013 by Colin Elman, who received further suggestions for changes in that draft in the summer of 2015. In addition to the Büthe-Jacobs process, these draft guidelines could provide one basis for the discussion over the coming year, with perhaps a panel or series of panels at the 2016 annual meeting dedicated to discussing the more controversial aspects. Without a delay to clarify these guidelines, we are asking the journal editors both to implement and to police policies whose implications are simply not clear.
Second, to the concern that stopping the implementation now would “hinder the overall progress” already made, we think that going forward with implementation now would actually do more to hinder the overall progress already made than would a delay. We sympathize greatly with the organizers of DA-RT, who have worked for five years to bring these issues to the profession. Yet the journal editors’ statement of October 2014 shifted the situation considerably — from a regime of self-enforcement by individual scholars as an ethical matter to a set of requirements that editors of some major journals took responsibility for policing. That is a very different kind of regime. If there is no delay, the editors will begin implementing the statement in a few months (January 15, 2016). We believe that the fallout to their doing so without better guidelines or greater clarification is likely significantly to impede the overall progress that has been made.
Third, to the concern that we would send a questionable signal to the other disciplines if we delay implementation now, we think not. Rather, we would send a signal that we are proceeding by setting up a process of informed deliberation, working carefully and with concern for the many aspects of the issues and the many different forms of research involved, to evolve processes, language, and understandings that take into account the needs and perspectives of all parts of the discipline. This is not a bad signal.
We hope that we have said enough to indicate that we are not against transparency. We appreciate greatly much of the work that has gone into this process so far. Our hope is that a delay in implementation, which we believe necessary for the reasons given above, will allow scholars to deliberate these issues during the next year in a structured and timely way.
Thanks for your consideration —
Nancy Hirschmann (U of Penn), Mala Htun (U of Arizona), Jane Mansbridge (Harvard), Kathy Thelen (MIT), Lisa Wedeen (U of Chicago), Elisabeth Wood (Yale)
LikeLike
Hi everyone –We write as some of those who first circulated the petition asking the editors for a delay. A quick clarification and then some answers to the specific concerns raised in this blog.
First of all, the clarification. The blog reads, “…The petition states: “ [T]he decision about whether or not to make research materials publicly available should belong to scholars and not journal editors or reviewers.”
The quotation suggests that the signers of the petition took this position. But those words appear in the following context:
1) They are preceded by the introductory statement, “At this point, many key questions remain unresolved. Some of the issues raised at the Annual Meeting and in other venues include: …,” and are part of the last bullet point after that introduction. So the signers merely state that the issue has not been resolved.
2) That particular bullet point begins, “Finally, many scholars have expressed the view that the decision about whether or not… etc.” Those who signed the petition assert that many scholars have done so but not that s/he holds that opinion.
3) The letter concludes, “The signatories below may or may not support any particular formulation of the issues among the examples we give, but all join in hoping that you will agree to this delay.”
This wording was crafted to make it clear that the signers of the petition/letter did not take a position on the highly controversial question of whether the decision to make research materials publicly available should belong to the scholars or the journal editors/reviewers.
It is a fact that many scholars have expressed this view. We believe that the existing reasons for and against each position (scholars should decide vs. journal editors and/or reviewers should decide) should be the subject of deliberation so that those who have not made up their minds on the issue can contemplate those reasons, perhaps come up with new reasons one way or another, and decide. We expect exactly this kind of reflection will take place as a result of the excellent proposal for a deliberative process being developed by Tim Büthe and Alan Jacobs at the request of the Qualitative and Multi-Method Research section.
To respond to the specific concerns raised in the blog:
First, to the idea that a delay “is simply not necessary,” we think it would be divisive to proceed when there is such consternation and puzzlement among several affected research communities about what the implementation of these as-yet-not-fully-articulated policies might mean in practice. An excellent start on a draft of “Guidelines for Data Access and Research Transparency for Qualitative Research in Political Science” was made in August 2013 by Colin Elman, who received further suggestions for changes in that draft in the summer of 2015. In addition to the Büthe-Jacobs process, these draft guidelines could provide one basis for the discussion over the coming year, with perhaps a panel or series of panels at the 2016 annual meeting dedicated to discussing the more controversial aspects. Without a delay to clarify these guidelines, we are asking the journal editors both to implement and to police policies whose implications are simply not clear.
Second, to the concern that stopping the implementation now would “hinder the overall progress” already made, we think that going forward with implementation now would actually do more to hinder the overall progress already made than would a delay. We sympathize greatly with the organizers of DA-RT, who have worked for five years to bring these issues to the profession. Yet the journal editors’ statement of October 2014 shifted the situation considerably — from a regime of self-enforcement by individual scholars as an ethical matter to a set of requirements that editors of some major journals took responsibility for policing. That is a very different kind of regime. If there is no delay, the editors will begin implementing the statement in a few months (January 15, 2016). We believe that the fallout to their doing so without better guidelines or greater clarification is likely significantly to impede the overall progress that has been made.
Third, to the concern that we would send a questionable signal to the other disciplines if we delay implementation now, we think not. Rather, we would send a signal that we are proceeding by setting up a process of informed deliberation, working carefully and with concern for the many aspects of the issues and the many different forms of research involved, to evolve processes, language, and understandings that take into account the needs and perspectives of all parts of the discipline. This is not a bad signal.
We hope that we have said enough to indicate that we are not against transparency. We appreciate greatly much of the work that has gone into this process so far. Our hope is that a delay in implementation, which we believe necessary for the reasons given above, will allow scholars to deliberate these issues during the next year in a structured and timely way.
Thanks for your consideration —
Nancy Hirschmann (U of Penn), Mala Htun (U of New Mexico), Jane Mansbridge (Harvard), Kathy Thelen (MIT), Lisa Wedeen (U of Chicago), Elisabeth Wood (Yale)
LikeLike
Universal experience from other disciplines (from psychology to medicine) has shown that relying on individual scholars to be transparent out of the goodness of their hearts is a massive failure. Promises to share data on request are usually a meaningless gesture, for example, whereas a journal requirement to put data in a qualified repository, well, that actually means something.
If the petitioners are worried about the application of transparency guidelines to qualitative research, the petition could just ask for DA-RT to separate out qualitative research, rather than asking for the whole thing to be subject to an indefinite delay while unspecified further discussions take place. Why not try for a more targeted petition that directly tackles the putative concerns, rather than trying to block transparency even in areas where there is no good faith argument left against moving forward?
LikeLike
[…] should be fearful. Neither DA-RT nor TOP mandate data archiving (see the useful discussion in Political Science Replication). As I note above, it would be ridiculous to insist that this be done. However, “beliefs” about […]
LikeLike
The arrogance of blog defences of DART is stunning, and kind of interesting for its combination of rhetoric and bad political analysis.
1. The rhetoric, a mixture of snark and appeals to “transparency” is designed to cut off debate by making qualitative researchers seem positively medieval.
2. Making qualitative research with elites dependent on a series of exceptions to rules promulgated in the name of transparency and apple pie simply adds another obstacle to research on elites.
3. Trying to choke off close examination of political elites seems like a rather odd project for the discipline most closely engaged in their study. Or it confirms what my friends tease me about, which is political scientists’ apparent dislike of politics.
LikeLike
[…] access to the raw data and not only to the final, processed dataset. Data access is also crucial to cross-checking for mistakes in data preparation and data analysis; we all want to avoid mistakes, but they can happen and, potentially, making data available can be […]
LikeLike
[…] that gets support from both sides of the aisle (quantitative and qualitative). Then again, here and here (and the reattack here) scholars assailed the petition and its aims. Perhaps more […]
LikeLike
[…] researchers see deep dilemmas in balancing principles of research transparency against legal and ethical obligations. For […]
LikeLike