Journal editors can enforce replication policies. Authors can decide to work transparently. Most initiatives for open science and reproducibility agree that editors and authors are are the key actors to enforce the gold standard of research integrity. However, peer-reviewers can use their leverage as well: just say you will only review an article once the author provides the data.
When I taught a session on replication at the BITSS summer institute for transparency last year, a student asked: Should we, as peer reviewers for journals, refuse to assess manuscripts where the authors do not provide data and code? I said yes – that’s a great idea to push journal editors towards enforcing transparency, and it prevents us from wasting our time on articles that can never be checked.
I had assumed that this is my personal opinion. But now a new initiative was announced in the prestigious journal Royal Society Open Science. In their article “Peer Reviewers’ Openness Initiative“, a team of authors around psychologists Richard D. Morey and Rolf Zwan has drafted a statement that scholars can sign to indicate that—from 1 January 2017—they will apply open research standards to the manuscripts they review. The main goal of the Peer Reviewers’ Openness Initiative is that reviewers can “engage authors on issues of scientific openness during the review process”, as the authors write in their paper.
The statement reads:
[A]s reviewers, starting 1 January 2017, we will not offer comprehensive review for, nor recommend the publication of, any manuscript that does not meet the following minimum requirements.
What are the minimum openness criteria?
- Data should be made publicly available: online, hosted by a reliable third party
- Stimuli and materials, e.g. survey questions, should be made publicly available, hosted by a reliable third party.
- In case some data or materials are not open, clear reasons (e.g., legal, ethical constraints, or severe impracticality) should be given why.
- Documents containing details for interpreting any files or code, and how to compile and run any software programs should be made available with the above items.
- The location of all of these files should be advertised in the manuscript, and all files should be hosted by a reliable third party.
Authors of submitted drafts can show their willingness to provide these data for transparency in a statement to the author note, saying exactly where the data are; or their justification why some of the data will not be shared.
Read the full statement at https://opennessinitiative.org/the-initiative/.
How do you tell journals that you refuse to review?
Obviously, no one encourages peer-reviewers to be aggressive or to outright decline to review. Rather, the initative suggests that you first encourage journal editors to ask authors for the data, and only then you agree to review the paper. Give them some time to do this. I think this is a good strategy to gently push authors to comply, and to slowly change norms in your discipline. Writing to the journal editors, the initiative recommends to use clear phrases such as:
I believe strongly in the value of openness and transparency. Please ask the authors on my behalf whether they can certify that they have met the standards of the Peer Reviewers’ Openness Initiative (https://opennessinitiative.org/).
I cannot recommend this paper for publication, as it does not meet the minimum quality requirements for an open scientific manuscript (see https://opennessinitiative.org/). I would be happy to review a revision of the manuscript that corrects this critical oversight.
Is the initiative asking too much of authors?
No. Journals should already enforce such minimal standards (many journals now have a replication policy and subscribe to the TOP guidelines). Authors should strive to meet such standards anyways. Working reproducibly is good for yourself and science at the same time (see here).
What I like about the initiative is that its initiators provides concrete ideas and basic guidelines for authors trying to make their drafts reproducible, so that journal editors can find willing peer-reviewers. Under each criterion for transparency, there is a link “[I’m an author; help me comply!]” that leades to tips such as where to upload your data etc.
Why should you support the initative?
By adding your name to the currently about 230 signatories, you can
- join a new trend to use your leverage as peer-reviewer
- demonstrate that you promote openness and transparency in research
- stop wasting your time on manuscripts that can not be cross-checked
- shape the debate about how to enforce quality standards in science
- force yourself to be even more diligent in your own work (anyone who signs will obviously have to follow the rules as well)
The Guardian, which also reports about the initiative, summarized the benefits nicely:
The PRO Initiative isn’t a panacea that will magically solve all of the issues surrounding open science overnight – but then again, it’s not meant to be. What it does do is provide scientists who have a passion for open science to be able to make an actual difference within their discipline at an individual level.
Read more
An article in the Guardian about the initiative: How peer reviewers might hold the key to making science more transparent
Political Science Replication 