Following an article in the New England Journal of Medicine, which portrayed scientists who re-use data as parasites, we now hear more on this from Nature. Apparently, data transparency is a menace to the public. The Nature comment “Don’t let transparency damage science” claims that the research community must protect authors from harassment by replicators. The piece further infects the discussion about openness with more absurd ideas that don’t reflect reality, and it leads the discussion backwards, not forward.
Apparently, the trend towards more transparency has left all of us researchers in despair. It’s nearly damaging all of science! The comment published in Nature yesterday starts off by listing all things bad around data sharing:
- endless information requests
- complaints to researchers’ universities
- online harassment
- distortion of scientific findings
- threats of violence
This can happen to all of us … beware!
The authors also say that “these were all recurring experiences shared by researchers from a broad range of disciplines at a Royal Society-sponsored meeting last year.” I was at a different meeting of the Royal Society – and the tone there was completely different. The overall notion at the event was that openness must be supported, and that science without transparency is – simply – not science. The Royal Society strongly supports data transparency (see here and here).
Replicators are automatically attackers?
Replicators are – throughout the whole piece – named “critics”. Asking to see data for published work does not mean that you are a critic, it means that your are a researcher. Actually, wait! Aren’t all scholars supposed to be critical and rigorous?
The article then sets out to “distinguish scrutiny from harassment”. That implies that any time a scholar receives a replication data request, he is likely to face one of two consequences: a) scrutiny of their work, and b) harassment (or both). So if you publish an article, you better not upload the data for everyone – because who knows what will happen next!
This also implies that anyone must now justify their attempt to access data, and prove that they will not harass the original author. Maybe that should become a section on all of our CV’s to be sent along with data requests: “I have replicated x published articles and all findings were confirmed.”
Absurd language enters the openness debate
Trying to persuade us of transparency’s menace to the public, the Nature comment uses a type of language that really does not belong in the discussion about data sharing (but provides for an entertaining read):
- Orchestrated and well-funded harassment campaigns against researchers
- risk making science more vulnerable to attacks
- masquerade as scientific inquiry
- Increasingly … calls for retraction are coming from people who do not like a paper’s conclusions
It may be true that some researchers, e.g. in climate change (apparently a good example for all of science), are subject to attacks by opponents. But that has nothing to do with data sharing and openness. Bringing such language and examples into the openness debate is a distortion of the discussion. Anyone working on climate change who does not share their data is automatically subject to criticism, because opponent can claim the author is ‘hiding something’. Being transparent on how conclusions were reached is exactly the right way to protect yourself from criticism. Holding back your data won’t make opponents go away. And if you really did some mistakes in the analysis, then why should that remain undiscovered? Or wait, I’m not sure I understand science anymore. What was the goal again?
The comment goes on to provide a list of red flags about researchers and red flags about the so-called critics, pretending to provide a balanced view of benefits of transparency versus withholding your data. Again, the list suggests that anyone asking for replication materials could potentially be an amateur, have a financial interest in publishing a failed replication, might plan personal attacks, hack p-values, or insist on access to confidential data.
The list is absurd, and almost comical.
If it weren’t published in Nature, and if it didn’t have the potential to lend harmful arguments to opponents of transparency, I would not have taken it seriously.
It’s also unclear who the critics are supposed to be. Do they have a PhD? Or is it only Dr A. who is qualified to analyse his/her data?
There is also a list how scholarly communication can be used as a “weapon”.
Transparency does not damage science – people do.
I have no problem with voicing concerns about internet trolls, anonymous personal attacks against researchers etc. But these problems are not caused or aggravated by transparency. The story simply does not work. Worse still, articles like this Nature piece could undermine a healthy trend in the social and natural sciences which aims to improve research integrity and quality of output. The Nature comment was titled “Don’t let transparency damage science”. Transparency does not damage science – people do.
Political Science Replication 
[…] Read more at https://politicalsciencereplication.wordpress.com/2016/01/29/getting-the-idea-of-transparency-all-wr… […]
LikeLike
While I agree with the author that the title of the piece was misleading –it is indeed people rather than transparency itself that is damaging – it is concerning to see our arguments being either dismissed or misrepresented, when we are trying to encourage debate about a very serious issue – scientists coming under concerted attack by those who object to their conclusions regardless of the evidence. This can be damaging to the scientists themselves and can put people off researching particular topics. It would appear this does not happen in political science, but it certainly does occur in other areas, and in our article we cite testimonies of some of those who have experienced it.
The author starts by describing how she thinks our piece does not reflect the experience of most scientists. Fortunately, she is right. Few people work in controversial areas. We are not claiming this is a common problem. At our meeting we deliberately involved people who work in areas where scientists have recurring experiences of being targeted. Our point was that in these areas, the strategies used to attack people are similar, even though the subject matter is very different. The costs for the individuals who have been subject to attack are, however, very serious. The fact it is not a common experience explains why it has not been adequately tackled.
The critique becomes absurd with the statement “Replicators are – throughout the whole piece – named ‘critics’.” No. Critics are referred to as critics, and they engage in many behaviours, only one of which relates to reanalysing data. It’s a major logical flaw to treat our statement that ‘requests for data can be used to harrass’ as equivalent to ‘requests for data amount to harassment’. Of course they don’t. But there are clear instances where this has occurred – perhaps the most memorable of which is when creationists repeatedly harassed Richard Lenski for his data: see http://www.conservapedia.com/Conservapedia:Lenski_dialog (the 2nd reply is priceless).
As we note at the outset, both I and my co-author are on record as being strongly in favour of open science, transparency, and having data available for re-analysis. I have myself on several occasions requested original data from authors.
So to be clear: I thoroughly approve of the efforts of @PolSciReplicate to do replication studies and think it’s great that she is spearheading this initiative. But she reveals a degree of naivety in denying the existence of groups who will try to subvert science for their own ends. She regards our list of ‘red flags’ as almost comical – they aren’t so funny for those who have been on the receiving end of these behaviours.
LikeLiked by 1 person
Dorothy Bishop states that her piece was intended to encourage debate on “scientists coming under concerted attack by those who object to their conclusions regardless of the evidence”. Yet in her commentary, Chronic Fatigue Syndrome (CFS) was one of the areas where it was claimed that “hard-line opponents” employed identical strategies to the orchestrated harassment campaigns used against researchers working on climate change and tobacco control.
The report of the Royal Society meeting on this topic arranged Bishop and Lewandowsky was divided in to ‘Endogenous Threats’ to science from poor practice by researchers, and ‘Exogneous Threats’ to science from those outside of science. It was decided to place two reports from CFS researchers in the ‘Exogenous Threat’s section, although it is not clear how this decision was made, and any examination of the work being described by these researchers would quickly reveal that they should have been included in the ‘Endogenous Threats’ section. Did anyone think to check? CFS is a condition surrounded by prejudice, particularly within the UK medical research community, and it would not surprise me if no-one did think to check whether the patients making complaints or freedom of information requests were threats to science, or were working hard to help improve the quality of research into their condition, and draw attention to problems which were having a harmful impact upon their lives.
Since the Royal Society meeting took place a number of researchers have been looking at the concerns expressed by CFS patients about some researchers, and then speaking out about the problems they have found for themselves. James Coyne recently stated: “it’s really frustrating to me that many of the things that I’m saying that are taken so seriously are things that patients have said before and they weren’t taken seriously. And that’s something we’ve got to change. It pisses me off.” phoenixrising.me/wp-content/uploads/A-skeptical-look-at-PACE-16-11-2015-edited-transcript.pdf
David Tuller has published a series of investigative pieces into CFS research, and introduced his articles on the PACE trial (one of the pieces of research for which patient’s complaints were presented as an ‘Exogeous Threat’ to science) with comments from others:
“Top researchers who have reviewed the study say it is fraught with indefensible methodological problems. Here is a sampling of their comments:
Dr. Bruce Levin, Columbia University: “To let participants know that interventions have been selected by a government committee ‘based on the best available evidence’ strikes me as the height of clinical trial amateurism.”
Dr. Ronald Davis, Stanford University: “I’m shocked that the Lancet published it…The PACE study has so many flaws and there are so many questions you’d want to ask about it that I don’t understand how it got through any kind of peer review.”
Dr. Arthur Reingold, University of California, Berkeley: “Under the circumstances, an independent review of the trial conducted by experts not involved in the design or conduct of the study would seem to be very much in order.”
Dr. Jonathan Edwards, University College London: “It’s a mass of un-interpretability to me…All the issues with the trial are extremely worrying, making interpretation of the clinical significance of the findings more or less impossible.”
Dr. Leonard Jason, DePaul University: “The PACE authors should have reduced the kind of blatant methodological lapses that can impugn the credibility of the research, such as having overlapping recovery and entry/disability criteria.”
http://www.virology.ws/2015/10/21/trial-by-error-i/
I apologise for including quotes from authority rather than focusing the arguments and evidence but I’m afraid that I have seen how many people working in science are unwilling to engage with patients’ concerns about the work of their colleagues.
To briefly look at the CFS submissions published as a part of Bishop and Lewanky’s conference: Prof White discusses the PACE trial, the most controversial piece of CFS research, and the one mentioned in the Nature commentary. Even before the Royal Society conference an evidence review for the Agency for Healthcare Research and Quality had criticised the PACE trial’s post-hoc recovery criteria, and described them as “contradictory”: http://www.effectivehealthcare.ahrq.gov/ehc/products/586/2004/chronic-fatigue-report-141209.pdf
The misleading claims about ‘recovery’ from the PACE trial has been a key area of concern since the first paper led to the trial’s researchers claiming that 28/30% of patients had got “back to normal” following CBT/GET, and a commentary in the Lancet reviewed and approaved by the PACE trial researchers had described these patients as fulfilling a “strict criterion for recovery”. In fact, patients could have reported declines from baseline for all outcomes, and yet still fulfilled this criteria for ‘recovery’. I think that patients’ anger about this was justified.
A FOI request was made for results for the recovery criteria laid out in the PACE trial’s protocol, but this was denied, as it was claimed that these results were due to be published in a peer reviewed paper:
http://www.meassociation.org.uk/2011/05/6171/
http://www.meassociation.org.uk/wp-content/uploads/2011/06/FOI+from+Queen+Mary.pdf
When these results did not emerge, a new request was made and responded to thus: “The information you have requested is not held. The requested data relating to the recovery rates and positive outcomes do not exist. That is to say that such analyses have not been done and there is no intention to do so.”
https://www.whatdotheyknow.com/request/pace_trial_recovery_rates_and_po
A request for the data to be calculated and then released was met by the claim that it would be too expensive as “there is no longer a statistician employed by the PACE trial, one would need to be recruited for this operation and trained.”
https://www.whatdotheyknow.com/request/pace_trial_recovery_rates_and_po_2
A request for the annonymised data that would allow for the calculation of results for the outcomes laid out in the trial’s protocol was then made: https://www.whatdotheyknow.com/request/selected_data_on_pace_trial_part
Although denied by QMUL and the PACE trial researchers, the Information Commisioners Office went on to order the release of this data. QMUL and PACE are currently appealing this decision. In his submission on PACE Prof White stated of FOI requests that “These requests and appeals took up a considerable amount of time to address, which meant that some secondary papers were delayed.” I do not think that anyone should have much sympathy for him here. He went on to claim that: “Our deliberate policy, to help allay concerns about the trial, was to be as transparent as possible regarding what we did, while also protecting medical confidentiality and our staff and patient supporters; this included publishing the protocol and the statistical analysis plan, and paying for open access to all publications.” It is difficult to believe that White was trying to be honest here.
The history of spinning results and failing to release data is what led to other researchers taking the time to request data from the PACE trial. Bishop and Slewandowsky summarised the controversy thus: “We suspect that explicit discussion of what data are and are not available as part of the original publication process might have averted some of the ongoing controversy surrounding the PACE clinical trial, a UK study on chronic fatigue syndrome. The issue involves requests for data by transparency advocates, and the refusal by researchers and institutions to release data citing patient confidentiality, limited consent and requestors’ intent.” They have rather missed the key points.
The other CFS submission came from Esther Crawley, a researcher who went even further in exaggerating recovery rates from the PACE trial, claiming in one paper that the “trial of cognitive behavioural therapy and graded exercise therapy indicated a recovery rate of 30-40% one year after treatment”. http://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-11-217
She complained that she’d had FOI requests related to the SMILE trial and complaints about her had been made. The SMILE trial is designed to assess the Lightning Process for children with CFS. The Skeptic’s Dictionary has an article on LP here: http://skepdic.com/lightningprocess.html and the LP website is here: http://lightningprocess.com/ . This would be the first trial run for LP, it is founded upon known quackery such as osteopathy, and makes a number of inaccurate medical claims within its’ materials. I think that concern about choosing to use limited research funds for a trial on children, particularly when no earlier trial had been conducted on adults to assess safety and it is questionable whether they had been properly informed about the nature of the intervention before being asked to give consent, are entirely legitimate.
Both PACE and SMILE are non-blinded trials using subjective self-report measures as primary outcomes. Homeopaths are not allowed to get away with claiming such trials provide reliable data on the efficacy of their intervention, so should our standards be lower for biopsychosocial interventions for CFS? Homeopaths also complain about facing harassment from militant skeptics – why were they not invited to give submissions to the Royal Society on the ‘Exogenous Threat’ that they face?
Both submissions noted that CFS campaigners were concerned about PACE and FINE before results had been collected or released. This is not patients objecting “to [researchers] conclusions regardless of the evidence”, but patients objecting to poorly designed research that affects their lives. Complaints about these trials, and attempts to get further information on them, should not be seen as ‘Exogenous Threats’ to science, but as patients expressing legitimate concerns about the research being conducted into their condition.
In his comment here Slewandowsky states “The good news embedded in this blog post, as well as in the reaction on Twitter by some members of the CFS/ME community, is that our commentary clearly serves to differentiate between those who endorse open data based on reasoned argument, and those whose motivations lie elsewhere.” This is a bold claim for someone who had two days earlier stated of the controversy around the PACE trial that “we are not intimately familiar with the details”.
I have not seen Slewandowsky even attempt to engage in reasoned argument with those patients concerned that his piece further perpetuates the prejudices which surround their condition. I think that their work is harmful to CFS patients, and all those who want to improve the quality of research in this area, even if that was not their intent and they are not sufficiently well informed to be able to realise it.
The decision to present the campaigning and controversies around PACE and SMILE as endogenous threats to science is something which should be defended through reasoned argument, or apologised for. Simply pointing uncomprehendingly to the anger and pain of patients may be justification enough for some researchers to dismiss their concerns, but such snobbery cannot be combined with appeals to reasoned argument.
LikeLiked by 3 people
From the nature article: “All who participate in post-publication review should identify themselves. The drawbacks of anonymity (its encouragement of bad behaviour) outweigh its advantages (for example, it allows junior people to criticize senior academics without fear of redress)”
I am “anonymous” and i may have spotted an error in the nature article. Could it be that reference no. 1 should have been “Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). Doi: 10.1126/science.aac4716” ?
LikeLiked by 1 person
I think the reason people are understanding it as being primarily a warning against data seekers is easy to understand. You gave a chart which you explained on Twitter as showing potential “red flags” for either a researcher or a data requester.
However in your fairly long article, nearly all the text was devoted to examples of how data requests could go awry. There was really no substantial discussion of how research could do awry or how post-publication review could play a role in checking for errors.
Then in your specific examples, you included patients* and researchers for a particular medical condition lumped in the apparently “bad” examples with companies that had a financial interest in selling a particular type of product (tobacco industry), and other people who are generally thought to not have good science skills (people who think climate change is not caused by human activity).
This is in the first place illogical, because the patients (and researchers who are working in other areas, or researchers who are working on a diverse set of solutions) have no interest other than good science and treatment that actually works “where the rubber meets the road”. It is in the second place a logical fallacy of “guilt by association.”
*Since the patient population is substantial, it is reasonable to expect that some patients would, in their non-patient lives whether currently or previously, would actually have researcher credentials or other appropriate training and skills to be able to evaluate methods and datasets.
You presented no evidence at all, that there was some kind of misconduct or inappropriate agenda by the named group of patients (and researchers who don’t support PACE’s conclusions, or want them evaluated independently before taking a position), as you imply.
If the research is good, it should stand. If it is bad, it should fall. Anyone who has tied their reputations to this or related research could have their reputations affected, but that is beside the point of getting good research for public policy so patients can get useful treatments and better research. That much should be obvious.
At the risk of sounding cliche, anyone who is seeing something different from patients (and researchers who care about wellbeing of patients) trying to get appropriate treatment and useful public policy, needs to check their privilege.
LikeLiked by 1 person
“it is concerning to see our arguments being either dismissed or misrepresented”
Then maybe you should have kept them secret, dear.
I promise we’ll all stop making fun of the philistine misosophical musings of ‘you and your co-author’ if you keep them to yourselves.
LikeLiked by 3 people
Dorothy:
“we are trying to encourage debate about a very serious issue – scientists coming under concerted attack by those who object to their conclusions”
Concerted attack? You mean they’re being plotted against?
“denying the existence of groups who will try to subvert science”
Groups trying to subvert science?
While remote diagnosis is of course meaningless, your comments do appear to suggest conspiracist ideation. Perhaps you picked up this manner of thinking from extended contact with Prof. Lewandowsky, a renowned practitioner of the Paranoid Style in American [Scientific] Literature.
Not to say that there IS no group of human beings that want, and actively work, to subvert science.
Such a set certainly exists. It’s just that:
1) it’s a rather small set
2) it’s hamfistedly obvious in its intentions, rather than secret
3) Professor Lewandowsky is a member. Whether or not you meant to, you became one too the moment you co-authored this toxic misosophical tract with him.
LikeLiked by 2 people
I have to side with Dorothy on this one. I read the Nature piece before reading your critique, and certainly didn’t come away with the impression that she and Stephan Lewandowsky were talking about replicators in particular, if at all. The word “replication” shows up in the piece exactly once. Nor did I come away with the impression that the piece was any kind of attack on open science. And, as she points out above (a bit understatedly maybe), Dorothy Bishop is one of the most well-respected and tireless advocates the open science movement has, so the suggestion that she’s against transparency or data sharing is not tenable. The problems discussed in the piece may be uncommon, but they’re important ones that undoubtedly do arise, and should be taken seriously.
LikeLiked by 2 people
Please check out my detailed response to Bishop and Lewandowsky
https://jcoynester.wordpress.com/2016/01/31/further-insights-into-the-war-against-data-sharing-the-science-media-centres-letter-writing-campaign-to-uk-parliament/
LikeLike
“The word “replication” shows up in the piece exactly once. ”
It doesn’t play a big part in the day-to-day life of a “scientist” (if I can use the word so loosely) of Lewandowsky’s calibre, so why would you expect him to mention it at length?
LikeLiked by 1 person
Thank you Dorothy for your comment. I know and respect your work on transparency and research integrity. This is why I was very surprised to see you as a co-author of the Nature comment. I don’t dispute that in some instances, e.g. climate research, researchers are under attack and need to know about tools to protect themselves. But that would be a different article, and it does not fit in the transparency debate. I still stand by what I wrote earlier. The headline and the general approach of the piece, which suggest that transparency can be dangerous, does not fit the points you try to make. It is catchy, for sure, but it discredits the transparency initiatives that you have supported in the past.
LikeLiked by 1 person
Could you please specify in what way researchers in climate science are “under attack”? The point is important, because Professor Lewandowsky has consistently referred to all criticisms of his papers on the psychology of climate scepticism as “harassment”, comparing critics to people who send him anonymous anti-semitic emails.
I was one of the critics accused in a paper by Lewandowsky of various character defects, including feelings of persecution and an inability to reason, and one of a small number of people whose “cogent and well-argued” letters to the journal got the paper retracted. The paper has since been republished in a slightly altered form, with the identities of the people accused of psychological defects still identifiable by anyone who can use a search engine. I asked Lewandowsky for the data for this new paper but he hasn’t replied. The Bristol University website states that the data is not available.
Your unsupported statement that climate science is “under attack” will only encourage Lewandowsky and his colleagues to continue in his campaign to conflate the various forms of scepticism about climate science with the sinister machinations of creationists, and conspiracy theorists, and the paid shills of Big Oil.
LikeLiked by 1 person
Dorothy Bishop has already pointed to the fundamental logical flaw in this post.
The author of the post has also regrettably resisted the temptation to acquire much familiarity with what we actually wrote.
At a serious level, this lack of care has her caricaturing our position as “Trying to persuade us of transparency’s menace to the public”, when in fact we said in our commentary that “The progress of research demands transparency”; that “We strongly support open data, and scientists should not regard all requests for data as harassment” and that “the right to privacy should not extend to funding arrangements. Researchers should scrupulously disclose all sources of funding; even small undisclosed amounts can create an impression of undue influence.”
At a more comedic level, the author admonishes us that, in presumed contrast to our position, “The Royal Society strongly supports data transparency (see here and here).” Remarkably, the first “here” links to a paper on a peer-review openness initiative on which I am a co-author.
This is perhaps the first time that something I never said has been attacked by citation to one of my own papers.
The good news embedded in this blog post, as well as in the reaction on Twitter by some members of the CFS/ME community, is that our commentary clearly serves to differentiate between those who endorse open data based on reasoned argument, and those whose motivations lie elsewhere.
LikeLiked by 2 people
Stephan, may I quickly express the gratitude of the overwhelming silent majority for the groundbreaking work you’re doing in the ecneics field. If they ever endow a Nobel for Ecneics, you’re a shoe-in for the inaugural laurel.
Since fleeing Australia, you’ve told UWA Professor Carmen Lawrence that you were “still being pursued by climate deniers.” Have you managed to get a look at your pursuers yet, or a partial licence plate? Or any other information that might help the police catch this group of people who clearly have an unhealthy fixation on you?
LikeLiked by 3 people
I think most everyone can agree with two statements (though we might disagree on what events fall into the two categories):
* First, requests for transparency have in the past — in certain kinds of controversial research areas — been used in bad faith as a bludgeon against researchers.
* Second, requests for transparency that have been entirely in good faith have been dismissed as being in *bad faith*.
Assenting to the truth of these two statements is all that is necessary to see the value of the Steve and Dorothy’s Nature commentary. We need a thoughtful discussion of these issues in the context of a pro-openness agenda. Steve and Dorothy provided that; as they said in the piece, and as their other work makes clear, they are both pro-openness.
Their “mistake” — if it can be called that — is mentioning several examples. Examples are necessary in writing, because you cannot stick with only abstract ideas. But any examples you come up with are going to elicit disagreement as to which of the categories above it belongs in, making people angry.
But that’s just their point, isn’t it?
LikeLike
Richard D. Morey says:
January 30, 2016 at 1:17 pm
I never agree with such vague statements. They have no meaning, they are just anecdotes. Did it happen once? Probably. Would it be meaningful if it did happen once? No. How does one distinguish between a “request for transparency” which is or is not a bludgeon.
I also don’t understand how a request for my data or my methods could be a “bludgeon against me”. They ask, I give them the data. They ask, I explain the method in greater detail … where’s the bludgeon?
Finally, if you don’t want your work product to be subject to FOIA, the answer is simple—don’t do the work using the money from myself and other taxpayers.
In any case, Richard, If you could provide actual examples of this claimed behavior it would move the discussion forwards. For example, are you claiming that the people making the FOI requests to the University of East Anglia (as revealed in Climategate) were simply trying to “bludgeon” Phil Jones? Phil certainly thought so … but do you think so?
Many thanks,
w.
LikeLiked by 3 people
‘I never agree with such vague statements. They have no meaning, they are just anecdotes. Did it happen once? Probably. Would it be meaningful if it did happen once? No. How does one distinguish between a “request for transparency” which is or is not a bludgeon.’
Bingo. The statements in question are firmly in the Not Even Wrong genre of emotivist truth so precious to those on the other side of the Sokal chasm. I despair of ever completing construction of the bridge thereacross. It’s just too big an engineering challenge.
LikeLiked by 1 person
Hi, I think there is something else to consider. Harassment is not best defined by negative feelings. Harassment is defined by a power imbalance.
http://hillsdaleschools.com/Page/62
Otherwise it’s a discussion, conflict, etc.
http://adrfaq.com/3.html
Rhetorical question, how is the power balanced here:
1) (Pro-GET side) Certain researchers with paid positions in well-respected universities, usual channel access to publishing in peer-reviewed journals, and positions on the board of the Science Media Centre, etc.
2) (Those asking for an independent review of GET studies): Patients, most of whom have lost their jobs due to disability even those of whom may have used to work (or study) in science or technical fields, and many of whom are homebound and may require (but may not have access to, or may in some cases be dependent on the actual people represented in #1 for access to) home help, assistive devices and technology, blue badges, and so forth.
Lately added researchers who may not have a steady source of income to study ME/CFS, because public funding has traditionally gone towards CBT/GET studies, and universities tend not to support biomedical research of the condition because relatively little such research exists or is funded (enough to conclude, in the words of a Harvard doctor, that the debate over the nature of the condition has been settled that it is medical, since 1999, and for the IOM study to reiterate this in 2015, but not enough to support diagnostic procedures or any treatments which can be broadly recommended–CBT/GET excluded since no proper blinded studies have been done and PACE and FINE are the main large enough studies and they both failed to show any real-life or long-term benefit).
Something to think about….
LikeLiked by 1 person
But don’t forget that a researcher with a guilty conscience and something to hide is on the losing side of science, and knows it. To him or her, even the lowliest muggle with an FOI form is an existential career threat.
Not naming any names, of course.
LikeLike
Let me first thank you for expressing your concern about this article in a reasonable way. This is an important issue and requires level-headed discussion. People have mostly said what I think but let me just add a few things:
I think you are misrepresenting the point of the piece by Lewandowsky and Bishop. They are clearly not arguing against transparency and openness. Given what a strong proponent of open science Dorothy Bishop it seems obvious that this interpretation of their article is wrong. But every major social change comes with problems and turmoil and it is important to discuss this. Nobody here is arguing that data should *not* be open – we all agree that it must be. But we do need a discussion of the human issues surrounding this.
The social media reaction to this article is a perfect example of how things can go horribly wrong. Lashing out against anyone who disagrees with you and slandering them as ‘attacking patients’ or ‘defending’ their enemy (aka the George W Bush War on Terror doctrine) is counterproductive. By doing so they hurt open science as it only confirms the fears of many scientists who remain skeptical of open science. (From my own experience I think there are still a large number). Above all even hurts their own cause as to neutral observers these people appear as bullies who cannot be taken seriously.
LikeLiked by 1 person
“They are clearly not arguing against transparency and openness.”
Yes. Yes they are.
“Nobody here is arguing that data should *not* be open…”
Yes. Yes they are.
“… we do need a discussion of the human issues surrounding this”
Now you are.
They are not subject to “human issues”. The whole point of the scientific method is that it is contrary to human nature, human preferences. If you cannot accept that then I am sorry but you are not a scientist. You might have a research post, you might have a title. But science must be open to challenge and must be capable of being reproduced. If you are not open to this then what you are doing is not science, and you are not a scientist.
Lewandowsky is sounding like the freedom of speech opponents masquerading as supporters, who says that such freedom is only for those with “good” views (i.e. views whose expression need not be protected, as no-one with any power wishes to silence them). He is saying, in effect, that researchers should only be expected to be open in their science with those with whom they wish to be open. That is meaningless. I think you need to rethink the meaning of “transparency” and “openness”.
LikeLiked by 1 person
Here is the text of my comment that was deleted at Nature:
——————————————————————–
Perhaps a follow up article could be written: Academic Integrity – Don’t let Activist Academics Damage Science – I’d recommend Prof Lee Jussim.
Prof Lewandowsky has experienced a number of calls for retraction of I think 4 of his papers. Frontiers In Psychology did in fact retract one paper, and it is clear from the original retraction statement that they tried to let the authors go gently. When Prof Lewandowsky went to press saying the journal gave in to bullying and harassment, the journal responded twice saying this was not the case. And Prof Markram the co-founder of the journal went as far to publicly state that Professor Lewandowsky and his co-authors (including John Cook) actions were – to quote – “activism that abuses science as a weapon”
———————————————————————
Here are references:
to the original retraction notice,
http://journal.frontiersin.org/article/10.3389/fpsyg.2014.00293/full
and the 2 subsequent refutations by Frontiers of Professor Lewandowsky’s cliams that the journal gave into bullying and harrassment.
http://www.frontiersin.org/blog/Retraction_of_Recursive_Fury_A_Statement/812
http://www.frontiersin.org/blog/Rights_of_Human_Subjects_in_Scientific_Papers/830
The quote of Markrams – where he describes Lewandowsky’s actions (that resulted in the retraction) as – “activism that abuses science as a weapon”, is from a personal comment of Prof Markram, made in the comments of the last link.
LikeLiked by 6 people
Dr Paul Matthews (Nottingham University) discusses the Nature article, which may also be of interest.
LikeLiked by 2 people
The Nature piece is silly. It is also ironic. Lewandowski has a reputation for intransparency, and Bishop played a small but key role in the harassing of Tim Hunt.
Harassment and transparency are largely unrelated. If your work is prominent but unpopular in some circles, they will come after you, whether you share your data or not. If they want to beat you, they will find a stick.
The Nature article seems to say that you should only discuss your work with those who agree with you. Rubbish. If your work speaks to a wide variety of people, then the onus is on you to defend your work to all of them. If you don’t like the rough and tumble, then there is always the option to research something that only 7 other people, who are much like you, care about.
LikeLiked by 4 people
“The Nature article seems to say that you should only discuss your work with those who agree with you.”
It baffles me how anyone can read it that way. This is certainly how that “Research Parasites” editorial sounded (they probably didn’t mean it that way either but it definitely sounded that way) but this is distinctly not what I’m getting from the Nature piece. All they are saying is that scientific discussions should be conducted in a reasonable way and harassment must be stopped.
I actually agree with you that transparency and harassment are not necessarily directly connected. But if the data are shared with anyone (as I think we all agree it should) then there is the potential that large numbers of people sift through and cherry-picking things to support their agenda or criticise some minor, inconsequential flaws we know every study inevitably has. Of course they should be able to do so but the volume of such reanalyses could easily overwhelm and paralyse a researcher. And I think this is precisely what a lot of people are concerned about.
LikeLiked by 1 person
I think that concern is overblown. The majority of papers is never cited, so I would not worry about large numbers of people looking at the data. Only a handful of paper would command sufficient interest to combed through by more than a few — and those papers deserve that scrutiny.
LikeLiked by 1 person
Oh I completely agree. But that’s why we are talking about research on controversial topics. It is only going to be a handful but it can be pretty bad. And if so that would drive people away from doing research on topics that could really need it.
LikeLike
Sam: I don’t know about you. I work in climate. It can be pretty bad. A friend recently had his grandchildren threatened. This is not for the faint of heart. And some people walk away from climate research for exactly this reason. That’s sad.
But the solution suggested by Lew & Bish is wrong. Differentiate transparency along political lines will not avoid these issues. In fact, at least for climate, it made things worse.
LikeLiked by 1 person
Richard: That’s also not what they are saying. What exactly about their article gives you that impression?
LikeLiked by 1 person
Sam: I am not the only one who has that impression. Our host Nicole, for instance, writes that the Nature paper implies that “anyone must now justify their attempt to access data”. Lew practises what he preaches: His data is available to “credentialed scholars” only. Even if well-intentioned, these sort of requirements easily turn into political screening.
And indeed, that is what we have seen Mann do: Code is available to a trusted circle only. That is what we have seen Jones do: Data is not available to people who are suspected to disagree. That is what we have seen Cook do: Data are placed out of reach and those who ask for it are attacked.
LikeLiked by 1 person
This might be a good illustration of the issue. For those who don’t know, Cook et al. is a paper about consensus in climate science. They downloaded around 12000 abstracts from Web of Science and rated them according to whether they endorsed the consensus, rejected the consensus, or took no position. They used volunteers to do the ratings. Each abstract was rated twice and then a reconciliation procedure was used if the two ratings were different. Their analysis suggested that 97% of those abstracts that took a position with respect to anthropogenic global warming, endorsed the consensus position. They also emailed the authors and asked them to rate their papers (not their abstracts) and the author responses also returned a 97% consensus.
On this website one can download details about the papers, the author ratings, the individual ratings, the reconciled ratings; everything that would be required to replicate/evaluate their results. As far as I’m aware, this information has been available since the paper was published – or very soon afterwards. Furthermore, one could simply redo the Web of Science search to extract the abstracts again. Replicating the Cook et al. would simply take some effort, but has always been possible.
Despite that Richard Tol continues to claim that John Cook is not releasing data. In my view this is a classic example of what the Nature article was trying to illustrate; someone who continues to claim that data is being hidden despite it being patently obvious that all that would be required to replicate/evaluate the original result has always been available.
LikeLiked by 1 person
Ken Rice: As you know, many data are omitted from that site, including crucial variables such as time stamps and rater IDs. Also missing are the study protocol (which is not the same as the survey protocol) and the lab notes.
Time stamps are important to test for fatigue. Recall that the raters performed many tasks in a short time.
Time stamps would also reveal whether there are systematic differences between the three periods of data collection.
Rater IDs are important to test for rater bias.
The study protocol is important as categorization seems to have changed during data collection, and perhaps in response to data collected.
Lab notes are important to explain the discrepancies in the sample size.
LikeLiked by 1 person
Richard,
As you should know, to evaluate, or replicate, the results of Cook et al. you simply need their data, which is both available through Web of Science directly, or – as I’ve illustrated – through a website set up by the authors. You’re not actually asking for data that is really relevant to the results, you’re asking for data (that may not even exist) that would allow you to criticise the procedure.
Given that all the ratings for all the abstracts are actually available, it is entirely possible for you (or anyone) to go through the ratings directly. Whether someone was tired, or not, when they did a rating is largely irrelevant.
There is no real evidence for a discrepancy in the sample size. There is, however, evidence that you don’t understand how databases work, and don’t know how to use Web of Science.
It also appears as though you seem to think that Cook et al. was some kind of survey of the raters, rather than what it actually was, a survey of the abstracts.
However, I do think that our discussion is a good illustration of what Dorothy Bishop and Stephan Lewandowsky were trying to highlight in their nature article; the continual requests for more data, despite it being patently clear that all relevant data was available.
LikeLiked by 1 person
Richard: Thanks for clarifying. I don’t know Lewandowsky nor do I know his work very well. But given that this piece is also co-authored by Dorothy Bishop I think your interpretation of the piece is wrong. Dorothy has been a long-term vocal proponent of open science, of public data rather than mere data sharing in which case the question of who can access the data becomes irrelevant. I read their red flag list as a guide to outsiders for judging whether or not a criticism of some piece of research is valid (a judgement that will always be subjective).
LikeLiked by 1 person
Sam: I guess we agree that access to data should never depend on the (presumed) political outlook of the one who requests access.
LikeLike
Ken: I know you are fond of repeating moves. I am not.
LikeLiked by 1 person
Richard,
What? You’ve repeated the same claims and over again.
LikeLiked by 1 person
It seems that some people really don’t like the rough and tumble.
LikeLike
John Cook’s consensus paper was, and still is multiply fraudulent as described here:
http://www.joseduarte.com/blog/cooking-stove-use-housing-associations-white-males-and-the-97
From the article:
“The Cook et al. (2013) 97% paper included a bunch of psychology studies, marketing papers, and surveys of the general public as scientific endorsement of anthropogenic climate change.”
“Let’s walk through that sentence again. The Cook et al 97% paper included a bunch of psychology studies, marketing papers, and surveys of the general public as scientific endorsement of anthropogenic climate change. This study was multiply fraudulent and multiply invalid already – e.g their false claim that the raters were blind to the identities of the authors of the papers they were rating, absolutely crucial for a subjective rating study. (They maliciously and gleefully revealed “skeptic” climate science authors to each other in an online forum, as well as other authors. Since they were random people working at home, they could simply google the titles of papers and see the authors, making blindness impossible to enforce or claim to begin with. This all invalidates a subjective rater study.) But I was blindsided by the inclusion of non-climate papers. I found several of these in ten minutes with their database – there will be more such papers for those who search longer. I’m not willing to spend a lot of time with their data – invalid or fraudulent studies should simply be retracted, because they have no standing.”
Here is the Skeptical Science forum, where the raters are collaborating with each other on their ratings.
http://www.hi-izuru.org/forum/The%20Consensus%20Project/2012-02-27-Official%20TCP%20Guidelines%20%28all%20discussion%20of%20grey%20areas,%20disputed%20papers,%20clarifications%20goes%20here%29.html
“The inclusion of non-climate papers directly contradicts their stated exclusion criteria. The Not Climate Related category was supposed to include “Social science, education, research about people’s views on climate.”
“The authors’ claim to have excluded these unrelated papers was false, and they should be investigated for fraud.”
The authors claimed that the raters were blind to the authors of the papers they rated.
“Abstracts were randomly distributed via a web-based system to raters with only the title and abstract visible. All other information such as author names and affiliations, journal and publishing date were hidden.”
“The rater openly outed the author of one the papers to all the other raters. He was never rebuked, and everyone carried on as if fraud didn’t just happen, as if the protocol hadn’t been explicitly violated on its most crucial feature (in addition to rater independence.)”
“Cook helpfully adds “FYI, here are all papers in our database by the author Wayne Evans:”
LikeLiked by 1 person
Chad,
I suspect the the author of this blog would rather we didn’t start another debate about Cook et al. (2013). However, I think you may have helpfully illustrated some of what Bishop & Lewandowsky were trying to highlight.
LikeLiked by 1 person
“However, I think you may have helpfully illustrated some of what Bishop & Lewandowsky were trying to highlight.”
Boom-tish.
Hey, here’s an idea, Dr Rice: if the authors want to whine about the unbearable injustice of Jose Duarte’s evisceration of Lewandowsky’s henchboy’s “paper,” let them mention it themselves. It’s boring the way you keep running post-facto interference for these inept clowns who apparently can never seem to express their thoughts as clearly as you.
LikeLiked by 2 people
[…] Updates: I will add in links to other blogs here. Getting the idea of transparency all wrong […]
LikeLike
[…] “Don’t let transparency damage science,” say Stephan Lewandowsky and Dorothy Bishop in Nature, urging ways to “distinguish scrutiny from harassment.” More from Lewandowsky here, and a response from Nicole Janz, on which Lewandowsky and Bishop comment, here. […]
LikeLike
I would characterize Lewandowsky’s oeuvre as serial abuse of the common precepts of scientific data exchange and interaction.
All data requests regarding controversial papers published by him are classified by him as harassment or ‘motivated’ requisitioning of data – what ever that means.
Wherever possible, Lewandowsky has refused to release or provide access to crucial data/information that could result in exposure of flaws in his own work, citing non-existent, irrelevant or inapplicable academic constraints.
His interactions with journal editors and replying authors are ring-fenced by confidentiality agreements, presence of legal teams and outright legal threats from his associates
Every instance where data or information has been pried loose from his grasp, the resulting picture has been damaging to his claims, and not his critics. These have resulted in retraction of a paper, re-submission of the same data in a different paper, university inquiries into ethics approval of his projects, possible back-dating of blog posts, and so forth
With the article, Lewandowsky has partnered with Bishop in formalizing his data habits and sleights of rhetoric into a piece for Nature. If the broader scientific community, which does not know or appreciate the details behind his work in climate change, imbibes his message, it would be a significant setback. This is precisely his objective: to spread a miasma of suspicion and confusion about science’s critics and to provide intellectual tools for self-justifiable evasion for conduct of open science.
This is evident in the author’s own stance as she grants that when it comes to climate science its critics must somehow be morally corrupt attackers of scientists who can simply be cast out without consideration. ‘oh, the deniers, they are clearly evil’.
The numerous examples used by Lewandowsky and Bishop are puzzling and contradict each other because they are an attempt to smuggle through and mask the Lewandowskian approach to data sharing, whose precepts are fundamentally disagreeable to scientists.
LikeLiked by 2 people
I think most agree that the title of the article was poor, transparency doesn’t damage science, it is a crucial part of how we should be conducting ourselves. The complexity, in my view, and what I think the article was getting at, is what defines being transparent. As far as I’m concerned, if it is possible to reproduce what another study has done, then that is transparent. Also, by “possible” I do mean actually possible, not theoretically possible. People can’t be expected to build a large hadron collider, or launch a space telescope. People wanting to check/reproduce some other result also shouldn’t be expected to rewrite a massive code that would require a super-computer. On the other hand, competent researchers should be able to redo much of an analysis without requiring the original authors to hand over every little thing that they did. In fact, that kind of thing is – in my view – an important part of the scientific method – running the same code using the same data is not really telling us anything.
In my view, the issue is really how do we determine if requests for more data/information is reasonable or not, and how do we establish the validity of claims that some researchers are not being transparent? I don’t think that simply saying “hand over everything” is sufficient, given that that won’t necessarily stop some from claiming that there is still more.
So, even though the title of the article may have been poor, I do think that much of what the article was suggesting was reasonable and that it shouldn’t simply be dismissed. I also don’t think that suggesting that the article could undermine a healthy trend is a fair assessment.
LikeLike
Time is of the essence. How much time is owed to a somebody making a request? Is there a difference between those making a request and those demanding it. How does one distinguish between naive request, expert requests and rent seeking?
LikeLike
It has been pointed out that harassment is a reason that young (and old) researchers may avoid areas of controversy. Thus saying that authors of most papers are not subject to harassment is not a very strong argument.
LikeLike
“It has been pointed out that harassment is a reason that young (and old) researchers may avoid areas of controversy.”
Yes, I’m sure it’s caused many a 29-year-old woman to avoid working anywhere near Rajendra Pachauri.
Oh, hang on—you were talking about imaginary, kindasorta virtual harassment, not actual harassment harassment.
My fault. I do this all the time: assume people are speaking English when they’re speaking Orwellian.
LikeLiked by 1 person
Here are three cases stripped of their identifying context for people to consider
1. A person demands the actual files the author used rather than being satisfied with a pointer to a data depository
2. A person demands data used in a paper by an author. The author points out that the data belongs to a collaborator and that the requestor should write to the collaborator. The collaborator sends the data, but the requestor publicly and vociferously continues to demand the data from the author. When confronted with the fact that he has the data, the requestor says he is not convinced that the author did not make changes.
3. A person demands data from an author. The author replies that some of the data was furnished under confidentiality agreements and that the author can purchase the data from the holders (they sell it for profit). The author organizes a FOIA circle to demand the data.
And, BTW, it ain’t just climate science, but GMO issues, evolution, vaccinations, etc.
LikeLike
The first two cases sound pretty bad and they should be publicly visible for all to see. Clearly in either case the requests should be ignored by the researcher and if they come repeatedly this would probably justify an injunction? In the second case, if there were some substantial evidence that the data were indeed changed that would of course change things but that would have to be pretty hard evidence.
The third case just sounds bizarre to me but I am not surprised that such a thing as data for sale exists
LikeLike
Data for purchase is actually quite common. Here are a few examples of databases for sale and those for free from NIST.
http://srdata.nist.gov/gateway/gateway?keyword=thermodynamic+data
LikeLike
That’s sad but not surprising I guess. But I think the days of such a thing being acceptable are numbered.
LikeLike
Lewandowsky had a 32,757-year-old, a 5-year-old, and several other minors in his data. He had falsely reported that age didn’t correlate with his variables, when in fact it did — the 32,757-year-old had blown open his deviation scores and snuffed out the correlation. When he was informed of the presence of a 32,757-year-old, a 5-year-old, et al in his data, he did nothing. He didn’t run a correction, didn’t clean his data — he literally did nothing and let his false paper just sit there in the literature as though it were true. More than a year passed. When I reported all this to PLOS ONE, they made him run a substantial correction: http://www.joseduarte.com/blog/minors-lewandowsky-and-ceremonial-ethics
His 2012 Psych Science paper is comprehensively false. The titular effect turns out not to exist — only three people held those beliefs. And look at Table 1. He and his collaborators gave uniform and false factor loadings for their main conspiracy variable. This somehow slipped past everyone, much like the sadness and color perception paper: http://retractionwatch.com/2015/11/05/got-the-blues-you-can-still-see-blue-after-all-paper-on-sadness-and-color-perception-retracted/
Every factor loading on that index is false and inflated (and uniform, which would necessitate falsity). I think by briefly saying that they were doing this in the footnotes (but not explaining why), they got people to ignore it. No one has ever done this before as far as anyone knows — no one has ever presented identical factor loadings for every item (which, again, would necessarily make them false for any realistic psychometric instrument.)
He also engaged in post hoc DV removal — he had an Iraq conspiracy item endorsed by liberals, which he deleted from his conspiracy index. He did not disclose this in the paper. The paper is comprehensively false anyway. The items he retained didn’t hang together, but the reader didn’t know this because he gave false and inflated factor loadings. There was no conspiracy factor as described in the paper.
2016 will be a great year for transparency and Stephan Lewandowsky.
LikeLiked by 4 people
Climate psychologists are the most precious, and emotionally fragile, among us.
Please stop harassing them with evidence, Mr Duarte!
You’re being illegitimate!
LikeLiked by 2 people
It just suddenly occurred to me, it it were 600 years ago we might all have been engaged in an almost identical debate about whether the Bible should be translated into the vernacular.
Pros: “Its good for transparency. The people have a right to make their own minds up about what is an isn’t God’s word by reading the original source, not just relying on some priest who may or may not give a valid interpretation.”
Cons:
“We’d be giving the public all this information, and they might misuse it! They’re simply not qualified to understand the bible’s deeper meaning.”
“What if they discover in it that makes us look bad – you know, about how god said we’re not suppose to create or worship idols (a worry for all those statues of the holy mother) – or they don’t find any mention of some of the stuff we are big on, like virgin births. They could use the information to question the True Church, and undermine its important good work. ”
“Not the good folk, of course, most folk are decent. But what about those with dangerous ideas and ambitions to lead our flock into their own temples of heresy”
All these “cons” turned out to be well justified fears. But looking back, the pro’s still far outweigh thecon’s, don’t they? Why is that?
Something to do with the way we value a person’s right to come to their own conclusion on any matter by looking at the evidence themselves, without having to an elite intermediary to tell then what they can and can’t know.
(PS No offence intended to Catholicism here. The Holy Mother is just a nice example, as a concept its just much bigger in the Catholic church than it is in the Bible, so makes this point well).
LikeLiked by 1 person
Given that my comment to Richard Tol above seems to have been lost and I said the same thing there I will just repeat it here. I do not believe that this Nature article argues anything of the kind. Dorothy Bishop has been a long-term proponent of open science, not just data sharing but public data. This is certainly what I support. Their list reads to me as a guideline for outsiders to judge the validity of a scientific argument not about whether or not data should be shared. Such a judgement is by its very nature subjective but in the end it is the broader consensus that matters.
LikeLiked by 1 person
1# There is an grass roots campaign by chronic fatigue syndrome patients, advocates, and academics to challenge the bad science claiming that the condition can be cured by positive thinking and exercise.
#2 There is no campaign to harass researchers.
The distinction is important.
The authors of said research conflate #1 with 2# at every opportunity.
Now let’s see some of reasons patients are upset about the PACE trial:
The study included a bizarre paradox: participants’ baseline scores for the two primary outcomes of physical function and fatigue could qualify them simultaneously as disabled enough to get into the trial but already “recovered” on those indicators–even before any treatment. In fact, 13 percent of the study sample was already “recovered” on one of these two measures at the start of the study.
In the middle of the study, the PACE team published a newsletter for participants that included glowing testimonials from earlier trial subjects about how much the “therapy” and “treatment” helped them. The newsletter also included an article informing participants that the two interventions pioneered by the investigators and being tested for efficacy in the trial, graded exercise therapy and cognitive behavior therapy, had been recommended as treatments by a U.K. government committee “based on the best available evidence.” The newsletter article did not mention that a key PACE investigator was also serving on the U.K. government committee that endorsed the PACE therapies.
The PACE team changed all the methods outlined in its protocol for assessing the primary outcomes of physical function and fatigue, but did not take necessary steps to demonstrate that the revised methods and findings were robust, such as including sensitivity analyses. The researchers also relaxed all four of the criteria outlined in the protocol for defining “recovery.” They have rejected requests from patients for the findings as originally promised in the protocol as “vexatious.”
The PACE claims of successful treatment and “recovery” were based solely on subjective outcomes. All the objective measures from the trial—a walking test, a step test, and data on employment and the receipt of financial information—failed to provide any evidence to support such claims. Afterwards, the PACE authors dismissed their own main objective measures as non-objective, irrelevant, or unreliable.
In seeking informed consent, the PACE authors violated their own protocol, which included an explicit commitment to tell prospective participants about any possible conflicts of interest. The main investigators have had longstanding financial and consulting ties with disability insurance companies, having advised them for years that cognitive behavior therapy and graded exercise therapy could get claimants off benefits and back to work. Yet prospective participants were not told about any insurance industry links and the information was not included on consent forms. The authors did include the information in the “conflicts of interest” sections of the published papers.
This is from an article by David Tuller who has done excellent work exposing the PACE trial as the bad science it is:
http://www.virology.ws/2015/10/21/trial-by-error-i/
There are many more problems. The PACE trial is so bad it should never have passed peer review. Yet the critics are labelled as harassers with a personal vendetta.
By the way, there are millions of sufferers worldwide, about 200k in the UK alone.
LikeLike
1. The PACE researchers clearly do not argue that CFS/ME is all about “positive thinking and exercise”. I agree with the critics that there are some unanswered questions about the PACE trial but this is a strawman argument.
2. Whether or not harassment of the PACE researchers took place is becoming irrelevant at this point. There is evidently harassment of anyone who does not agree with the critics’ stance. Doesn’t this strike you as ironic? This obviously doesn’t mean that all patients/carers are involved in this. Rather it’s a small vocal group whipped up into a frenzy by irresponsible figures on social media who are not helping the cause.
LikeLiked by 1 person
I’m curious as to why you have spent 2 days portraying critics as harassers or even maniacs in social media.
How do you define harassment? Is engaging in a debate harassment? If a lot of people join in a discussion with a particular point of view, is that harassment?
Could you point out exactly which critics are engaging in harassment and which are engaging in legitimate debate? (Hint: Flawed debate or logic is not harassment) How many tweets can one send before it is harassment?
You cannot police a large group of people with different views and agendas. You cannot immunize a large group of people from agent provocateurs either. This to me is power trolling. You claim harassment when in fact you conflate and denigrate. When you generalise all critics and falsely associate them, you are marginalising the legitimacy of individuals.
There is plenty of disagreement among the critics of PACE, there variances of approach, opinion, tone and expertise.
You cannot simply block people on twitter, call them maniacs and harassers indiscriminately and with scatter-gun rope fences and expect to be taken seriously.
Small Vocal Group sounds like awfully familiar language to me. Which individuals are in this ‘Small Vocal Group’ and which individuals are not?
I could swap Small Vocal Group for Small Brave Group. It really depends which individuals we’re talking about. It’s scary to be an advocate when you’re being conflated with criminal or oprressive activity for simply voicing an opinion.
Over 11,000 people have signed the petition to retract PACE. There aren’t 11,000 patients online extolling the virtues of CBT/GET.
LikeLike
I don’t think summarizing the treatment as positive thinking and exercise is fundamentally wrong. One could be more specific and describe it as “cognitive behaviour therapy that aims to eliminate unhelpful thoughts and beliefs about the conditions, as well as a program of exercise of gradually increasing intensity”, but that’s a bit long.
As mentioned before, I have not seen any organized harassment campaign. I do see many people criticising the research, and the distinction is important.
I also don’t see any particular organization or individuals behind the criticism. It’s a grass roots phenomenon. There are many patients with this condition, and bad science that hurts them will also generate a lot of criticism. Recently James Coyne became interested in the topic and started tweeting about it, but patients have been pointing out the same problems for many years.
I also want to say more about the notion that this is only a “small vocal group” of patients that opposes CBT and GET.
A survey of patients conducted by the ME Association found that graded exercise therapy was reported to be harmful by 74% of patients, and that cognitive behavioural therapy was largely ineffective.
http://www.meassociation.org.uk/2015/05/23959/
In Belgium, during 5 years, the Belgian Government subsidized 5 “CFS Reference Centers” who treated patients solely with CBT/GET. They used the Fukuda criteria for selection. Every year, 1.5 million Euro was distributed to the 5 centers. This makes a total of 7.5 million Euro during 5 years. 800 patients were treated
during this period. The Belgian Health Care Knowledge Centre (KCE) reviewed the result after 5 years (2009). The KCE is a semi-governmental institution which produces analyses and studies in different research of health topics.
Conclusion :
patients feeling better : 6 %
Patients feeling worse : 38 %
Patients with no result, not better nor worse : the remaining 56 %
The ultimate goal of the Reference Centers and their CBT/GET therapy was
not met: NOT ONE PATIENT RESUMED WORK.
http://www.bmj.com/rapid-response/2011/11/02/belgian-experiment-cbt-and-get
If CBT and GET critics are only a small vocal minority, where is the silent majority? PACE authors assure us it exists but I have seen no independent evidence for it.
CBT and GET don’t work in the real world. They only work in biased studies that are bad science by many criteria.
Psychiatrists and psychologists that care about the reputation of their profession should join patients criticising the PACE trial.
LikeLike
@Matt: Honestly if you have to ask this question then I don’t think there is much point in my explaining it to you. If you think the social media reaction to neutral people and potential allies is justified and you believe the conspiracy theories being spun about UK science, then I definitely have no hope of convincing you.
The fact some are still surprised that after what has been going on lots of people don’t want to talk about PACE or CFS/ME research in general demonstrates a severe lack of social awareness. Anyways, while I doubt it will clarify things in any way here is one answer to your question: https://www.citizensadvice.org.uk/discrimination/taking-action-about-discrimination/taking-action-about-harassment/
Have a nice day.
LikeLike
@AB: A harassment campaign need not be organised to be stressful and damaging to the victims. I have observed and experienced clear harassment and I’m not even associated with PACE in any way shape or form. Some of the criticisms of it sound very valid but it’s a two-edged sword. I haven’t read most of the responses to the critics but that which I’ve seen suggests that this is has more than one side.
I’m not saying that the PACE critics are a small vocal group. I’m saying that the harassment is caused by a few unsavory characters who give thousands of patients a bad name.
LikeLike
@Sam
What you are preaching for is a world that cannot possibly exist. In large groups of people you simply cannot expect a uniform standard of behaviour. You cannot expect patients who lives have been destroyed by illness to all tow the line. Disability creates anger. Contentious diseases create anger two because the patients are not just alienated from able society, they are alienated from the legitimacy of the experience of their own health.
Chronic Fatigue Syndrome (Oxford Criteria especially) by design is the perfect schematic for oppressing patients. It postulates that the patient does not act through a fear that exertion will cause them injury or sickness based on their experience during an initial physical illness. Once you define it by those terms, any patient protestation can be seen as proof of that fear, and rejection of the diagnosis has been portrayed as prejudice a stigma towards mental health issues. It’s easy to do, patients aren’t PR trained, they say things like “I’m not crazy/mad!”. But you can be offensive and yet still be oppressed.
The problem, and why patients are so angry, is that the literature simply doesn’t reflect that. Even if you took PACE for what it states, it uses the tamest of diagnosis criteria to obtain patients and fails to look at any of the signs and symptoms reported throughout the literature on ME/CFS and report on whether the treatment abated them or not.
Unfortunately, what you need to decide is, whether you want to live in a world where people who aren’t eloquent enough, are too repetitive or even insulting strips you of your sense of inquisitiveness or pursuit of truth. If you create a wall based on an unattainable state of existence, you’re only putting a fence around what you’re prepared to discover.
ACT UP didn’t behave. I dare say they did some pretty aggressive, rude and sometimes even morally reprehensible things. This is how things happen.
I know a lot of people are coming to you with beliefs, but beliefs still have to be tested. Even Scientists are often just masking their educated beliefs behind a different lexicon claims to logic and reason.
You say I believe in conspiracies, but I think it’s more like, there are conspiracies of convenience. Problems aren’t convenient, and sometimes it’s nice to listen to voices that tell us they aren’t really there. The thing is, when the problem is making you pass out, giving you migraines and stopping you from walking, the voices sounds a lot weaker.
LikeLike
@Matt: I’m not “preaching” for anything. I am trying to explain how this behaviour appears to outsiders and to outsiders who get dragged into it. This isn’t patronising either. It is simply saying that you lose the support of people who might have been on your side otherwise.
Also, believe it or not, I actually understand why someone who suffers from a debilitating illness or their carers might react angrily in this situation. I don’t particularly blame them, although while being a patient may be an explanation it is not an excuse. Either way, I know there are plenty of decent people in that community who are capable of reasonable discussion. This behaviour by a few just reflects badly on them too. More importantly, people like Coyne who purport to be rational scientists should know better and should behave responsibly. Aggressively RTing the same tweet over and over and misrepresenting what others say to his legion of followers who he knows are likely to have an angry reaction makes him directly responsible for the consequences.
I continue to have sympathy for the patients (I know some CFS sufferers myself albeit their cases aren’t as severe as it clearly can be). I also think the data should be shared, partly because I generally believe in open data but more specifically because criticisms can be evaluated only when you can see the data for yourself. But I no longer have any interest in this particular cause. I sympathised for the cause of PACE critics before – after this, no longer. And I am sure I am not alone. If this goes like this, support will only crumble further. If you think that is patronising, fine. It’s just a word of advice. And this is the last I will say to you or anyone about this matter. All the best.
LikeLike
In the PACE trial, the graded exercise therapy group achieved a mean distance of 379 metres in a six minute walking test, after 52 weeks of treatment. This is worse than the distance achieved by older patients with chronic heart failure and patients listed for lung transplantation.
http://www.meassociation.org.uk/2013/07/pace-trial-letters-and-reply-journal-of-psychological-medicine-august-2013/
These results are incompatible with recovery. There might a subset of patients for which this treatment works, but without individual patient data we cannot be sure. Despite numerous requests, the authors continue to refuse sharing data, and continue to act as if their favourite interventions constitute effective treatment.
At the 2.5 year follow up there was no difference between any of the treatment groups. The long term follow up paper also had some other problems:
http://blogs.plos.org/mindthebrain/2015/10/29/uninterpretable-fatal-flaws-in-pace-chronic-fatigue-syndrome-follow-up-study/
I hope this is sufficient to convince neutral observers that chronic fatigue syndrome patients have good reasons for being critical of this work (there are many more problems, too many to list them here).
LikeLike
I’ll make a potentially more controversial comment. My understanding of the Bishop & Lewandowsky article is that it is suggesting that we should be aware of, and be able to identify and deal with, situations where interactions have passed from what would be regarded as reasonable academic discourse into something that is less reasonable, and – potentially – becomes harassment. The author of this post appears to have interpreted it somewhat differently; essentially that the article is arguing against transparency. Three of those who appear to agree with the author’s interpretation are Richard Tol, Barry Woods, and Jose Duarte. Richard Tol has waged a campaign against John Cook, claiming that he has not released all of the data relevant to his 2013 paper, and that the paper should be retracted. Barry Woods appears in the comments of almost any article written by – or about – Stephan Lewandowsky to highlight various potential issues with how Stephan Lewandowsky does his research, and Jose Duarte has made a habit of accusing Stephan Lewandowsky of fraud.
Are these examples of crossing the line from reasonable to unreasonable academic discourse? My own view is yes; others may disagree. The author of this post may, of course, want to confirm what I’ve said above, but – if they do so – they may want to consider the significance of this in light of what was being suggested in the Bishop & Lewandowsky article.
LikeLike
“My understanding of the Bishop & Lewandowsky article is that it is suggesting that we should be aware of, and be able to identify and deal with, situations where interactions have passed from what would be regarded as reasonable academic discourse into something that is less reasonable, and – potentially – becomes harassment. ”
So your précis of the article, Ken, is that it was just making a saccharine, truistic motherhood statement everyone on earth would agree with, to the (very limited) extent that it was saying anything at all. Which would raise the question, in the minds of everybody who’s capable of questioning things: how did such a vacuity get published? (If your understanding is correct, the article is too inane even for the Internet.)
In other words, you didn’t understand it.
Just say so next time and we can avoid unnecessary debate.
LikeLike
Given your typical behaviour, I had assumed that you didn’t agree with the idea that we should be aware of, and determine how to deal with, situations where some party’s behaviour is no longer reasonable. So, no, I don’t think I was making a truistic statement that everyone on earth would agree with.
LikeLike
Gosh, you really are a delicate vase, aren’t you Ken?
From this day forth I shall make it my mission to protect you from parties whose behaviour is no longer reasonable. If you shatter, I fear we may never see your like again. I just couldn’t live with myself if I allowed that to happen.
LikeLiked by 2 people
You’ve interpreted my comment somewhat differently to how I expected. Oh well.
LikeLike
The poor baby Brad!
LikeLiked by 1 person
Extracts from Dorothy Bishop’s blog in November 2015:
“Fears about misuse of data can be well-justified when researchers are working on controversial areas where they are subject to concerted attacks by groups with vested interests or ideological objections to their work. There are some instructive examples here and here. Nevertheless, my view is that such threats are best dealt with by making the data totally open. If this is done, any attempt to cherrypick or distort the results will be evident to any reputable scientist who scrutinises the data. This can take time and energy, but ultimately an unscientific attempt to discredit a scientist by alternative analysis will rebound on those who make it. In that regard, science really is self-correcting. If the data are available, then different analyses may give different results, but a consensus of the competent should emerge in the long run, leaving the valid conclusions stronger than before.”
“The ‘fear of errors’ argument is, of understandable but not defensible. The way to respond is to say of course there will be errors – there always are. We have to change our culture so that we do not regard it as a source of shame to publish data in which there are errors, but rather as an inevitability that is best dealt with by making the data public so the errors can be tracked down.”
http://deevybee.blogspot.co.uk/2015/11/whos-afraid-of-open-data.html
LikeLike
Precisely my point. Dorothy has been a vocal support of public data for a long time. However, it is a valid concern that in controversial research areas there is a danger of a large volume of cherry-pickers and people with unscientific agendas trying to discredit the research. There must be a way to deal with that situation. As I have argued repeatedly, the solution cannot be to lock up the data. Rather scientists must be selective in how to respond and whether they respond to some criticisms at all. But more importantly, we also need ways to communicate to the non-expert observers which criticism have merit and which do not. The Lewandowsky & Bishop piece is a start in that direction. It clearly isn’t sufficient.
LikeLike
“Rather scientists must be selective in how to respond and whether they respond to some criticisms at all.”
There’s NO obligation on scientists to respond to criticisms at all. Never has been. If that was the point of Lew’s piece, then Lew’s piece was as pointless as has been pointed out, pointedly.
“However, it is a valid concern that in controversial research areas there is a danger of a large volume of cherry-pickers and people with unscientific agendas trying to discredit the research.”
No it isn’t.
The process you’re describing isn’t a problem, it’s science. Why don’t you like science, Sam? Hasn’t science been good to you?
Incidentally it’s curious that you advocate selectivity and then, in the next breath, condemn “cherry-pickers.” Well, curious to me anyway.
Sorry, but you forfeit the moral high ground to tut-tut at cherry-pickers for focusing on partial data if you don’t release ALL the data to ALL who request it.
“There must be a way to deal with that situation.”
There IS.
You publish the &$%&(ing data.
Then you can’t possibly be inconvenienced by requests for it, because you don’t own it anymore.
Only academics could possibly think this was complicated.
Actually I’m sure even they aren’t suffering genuine confusion—even such middling minds as Lew et al. must surely grasp something that has been crystal-clear for centuries. However, it suits their own short-term interests to muddy the waters.
Ho hum.
They can run, and they can hide, but they can’t hide forever.
LikeLiked by 1 person
“Fears about misuse of data can be well-justified when researchers are working on controversial areas where they are subject to concerted attacks by groups with vested interests or ideological objections to their work.”
The presumption here is that the researchers do not have vested interests or ideological biases.
LikeLike
There is no such presumption. This is why we need open data and post-publication review. We need to be able to evaluate a scientific claim critically. However, there is a massive power imbalance. One research lab may have vested interest or biases but they are only voice. This is nothing compared to the volume of dozens or hundreds of people sifting through your data will questionable motives.
LikeLike
Sam: Bishop argues that researchers should be protected from people of impure motive. She omits that the people should be protected from researchers of impure motive.
LikeLiked by 1 person
Richard Tol: “She omits that the people should be protected from researchers of impure motive.”
No she doesn’t. It’s all she’s talked about for years.
LikeLiked by 1 person
Sam: I don’t know Bishop from Eve. In the blogpost you linked to, she does not refer to the need to keep a check on naughty researchers.
LikeLiked by 1 person
Richard: I have no idea what you’re talking about. What blog post did I link to?
Anyway, while I think there are some important questions to be discussed here I feel I will leave it for another day and another place. Best regards.
LikeLike
Sorry: AM linked to Bishop’s blog.
LikeLike
https://jcoynester.wordpress.com/2016/01/31/further-insights-into-the-war-against-data-sharing-the-science-media-centres-letter-writing-campaign-to-uk-parliament/
Perhaps this explains Bishop’s Nature post.
LikeLike
The authors of the Nature article have said that their article has been misunderstood. If that’s the case then it seems to have been widely misunderstood because a broad section of readers have taken issue with it. I don’t understand the intent of the article; I find its message muddled and confusing. On the one hand the authors seem to advocate the usefulness of sharing data, but on the other hand they don’t seem to want the public (the unwashed masses) to have access to data. Allowing science only to take place behind closed doors confined to a hierarchy of colleagues who are vulnerable to peer pressure and institutional bias, is not healthy for science. Either data is open or it isn’t. If it’s open, then healthy pursuit of post-publication peer review runs the risk of making academics uncomfortable. But surely that’s healthy? Scientific literature must be able to stand up to scrutiny. And surely researchers should welcome scrutiny. Honest mistakes are not be something to be afraid of. Errors can advance science, and add to scientific discourse, as long as there is openness, transparency and scrutiny. No academic should be embarrassed by honest errors. It’s how an error is responded to that matters. In other words, open data is not something to be afraid of. However uncomfortable this new world of transparency may be at times, it will bring about positive and welcome change at all levels of academia.
LikeLiked by 3 people
I have ME/ CFS – have had for over 20 years (vaccination that did it). I’m sorry to have to correct Sam Schwarzkopf, but:
a) though normally silent, I am on the side of the critics of PACE: so is every other person with the disease that I have ever encountered; these people are not irresponsible or frenzied but brave campaigners for justice and they have the overwhelming support of the patient community.
b) The proponents of GET did indeed base the idea on the claim that CFS is a “false illness belief” in which the patient confuses fatigue due to deconditioning with actual illness. No other view of the condition could justify an attempt at cure by exercise and thinking oneself well (the CBT arm of the PACE trial – not the sort of CBT that people get for other physical conditions). The fact that these same individuals are now reluctant to give a clear endorsement of this view publicly is simply an indication of the poor nature of the PACE results and the clear evidence of physiological abnormalities emanating from real scientific studies. They have, however, continued to peddle the “all in the mind idea” to insurance companies (see David Tuller’s article here http://www.virology.ws/2015/11/17/trial-by-error-continued-pace-teams-work-for-insurance-companies-not-related-to-pace-really/). I can also assure Schwarzkopf that UK medics still overwhelmingly believe their CFS patients to be suffering from a form of hysteria, and treat them accordingly.
LikeLiked by 1 person
You have my sincerest sympathies and I only wish the best for you. But I’m afraid you are misreading what I’m saying. I haven’t accused you personally of anything and if you aren’t hauling abuse at people I assume you are one of a doubtless overwhelming majority of decent people. Either way, as I said above I have no more interest in discussing the PACE case.
LikeLike
The Nature article is one-sided; It discusses what the authors acknowledge (i.e. in the comments above) are very rare occasions whereby a small section of the public may be intent on causing actual harm (as opposed to discomfort) to an academic. The authors do not balance the article by discussing the opposite (hopefully rare) scenario whereby academics may use their research, status and influence to mislead the media, the public, other academics and institutional policy makers in order to further their own interests. In the case of medical research, such a scenario could have particularly harmful effects, and would equates to abuse and harassment of a patient population.
Misleading the public can be done in a very subtle way (e.g. exaggerating the headline of a press release) or it may involve the outright corruption of data. In any case, openness and transparency is one of the only ways that the public can hope to challenge the power of academic elites who hold great influence over their lives. With power, comes responsibility and accountability. Academics (especially if using public funds) should accept this responsibility as part of their public duty, however uncomfortable it may be at times.
LikeLiked by 1 person
The Nature article authors mention the PACE trial, which was an investigation of CBT and graded exercise therapy (GET) for chronic fatigue syndrome (CFS). The PACE investigators recently carried out a long-term follow-up study which assessed the treatments at a median of 2.5 years. The study found no difference between trial arms i.e. CBT or GET were no better than no treatment when add to usual care. However, despite these clear results, the investigator claimed (in a press release and in other media interactions) that CBT & GET were clinically beneficial at follow-up. This misleading characterisation of the results was heavily promoted by the investigators to the media. One newspaper went so far as to say that CBT & GET successfully treat CFS and so CFS cannot be considered a chronic illness. However, the study demonstrated the opposite: that the treatments have no long term effect.
It is this sort of misrepresentation of data that has caused a breakdown in trust between a small group of influential CFS researchers and the CFS patient community.
I would argue that misrepresenting medical trial results is unethical and an abuse of the patient population. Misinformation causes disproportionate and well-documented difficulties for patients in many areas of their lives.
Refusing to share publicly-funded data with patients or patient representatives, so patients can assess recommended treatments for themselves, could be considered patriarchal at best, and an abuse of authority and status at worst. However, in asking for access to the trial data, the CFS community had been characterised (and heavily promoted in the media) as causing harassment and being extremist etc. This seems to be a (very successful) distraction technique rather than a proper response to the requests. One can only wonder why the investigators refuse to make the data available, and why they are so hostile towards the patient community.
It doesn’t seen unreasonable for patients to request (entirely anonymised and unidentifiable) publicly-funded trial data, especially when all the protocol-defined primary trial outcomes were switched (i.e. post-hoc), and the data has been widely misrepresented (e.g. ‘recovery’ was redefined to include patients who deteriorated on the trials primary outcome measures), and when the media has repeatedly misrepresented the outcomes in a very harmful way.
The CFS field is exceptionally complex, and I would ask the Nature article authors to avoid assuming that their colleagues’ perspectives tell the whole story, especially if they haven’t widely consulted the CFS patient community to try to understand the whole complex and nuanced picture.
LikeLiked by 1 person
>> However, it is a valid concern that in controversial research areas there is a danger of a large volume of cherry-pickers and people with unscientific agendas trying to discredit the research.<<
Well yes – but “controversial research” is an a priori classification and “cherry-pickers and people with unscientific agendas” an a priori denigration which leads to: “I work in a controversial field, ergo anyone not of my perspective is trying to discredit my research”. This is precisely what has played out in the PACE drama. A rather ordinary piece of research produced insubstantial results which were then hyped beyond all reason by the researchers themselves via institutional press officers and contracted media professionals. The hype attracted criticism upon which the ‘controversial research’ = harrassers line was then employed as a tactic to discredit the critics by those same contracted media professionals. Lewandowsky and Bishop made choices in using that media same context in running a consultation and then producing an article; it is difficult to see those choices as neutral when Dorothy Bishop is on the Advisory Committee of the PACE contracted media organisation, while an advisor to the PACE trial is a Trustee of that same organisation and that organisation persists in a narrow perspective on CFS that is at complete variance to that expressed by patients http://forums.phoenixrising.me/index.php?threads/uk-cmrc-jan-2016-board-meeting-minutes.42566/#post-688940 . Note this linked comment is from Charles Shepherd who is most certainly not a cherry-picker or person with an unscientific agenda, is not one of a small vocal group whipped up into a frenzy by irresponsible figures on social media, and has himself experience of criminal harassment because of his advocacy of CFS patient interests.
One of the fundamental failings of Lewandowsky and Bishop’s article in Nature is that it fails to acknowledge any distinction between human subject research and research in other fields where people are in the main removed from individual impact of research findings. Lewandowsky and Bishop observe that a mechanistic solution – essentially telling patients that the authors will be the final arbiters of what data will or will not be released – as a Trial condition, would have precluded any FOIA difficulties for the PACE researchers. Sadly this solution is just the old patrician man/woman in white coat knows best approach that is simply not acceptable in modern medical research. Those researchers into human health and behaviour who need human subjects to particiate in trials, must recognise those particiants as partners, partners who have reasonable expectations of data transparency and of the acceptance of research criticism. Those partners may not be scientific experts, but they are citizens who have every right to expect that information about the use of public money and the use of data it helps generate will be freely available. Those citizens should not be denigrated because they become vocal when researchers refuse explanation of the research proces when it appears inadequate or refuse acknowledgement of the patient experience when that experience is at odds with researcher claims. Dorothy Bishop especially should understand the unique position of the human being as research subject, it is sad that she so singularly fails to acknowledge it in her and Lewandowsky’s analysis.
LikeLiked by 1 person
[…] Getting the idea of transparency all wrong […]
LikeLike
Open Society versus closed society. The practice of science has to be
the former, open to criticism otherwise error prevails as dogma.
Comment by J Duarte (@ValidScience) says:
January 31, 2016 at 1:04 am
Lewandowsky had a 32,757-year-old, a 5-year-old, and several other minors in his data. He had falsely reported that age didn’t correlate with his variables, when in fact it did — the 32,757-year-old had blown open his deviation scores and snuffed out the correlation. When he was informed of the presence of a 32,757-year-old, a 5-year-old, et al in his data, he did nothing. He didn’t run a correction, didn’t clean his data — he literally did nothing and let his false paper just sit there in the literature as though it were true. More than a year passed. When I reported all this to PLOS ONE, they made him run a substantial correction: http://www.joseduarte.com/blog/minors-lewandowsky-and-ceremonial-ethics
LikeLiked by 4 people
Sam Schwarzkopf says:
They are absolutely arguing against transparency. They want “pre-registration” before I can access data. Transparent? Not.
They also echo Phil Jones famous question about why should I give my data to you when you will only try to prove me wrong, when they say:
Here’s the part that Lewandowsky et al. don’t understand—you should give all of your data and your code to your strongest scientific opponent. If he/she can’t find problems with your work, you’re doing well. And I don’t give a ha’penny whether Lewandowsky thinks I am acting in “good faith” … in part because he seems to think that anyone who disagrees with him is acting in bad faith.
But instead of giving their code and data to their worst enemies, they are claiming that they should be able to keep their work private because they distrust the “good faith of the people requesting data”?
Sam, that is arguing against transparency and openness at the top of their lungs. Science doesn’t operate on “good faith”. You do remember “Nullius in verba”, I’m sure.
So no, I don’t take Lewandowski’s word for a damned thing, including whether those who disagree with him are acting in good faith, just as the Royal Society advises …
w.
LikeLiked by 3 people
Thanks you make a great point, Will. I can understand why you read it this way but I again think you are misinterpreting this. Their comment is not about data access! While preregistration of scientific protocols is gaining traction not even the most ardent supporters of this model seem to argue that it should be mandatory. As I repeatedly said already, Dorothy Bishop has been a vocal proponent of open data sharing, that is, all data being uploaded to public repositories. I don’t know about Stephan Lewandowsky but by coauthoring with Dorothy the article is hardly going to argue against something she has long been fighting for.
Thus with public data nobody needs to do anything to access data. What they are saying in this comment is that anyone who reanalyses the data should adhere to the same principles of transparency that we expect from the original authors. If you take the data to scrutinise somebody else’s claims – which is certainly part of good science – then you should also demonstrate that you aren’t just on a fishing expedition. The whole point of preregistration is to make the distinction between confirmatory and exploratory analyses clear as this is something science up till now has been very bad at. It is fair to ask reanalyses to do the same. And again, nobody is saying prereg should be mandatory but if you don’t do it you can’t expect the readers to believe you went in with clean motives.
So yes I think the preregistration idea is actually a great one and perhaps the only practical suggestion in that comment (other than the “red flags” which are quite subjective). The prereg has only one problem in that it would be possible to take the data, fish for your false smoking gun, and the preregister and publish the result after a while. The chronology would have to be made very clear and I am not sure that’s possible.
Your idea of giving the data to the worst enemy is idealist. You can find inconsequential, minor flaws in just about any study if you set your mind to it. That said, if you can convince your worst enemy that your claim is kosher, I agree that’s compelling evidence. But this is not going to happen often in the real world and even less likely when your worst enemy is a novice without the necessary expertise who reanalyses the data out of political motivations.
LikeLike
“If you take the data to scrutinize somebody else’s claims – which is certainly part of good science – then you should also demonstrate that you aren’t just on a fishing expedition.”
What in the world is wrong with a “fishing expedition”? Steve McIntyre went fishing on the hockey stick research and caught tons of fish–exposing major flaws in the proxies and methodologies of previously un-fished paleoclimatology papers. Who cares that his original motives in looking at Mann et al’s data were along the lines of “Hey, this smells fishy”? He was a smart, curious, dogged individual who wanted to see for himself if the highly influential claims of unprecedented global warming could stand up to careful scrutiny. When he asked for information he was met with stonewalling, obfuscation, and charges of “harassment!” And when he finally got some of the information he was looking for, it became clear to all what motivated the stonewalling.
Lewandowsky and Bishop are arguing for nothing more than the right to censor their critics. How is that not obvious to everyone?
LikeLiked by 3 people
Gd Holcombe: What in the world is wrong with a “fishing expedition”?
I’m in no mood to now also rehash the preregistration debate especially given the long personal journey I have been on with respect to that topic… Let’s me just say this: A “fishing expedition” a.k.a. exploratory science a.k.a. stabbing around in the dark until you hit something is perfectly fine and may very well discover something interesting – but it is also extremely prone to bias and on its own it is usually weak evidence.
I neither have the knowledge nor the interest to say whether or not the hockey stick controversy might have been affected by this problem.
LikeLike
“But this is not going to happen in the real world…” Crikey, what’s happened to the real world to make it so awful and sinister?
LikeLiked by 2 people
Sam, thank you for your comments. However, you’ve missed my point. They still are saying that the data should NOT be open to those with the wrong mindset, viz:
So according to Lewandowsky et al., if I don’t have “good faith” I should be denied the data. You claim that they advocate making the data public … but if so, that verbatim quote from their paper makes no sense. Why would I have to “request data” if they advocate making it public?
And you go on to make the same claim, that only the approved peoples’ opinions should make a difference:
Again, you don’t get it. You continue to claim that the person’s mindset is important in deciding whether to grant them access to the data. And you still think that someone’s lack of education means you can ignore their objections.
That’s as wrong as proposing an “arbitrator” to decide whether I am acting in “good faith”. I have no faith at all—science is not about faith, good or bad. Despite that, Lewandowsky et al. want to judge whether I get data based on whether I’m acting in “good faith”? And you think that is science? Really?
Let me be clear. Science is not a touchy-feely popularity contest. It has nothing to do with the motives of the participants. I assume that many, perhaps most people reading my work do so hoping to show I’m wrong. And I encourage them to do exactly that. I don’t care what their motives are, political, personal, or otherwise. I want them to find errors in my work—it has saved and continues to save me thousands of hours following blind alleys.
The only important issue is, can they falsify my claims? I don’t ask them about their degrees or their certificates. I don’t inquire about their motivations. I look at their scientific claims, not whether they are “qualified” to make those claims.
And the only way to have people find my errors is for everyone to have full, total, unimpeded access to my data and my code. Whether or not they pass the Good Sam test of lack of political motivations. Whether or not they are asking in “good faith”, whatever that might mean. Whether or not English is their first language. Whether or not they are educated. The only relevant issue is, can they falsify my findings?
They all need my data and my code to see what I’ve done and examine it for flaws. And yes, they are trying to prove me wrong, and that is how science progresses. Not by just giving access to the good guys or by examining political motivations.
Science moves forwards by giving access to everyone.
Thank you for your thoughts,
w.
LikeLiked by 4 people
No, that isn’t what they said, which should be obvious from the bit that you quoted
LikeLike
…and Then There’s Physics says:
February 1, 2016 at 10:43 am
Say what? That’s exactly what they said. They say that if I request data and the researcher refuses to give it to me because they don’t think I am acting in “good faith”, their solution is that I get examined by an “independent adjudicator”.
And if the adjudicator thinks I’m not acting in good faith, I don’t get the data.
Which is exactly what I said.
Are you seriously agreeing with Lewandowsky and Sam that someone’s state of mind, which is often unknowable even in good times, should be used to deny them access to scientific data? Because that is exactly what they are saying.
w.
LikeLiked by 2 people
No, it’s not exactly what they said.
LikeLiked by 1 person
Actually sorry but you’re the one not “getting it”. Perhaps Lewandowsky is arguing for restricted data access – you need to ask him. Bishop has always been a proponent of public data so I doubt that this is what she is saying.
“You continue to claim that the person’s mindset is important in deciding whether to grant them access to the data.”
Sorry but I’m saying no such thing. Even if L&B were arguing for restricted data access (which i don’t think they are) I am clearly not. I believe all fully anonymisable data needed to reproduce a result should be publicly available.
And yes, any researcher and neutral observer has the right to ignore the reanalyses made by people if they think that they lack the expertise or have a clear conflict of interest biasing the conclusions. This does not stop anyone from reanalysing the results.
LikeLike
‘Science doesn’t operate on “good faith”.’
Indeed—I was led to believe it operated on good science.
(But that might be too much to ask when the unflushable Lew is involved.)
Wasn’t that the point of science? To make faith, well, obsolete?
A climate scientist and a proper scientist were walking down the street.
Climate scientist:
“Why should I make my data available to you, when your aim is to try and find something wrong with it?”
Scientist:
“Because my aim is to try and find something wrong with it.”
LikeLiked by 4 people
Would someone like to comment on the irony of the SMC both advocating for Ben Goldsmiths “All Trials Project” where ALL clinical trial data has to be made available vs the current paper by Bishop and Lewandowski who appear to be saying the exact opposite and are also apparently being supported by the SMC.
The arguments put forwardby B&L sound remarkably similar to the arguments produced by the Drug Companies against the All Trials Project
LikeLike
“Public declarations can be particularly useful: in 2014, in response to the harassment of one of its professors, the Rochester Institute of Technology in New York publicly acknowledged the scientific consensus on climate change and its support for academic freedom…”
I’d have thought the right reaction to alleged harassment is investigation and, if proven, prosecution of the harassment.
Instead we get a statement supporting a consensus which no official or corporate body in the world would dare contradict. Even Big Oil and Big Baccy will agree with the consensus: the first to sell gas as “bridging” to nowhere, and the second because it already has enough problems.
Look at the environment policies of Coca Cola or Exxon and you won’t find an argument against the consensus. It’s an undignified scramble to be greener and more Bambi-friendly than the next brute of a corporation!
Rochester Institute of Technology has to ask itself whether freedom is more about safe space or open space.
LikeLike
Thank you for all your comments. I have tried to approve almost all statements – please no personal insults though. I am probably going to close the comment section in the next days because I will not be able to ‘approve’ while I’m offline due to travels (and I don’t want you to feel that I’m holding comments on purpose). Feel free to provide a few links where the discussion can be continued elsewhere. I have certainly learned a lot from all this in the last few days.
LikeLike
Thank you for the post. This is an important discussion although I don’t think there is much point continuing this here. It would probably better in a more formal forum. I entirely understand why you don’t want this to go on here in any case.
The L&B article has clearly hit a nerve. I must say I find it rather fascinating how fundamentally different the interpretations of their message are. I certainly can see why some interpret it differently even if I can’t agree with them.
Anyway, thanks again and enjoy your travels.
LikeLike
Thank you for providing an independent place for people to present their views. I am slightly disappointed that it seems we will hear more from Slewandowsky and Bishop, particularly given Slewandowsky’s reference to ‘reasoned argument’.
To me, the reports from the Royal Society conference show some of the dangers of discussions being moved to more formal forums. One needs to be careful with the biases and assumptions which determine who is invited, and who is excluded.
LikeLike
Anonymous hard-line critic: I may actually write a blog post outlining my thoughts on this and move this away from whatever Lew & Bishop did or didn’t mean with their article. I had already brought up the topic before their Nature article was published so it makes sense I continue it. On there you will have the opportunity to comment on this further as I normally approve all comments that stay respectful. However, I’m too busy right now to write this so it may be a while and I may end up abandoning the idea. My regards until then.
LikeLike
Sorry about my multiple posts (if you got multiples). Tech problem on my end. I knew it wasn’t your fault. 🙂 I thought they weren’t posting.
LikeLike
I’m pretty much just lurking, but I would like to say that I hope none of Brad’s comments will be deleted. I realize that some people don’t appreciate snark (especially when it’s directed at their own views), but IMO Brad Keyes is a unique talent and was making important points.
LikeLiked by 1 person
I can only echo what Sam says, plus so long and thanks for all the fishes.
w.
LikeLike
Who decides if a scientists is being harassed, or shouting ‘harrassment’ when inconvenient questions get asked. A long look at the case documented at RetractionWatch shows academics trying the latter.
Prof Lewandowsky is special pleading, he has made public claims people are bullying and harassing journals with resect to hos retracted paper, and that the journal caved in to legal ‘threats’, to retract his work
The editors of the journal of his paper that was retracted- said that his was nonsense,
“…As we published in our retraction statement, a small number of complaints were received during the weeks following publication. Some of those complaints were well argued and cogent and, as a responsible publisher, our policy is to take such issues seriously. Frontiers conducted a careful and objective investigation of these complaints. Frontiers did not “cave in to threats”; in fact, Frontiers received no threats. The many months between publication and retraction should highlight the thoroughness and seriousness of the entire process. ”
Costanza Zucca, Editorial Director
Fred Fenter, Executive Editor
http://www.frontiersin.org/blog/Retraction_of_Recursive_Fury_A_Statement/812
and the founder of the journal said that his actions were abusing science….
(he added a personal comment)
http://www.frontiersin.org/blog/Rights_of_Human_Subjects_in_Scientific_Papers/830
Henry Markram:
My own personal opinion: The authors of the retracted paper and their followers are doing the climate change crisis a tragic disservice by attacking people personally and saying that it is ethically ok to identify them in a scientific study. They made a monumental mistake, refused to fix it and that rightfully disqualified the study. The planet is headed for a cliff and the scientific evidence for climate change is way past a debate, in my opinion. Why even debate this with contrarians? If scientists think there is a debate, then why not debate this scientifically? Why help the ostriches of society (always are) keep their heads in the sand? Why not focus even more on the science of climate change? Why not develop potential scenarios so that society can get prepared? Is that not what scientists do? Does anyone really believe that a public lynching will help advance anything? Who comes off as the biggest nutter? Activism that abuses science as a weapon is just not helpful at a time of crisis.”
Perhaps Prof Lewandowsky is not the best person to be championing for this topic? When an academic peer describing his actions as
– “activism abusing science as a weapon” – might any reader think his motives are to protect himself from actually sharing his disputed data?
And as you can see Markram is very concerned about climate change (certainly no climate sceptic) he is making a case for the standards of psychology to be upheld.
so which academic should the public belief, which side needsa ‘red flag’
– Prof Lewandowsky, vs Prfo MArkram, Dr Zenta & Dr Fenter
LikeLiked by 2 people