This analysis was aimed at quantifying how many studies registered on the Open Science Framework (OSF) up to November 2017 are performed but not shared after at least 4 years. Examining a sample of 315 registrations, of which 169 were research studies, the researchers found that 104 (62%) were published. They estimated that 5550 out of 9544 (58%) registered studies on the OSF are published.
Researchers use registries to make unpublished studies public, and the OSF policy to open registrations after a four-year embargo substantially increases the number of studies that become known to the scientific community. In responses to emails asking researchers why studies remained unpublished logistical issues (e.g., lack of time, researchers changing jobs) were the most common cause, followed by null results, and rejections during peer review.
The authors concluded that their study shows that a substantial amount of studies researchers perform remain unpublished.
I find this truly shocking!
Researchers are able to do research only because they receive financial and other support from elsewhere. Therefore they have an ethical obligation to publish it. The reasons frequently given for not publishing research are nothing well and truly invalid:
- Lack of time is a mere excuse; if researchers had the time to get the grants, permissions, etc. they simply must have the time to finish the job properly.
- Researchers changing jobs is an equally flawed excuse; if someone changes position, he/she is obliged to finish the job they were doing. A surgeon can also not leave mid-surgery because he has a better offer.
- ‘Null results’ is even worse as a reason. Null results are just as important as positive findings – occasionally they are even more important. If researchers fail to realize this, they simply disqualify themselves as researchers.
- ‘Rejections during peer review’ is complete nonsense. Everyone who submits papers for publication gets rejected once in a while. In this case, one learns from the peer-review comments, improves the paper in question, and re-submits it to another journal.
I have seen many studies of so-called alternative medicine (SCAM) that, for this or that reason, never were published. And I feel strongly that this is a serious violation of research ethics – so much so that I would ban researchers who are guilty of this crime from conducting research in the future. I also feel that, in order to receive the necessary support (financial and other), researchers should sign that they will publish their findings within a given time after finishing their study. Failing to comply could then incur a penalty such as paying back part of the funds wasted. I think such measures would very quickly clear up the current intolerable situation.
14 Responses to The scandal of unpublished research
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Ernst, dear chap.
Ah yes, wasn’t that part of the substance of the “Replication Crisis” (from which this registration idea arose), where Pharma companies were running multiple research projects – often in countries noted for corruption – then only publishing the favourable ones, and withholding anything that was negative or showed their product to be actually dangerous.
I agree, they shouldn’t do that. It’s wrong.
Such unethical behaviour may well have led to the known outcome that conventional treatment is a major cause of death, just after heart attacks and strokes. Starfield Report. #death-by-medicine. Not a “conspiracy theory”.
And major conventional players really shouldn’t be doing “knocking research” either – research designed to fail or exaggerate dangers – to attack alternatives to their own products. Research designed to mislead.
But what can you expect from money-led research, another of the main complaints during the replication crisis?
Thank you for raising this important topic. Better than your recent posts.
—-
If your good readers go looking and can’t find the many articles on the Starfield Report on the internets any more, it’s still in the archive
https://web.archive.org/web/20111129100805/http://www.drug-education.info/documents/iatrogenic.pdf
[Starfield, B. Is US Health Really the Best in the World? JAMA 2000; 284(4)]
@Will
Indeed, not a conspiracy theory, but certainly a severely flawed report.
Unfortunately, all the media jumped on this highly sensational title (“third leading cause of death”), but not a single news outlet bothered to report about the debunking and correction of this dung beetle feast of a paper. To this day, this one piece of sensationalist rubbish is waved around by all manner of uninformed idiots as if it is the Gospel Truth.
The rebuttal mentioned above shows a very compelling reason why this figure of 440,000 deaths per year through medical error can’t be right — because “it translates to 62% of all hospital deaths, as was pointed out by Drs. Benjamin L. Mazer and Chadi Nabhan.”
Just think about it: almost two thirds of all deaths in hospitals would be the result of medical errors? I’d say that notion is utterly ludicrous.
Better research suggests that the real numbers are likely ten times smaller. and this is also supported by Dutch research from 2016. The latter concluded that medical error was responsible for approximately 1700 deaths per year, placing it at the 21th position in the list of causes of death, about the same as suicide. Extrapolated back to the US, this boils down to about 34,000 deaths per year. Yes, that’s still a lot of deaths, but less than e.g. gun deaths. Yet I never hear any indignation from quackery apologists about those deaths – which in fact are almost entirely preventable, contrary to medical errors.
From McGill:
“Not only are these scary comparisons derived from dodgy numbers, as demonstrated earlier, but to compare the harms of medicine to the harms of alternative medicine without looking at their respective benefits isn’t fair. The health benefits of acupuncture, homeopathy, chiropractic and herbalism are few and far in between. (For an in-depth review of the evidence, I would strongly recommend Simon Singh and Edzard Ernst’s book, Trick or Treatment? Alternative medicine on trial.)”
Where is the data, Richard? Is Ernst’s book an analysis of the sources published by the BMJ and JAMA? Share it all!
@Richard Rasker
The majority of gun deaths in the USA are suicide deaths.
@Pete Frampton,
Richard Rasker wrote: “Extrapolated back to the US, this boils down to about 34,000 deaths per year. Yes, that’s still a lot of deaths, but less than e.g. gun deaths.”
The number of gun deaths in the US during 2021 was 48,830, however, this total excludes deaths in which gunshot injuries played a contributing, but not principal, role.
What the data says about deaths caused ONLY by homeopathy or acupuncture in the U.S.? Share the data and proper peer reviewed study, Richard.
@Pete Frampton
Yes, I know, but does that change anything? Suicide by gun is also a preventable death – especially given that guns make impulsive suicides exceptionally easy. I wouldn’t be surprised if this contributes significantly to the US having one of the highest suicide rates of all western countries. And suicides make up slightly more than half of all gun deaths; some 45% of gun deaths are NOT suicides, but homicides and accidents, and these numbers are staggering in comparison with other western countries.
Anyway, we’re drifting off-topic here, i.e. the problem of publication bias – both with regard to regular scientific research as well as research into alternative treatments.
Then again, there is a legitimate link with gun violence: in 1996, the lobby of gun idiots in the US managed to effectively ban the CDC from researching gun deaths, ostensibly because directing government funds towards researching the impact of firearm availability on public health could lead to more gun control.
I find this cynical in the extreme, because by pushing this amendment, the lobby of gun idiots in the US tacitly admits that guns most likely have enough negative impact on public health to warrant stricter control. But instead of facing the downside of their deadly hobby, they choose to not only close their own eyes to the harsh reality, but also forbid other people from seeing that reality – all so that they can claim plausible deniability for all the deaths and suffering they cause by pushing guns in the hands of as many people as possible.
I am not sure what the policies are in Germany or in the UK, where Dr. Ernst has worked, but there might be a different set of issues in the U.S.and perhaps they have some relevance to the E.U. or U.K. as well. First, it might be important to separate or distinguish government-funded research from privately funded research. Second, it might be useful to distinguish forms of “publication.” Dr. Ernst’s philosophy about the obligation to publish research findings for government supported research in the U.S. might be met through the mechanism of the project final report that goes to the funding agency. At that point, funding agencies can make reports available to the public. This then satisfies the reporting obligation of the PI, and insures that research outcomes are available without the uncertainties of journal or book publication. The obvious downside to such a policy would be that academic researchers need peer-reviewed publications for promotions, etc, and might encounter reluctance if a paper/article is already in circulation through, say NIH or NSF. One way to moderate this might be to have a PI outline a publication schedule, and for the funding agency to withhold publication within, say, 2 years, if the PI is working on publications — after that period, the agency will release the manadatory final report, so that at least the core findings of the research are available. And so on.
My experience in conducting original research is now a bit dated. It applies to Germany, Austria and the UK where I worked at different times of my life. I remember vividly that in all 3 countries I had fights with non-government sponsors NOT to publish trials if the results did not suit them. In the UK, I was even threatened with a law suit. Neddless to say that I always stood my corner and published.
“Neddless to say that I always stood my corner and published.”
That might be a good example of the difference between two (or more) countries. In the U.S., privately funded research often has a proprietary quality that can limit the right of a PI to publish or otherwise use that research. This is very common in many realms, including market research/consumer research. If I am contracted by a company to discover what flavor of potato chips its potential customers prefer, they have an interest in blocking me from publishing that research in the Journal Of Market Research so that every other potato chip company can use it.
as far as I remember, there always was a contract and I just crossed out the passages that would curtail my rights to publish.
QUOTE
AllTrials (sometimes called All Trials or AllTrials.net) is a project advocating that clinical research adopt the principles of open research. The project summarizes itself as “All trials registered, all results reported”: that is, all clinical trials should be listed in a clinical trials registry, and their results should always be shared as open data.
At the center of the organisation is a petition signed by over 85,000 individuals and 599 organisations (as of August 2015):
Ben Goldacre, author of Bad Science and Bad Pharma, is a founder of the campaign and its most public spokesperson. In 2016, he participated in the launch of the OpenTrials database.
AllTrials is an international initiative of Bad Science, BMJ, Centre for Evidence-based Medicine, Cochrane Collaboration, James Lind Initiative, PLOS and Sense about Science and is being led in the US by Sense about Science USA, Dartmouth’s Geisel School of Medicine and the Dartmouth Institute for Health Policy & Clinical Practice.
Issues addressed
The project is a reaction to under-reporting of research.
…
https://en.m.wikipedia.org/wiki/AllTrials
END OF QUOTE
The whole Wikipedia article is worth reading.
Sometimes a promising line research proves to be a dead end, or leads to results that add little to the current corpus on the subject in question. In order for a researcher to be right about some important things, he or she must be prepared to be wrong about other things. Every successful researcher also has a list of failures. Researchers should not be required to publish their failures. The literature is quite cluttered as it is.
Researchers should be required to publish their failures to prevent others doing the same mistake!.