A study by an international team of researchers estimated the proportion of healthcare interventions tested within Cochrane Reviews that are effective according to high-quality evidence.
They selected a random sample of 2428 (35%) of all Cochrane Reviews published between 1 January 2008 and 5 March 2021 and extracted data about interventions within these reviews that were compared with placebo, or no treatment, and whose outcome quality was rated using Grading of Recommendations Assessment, Development and Evaluation (GRADE). They then calculated the proportion of interventions whose effectiveness was based on high-quality evidence according to GRADE, had statistically significant positive effects and were judged as beneficial by the review authors. They also calculated the proportion of interventions that suggested harm.
Of 1567 eligible interventions, 87 (5.6%) had high-quality evidence on first-listed primary outcomes, positive, statistically significant results, and were rated by review authors as beneficial. Harms were measured for 577 (36.8%) interventions, 127 of which (8.1%) had statistically significant evidence of harm. Our dependence on the reliability of Cochrane author assessments (including their GRADE assessments) was a potential limitation of our study.
The authors drew the following conclusions: While many healthcare interventions may be beneficial, very few have high-quality evidence to support their effectiveness and safety. This problem can be remedied by high-quality studies in priority areas. These studies should measure harms more frequently and more rigorously. Practitioners and the public should be aware that many frequently used interventions are not supported by high-quality evidence.
Proponents of so-called alternative medicine (SCAM) are fond of the ‘strawman’ fallacy; meaning they like to present a picture of conventional medicine that is overtly negative in order for SCAM to appear more convincing (Prince Charles, for instance, uses this trick every single time he speaks about SCAM). Therefore I am amazed that this paper has not been exploited in that way by them. I was expecting headlines such as
Evidence-based medicine is not supported by evidence
Less than 6% of all conventional treatments are supported by sound evidence.
Why did they not have a field day with this new paper then?
As the article is behind a paywall, it took me a while to get the full paper (thanks Paul). Now that I have read it, I think I understand the reason.
In the article, the authors provide figures for specific types of treatments. Let me show you some of the percentages of interventions that met the primary outcome (high quality, statistically significant effect, and authors interpret as effective):
- pharmacological 73.8%
- surgical 4.6%
- exercise 5.8%
- diet 1.2%
- alternative 0.0%
- manual therapies 0.0%
So, maybe the headlines should not be any of the above but:
No good evidence to support SCAM?
SCAM is destroying the evidence base of medicine.