The website of BMJ Clinical Evidence seems to be popular with fans of alternative medicine (FAMs). That sounds like good news: it’s an excellent source, and one can learn a lot about EBM when studying it. But there is a problem: FAMs don’t seem to really study it (alternatively they do not have the power of comprehension to understand the data); they merely pounce on this figure and cite it endlessly:
They interpret it to mean that only 11% of what conventional clinicians do is based on sound evidence. This is water on their mills, because now they feel able to claim:
THE MAJORITY OF WHAT CONVENTIONAL CLINICIANS DO IS NOT EVIDENCE-BASED. SO, WHY DO SO-CALLED RATIONAL THINKERS EXPECT ALTERNATIVE THERAPIES TO BE EVIDENCE-BASED? IF WE NEEDED PROOF THAT THEY ARE HYPOCRITES, HERE IT IS!!!
The question is: are these FAMs correct?
The answer is: no!
They are merely using a logical fallacy (tu quoque); what is worse, they use it based on misunderstanding the actual data summarised in the above figure.
Let’s look at this in a little more detail.
The first thing we need to understand the methodologies used by ‘Clinical Evidence’ and what the different categories in the graph mean. Here is the explanation:
So, arguably the top three categories amounting to 42% signify some evidential support (if we decided to be more rigorous and merely included the two top categories, we would still arrive at 35%). This is not great, but we must remember two things here:
- EBM is fairly new;
- lots of people are working hard to improve the evidence base of medicine so that, in future, these figures will be better (by contrast, in alternative medicine, no similar progress is noticeable).
The second thing that strikes me is that, in alternative medicine, these figures would surely be much, much worse. I am not aware of reliable estimates, but I guess that the percentages might be one dimension smaller.
The third thing to mention is that the figures do not cover the entire spectrum of treatments available today but are based on ~ 3000 selected therapies. It is unclear how they were chosen, presumably the choice is pragmatic and based on the information available. If an up-to date systematic review has been published and provided the necessary information, the therapy was included. This means that the figures include not just mainstream but also plenty of alternative treatments (to the best of my knowledge ‘Clinical Evidence’ makes no distinction between the two). It is thus nonsensical to claim that the data highlight the weakness of the evidence in conventional medicine. It is even possible that the figures would be better, if alternative treatments had been excluded (I estimate that around 2 000 systematic reviews of alternative therapies have been published [I am the author of ~400 of them!]).
The fourth and possibly the most important thing to mention is that the percentage figures in the graph are certainly NOT a reflection of what percentage of treatments used in routine care are based on good evidence. In conventional practice, clinicians would, of course, select where possible those treatments with the best evidence base, while leaving the less well documented ones aside. In other words, they will use the ones in the two top categories much more frequently than those from the other categories.
At this stage, I hear some FAMs say: how does he know that?
Because several studies have been published that investigated this issue in some detail. They have monitored what percentage of interventions used by conventional clinicians in their daily practice are based on good evidence. In 2004, I reviewed these studies; here is the crucial passage from my paper:
“The most conclusive answer comes from a UK survey by Gill et al who retrospectively reviewed 122 consecutive general practice consultations. They found that 81% of the prescribed treatments were based on evidence and 30% were based on randomised controlled trials (RCTs). A similar study conducted in a UK university hospital outpatient department of general medicine arrived at comparable figures; 82% of the interventions were based on evidence, 53% on RCTs. Other relevant data originate from abroad. In Sweden, 84% of internal medicine interventions were based on evidence and 50% on RCTs. In Spain these percentages were 55 and 38%, respectively. Imrie and Ramey pooled a total of 15 studies across all medical disciplines, and found that, on average, 76% of medical treatments are supported by some form of compelling evidence — the lowest was that mentioned above (55%),6 and the highest (97%) was achieved in anaesthesia in Britain. Collectively these data suggest that, in terms of evidence-base, general practice is much better than its reputation.”
My conclusions from all this:
FAMs should study the BMJ Clinical Evidence more thoroughly. If they did, they might comprehend that the claims they tend to make about the data shown there are, in fact, bogus. In addition, they might even learn a thing or two about EBM which might eventually improve the quality of the debate.
17 Responses to Is conventional medicine based on good evidence?
Leave a Reply to Olavius Cancel reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Here is an excellent example. Also note that these people believe that integrative medicine is the “emerging mainstream”
https://www.doctorportal.com.au/mjainsight/2011/25/alternative-medicine-debate-erupts/
The chart shows 35% of treatments showing benefits in RCTs.
What relevance does that have to the question of the evidence base for treatments received in the NHS?
If doctors base their decisions upon positive RCT evidence 30% of the time, that’s a fairly solid evidence base. If another 50% of GPs decisions, as appears to be indicated by Gill et al., are based on other forms of systematic clinical evidence (e.g. pragmatic trials) then general practice is fairly described as being evidence based.
It would be good to have more information about the nature of the non-RCT evidence.
What fascinates me in the above article (in the comment) is how ‘aggresive’ these people use these numbers and with a good measure of indignation. What they are saying is that “your rubbish doesn’t work, so how dare you question us’ – when I hear this argument doing the rounds then I know I am facing an Alternative medicine proponent, although they might claim to be Complementary.
I guess the third category might include mainly cancer chemo therapeutics?
Presumably the use of parachutes to prevent gravitation-related trauma would come under “unknown effectiveness”.
It saddens me no end when I see CAM apologists using one of the great strengths of science-based medicine – its ability to be so self-critical in order to continue improving standards – as a blunt instrument to beat it with. They have a fundamental and lamentable misunderstanding of how science works. They see a graph like this and think ‘it’s all rubbish’ whereas when someone with an understanding of science sees it they will seek a true understanding of its implications and the meaning behind it, as Prof Ernst has done, and then roll their sleeves up and set about making it better.
I will always try to see the best in people but I think anyone who sees this sort of statistic as a sign science-based medicine is ‘broken’ is beyond help. They are either incapable of or unwilling to understand their real meaning. And the bottom line is whether real medicine is good bad or indifferent it still doesn’t mean homeopathy, straight-chiropractic, acupuncture and the like are effective.
Niall
very well said!
thank you
Problem is that these people view disease and treatment in a completely different way. They have invented their own definitions, usually based on ‘ancient wisdom’, and EB medicine does not fall within their scope, and hence needs to be made suspect or even ‘destroyed’. Here is another example of how they view the world, very interesting but also very dangerous.
http://www.jackafoundation.org.au/images/stories/Natural_Therapies/naturopathy.pdf
These numbers are deceiving. If one chooses specific conditions the percentages are significantly different. For example, bacterial meningitis or even pneumonia in certain populations will have a much higher percentage with improvement than is shown on the graph after appropriate treatment, where if diagnosed or presented late or if the patient is immunocompromised the results will be lower. Hypertension in some settings is close to 100% improved with treatment as are many other conditions.
Edzard you quote from your own 2004 paper Gill et al which was a paper published in 1996 in the BMJ(rather ironic) This data was from one training practice.
I don’t think that this historic data from one practice can be used to help justify that GP’s currently use 81% treatments in their practices that show effectiveness demonstrated by clear evidence.
Edzard quoted from his 2004 paper
‘The most conclusive answer comes from a UK survey by Gill et al who retrospectively reviewed 122 consecutive general practice consultations. They found that 81% of the prescribed treatments were based on evidence and 30% were based on randomised controlled trials.’
Edzard you quote from your own 2004 paper Gill et al which was a paper published in 1996 in the BMJ(rather ironic) This data was from one training practice.
I don’t think that this historic data from one practice can be used to help justify that GP’s currently use 81% treatments in their practices that show effectiveness demonstrated by clear evidence.
I AGREE, THE FIGURE WOULD MOST LIKELY BE HIGHER TODAY
Cherry picked data from some studies at single clinics in the mid 90’s gives a 80% figure that suits you Edzard. Who knows what the true figures were in the UK at the time? It may be higher now but you don’t know what the levels were 20 years ago.
Thank you for bringing this up. Have read you article and must say it confirms what I said earlier. Of 3000 treatments conventinal as well as alternative only 330 can be termed beneficial by looking at the evidence. As stated earlier the beneficial group has shrunken over the years (from 17, to 13 to 11%).
I understand your point that treatments used by conventinal doctors are/must be from the 35% group, but I cannot understand why this statement is not backed up by solid evidence.
Furthermore many published articles in BMJ, Lancet and New England Journal and other openly admits that evidence based medicine is in a state of crissis as the evidence can not be trusted. The following quote is form the book he Truth about Drug Companies by former editor of the New England Journal of Medicine Marcia Angell
“A review of 74 clinical trials of antidepressants, for example, found that 37 of 38 positive studies were published. But of the thirty-six negative studies, thirty-three were either not published or published in a form that conveyed a positive outcome (Turner, 2008)”.
” I cannot understand why this statement is not backed up by solid evidence. ”
read the post again; perhaps you overlooked the evidence I supplied?
as to the ‘state of crisis’ – it relates mostly to poor quality trials; perhaps realising this made the % figures you quote decrease?
AS THIS BLOG IS ABOUT ALT MED, LET ME ASK YOU:
is it therefore better to go with alt med?
imagine you are in a burning house; you have 2 ladders to get out; one is far from perfect, the other one is awful with missing steps etc.; which one would you take?
The reason research cannot win pseudoscience is that researchers are used to critical thinking, mostly about their own work, also reporting their concerns. Pseudo-scientist sees a bit of correlation and believes he/she invented dihydrogen-monoxide.
On an additional note, I would recommend to produce food cans with homeopathic conservatives. They can be shared with flat-earthers.
The evidence you provided is hardly solid evidence.
And as I have stated before. Most patients going to alternative treatments – which they pay for out of their own pockets – have already tried the far from perfect ladder – which are for free.
What do you mean – for free?