The website of BMJ Clinical Evidence seems to be popular with fans of alternative medicine (FAMs). That sounds like good news: it’s an excellent source, and one can learn a lot about EBM when studying it. But there is a problem: FAMs don’t seem to really study it (alternatively they do not have the power of comprehension to understand the data); they merely pounce on this figure and cite it endlessly:
They interpret it to mean that only 11% of what conventional clinicians do is based on sound evidence. This is water on their mills, because now they feel able to claim:
THE MAJORITY OF WHAT CONVENTIONAL CLINICIANS DO IS NOT EVIDENCE-BASED. SO, WHY DO SO-CALLED RATIONAL THINKERS EXPECT ALTERNATIVE THERAPIES TO BE EVIDENCE-BASED? IF WE NEEDED PROOF THAT THEY ARE HYPOCRITES, HERE IT IS!!!
The question is: are these FAMs correct?
The answer is: no!
They are merely using a logical fallacy (tu quoque); what is worse, they use it based on misunderstanding the actual data summarised in the above figure.
Let’s look at this in a little more detail.
The first thing we need to understand the methodologies used by ‘Clinical Evidence’ and what the different categories in the graph mean. Here is the explanation:
So, arguably the top three categories amounting to 42% signify some evidential support (if we decided to be more rigorous and merely included the two top categories, we would still arrive at 35%). This is not great, but we must remember two things here:
- EBM is fairly new;
- lots of people are working hard to improve the evidence base of medicine so that, in future, these figures will be better (by contrast, in alternative medicine, no similar progress is noticeable).
The second thing that strikes me is that, in alternative medicine, these figures would surely be much, much worse. I am not aware of reliable estimates, but I guess that the percentages might be one dimension smaller.
The third thing to mention is that the figures do not cover the entire spectrum of treatments available today but are based on ~ 3000 selected therapies. It is unclear how they were chosen, presumably the choice is pragmatic and based on the information available. If an up-to date systematic review has been published and provided the necessary information, the therapy was included. This means that the figures include not just mainstream but also plenty of alternative treatments (to the best of my knowledge ‘Clinical Evidence’ makes no distinction between the two). It is thus nonsensical to claim that the data highlight the weakness of the evidence in conventional medicine. It is even possible that the figures would be better, if alternative treatments had been excluded (I estimate that around 2 000 systematic reviews of alternative therapies have been published [I am the author of ~400 of them!]).
The fourth and possibly the most important thing to mention is that the percentage figures in the graph are certainly NOT a reflection of what percentage of treatments used in routine care are based on good evidence. In conventional practice, clinicians would, of course, select where possible those treatments with the best evidence base, while leaving the less well documented ones aside. In other words, they will use the ones in the two top categories much more frequently than those from the other categories.
At this stage, I hear some FAMs say: how does he know that?
Because several studies have been published that investigated this issue in some detail. They have monitored what percentage of interventions used by conventional clinicians in their daily practice are based on good evidence. In 2004, I reviewed these studies; here is the crucial passage from my paper:
“The most conclusive answer comes from a UK survey by Gill et al who retrospectively reviewed 122 consecutive general practice consultations. They found that 81% of the prescribed treatments were based on evidence and 30% were based on randomised controlled trials (RCTs). A similar study conducted in a UK university hospital outpatient department of general medicine arrived at comparable figures; 82% of the interventions were based on evidence, 53% on RCTs. Other relevant data originate from abroad. In Sweden, 84% of internal medicine interventions were based on evidence and 50% on RCTs. In Spain these percentages were 55 and 38%, respectively. Imrie and Ramey pooled a total of 15 studies across all medical disciplines, and found that, on average, 76% of medical treatments are supported by some form of compelling evidence — the lowest was that mentioned above (55%),6 and the highest (97%) was achieved in anaesthesia in Britain. Collectively these data suggest that, in terms of evidence-base, general practice is much better than its reputation.”
My conclusions from all this:
FAMs should study the BMJ Clinical Evidence more thoroughly. If they did, they might comprehend that the claims they tend to make about the data shown there are, in fact, bogus. In addition, they might even learn a thing or two about EBM which might eventually improve the quality of the debate.
The new guidelines by the American College of Physicians entitled ‘Noninvasive Treatments for Acute, Subacute, and Chronic Low Back Pain: A Clinical Practice Guideline From the American College of Physicians’ have already been the subject of the previous post. Today, I want to have a closer look at a small section of these guidelines which, I think, is crucial. It is entitled ‘HARMS OF NONPHARMACOLOGIC THERAPIES’. I have taken the liberty of copying it below:
“Evidence on adverse events from the included RCTs and systematic reviews was limited, and the quality of evidence for all available harms data is low. Harms were poorly reported (if they were reported at all) for most of the interventions.
Low-quality evidence showed no reported harms or serious adverse events associated with tai chi, psychological interventions, multidisciplinary rehabilitation, ultrasound, acupuncture, lumbar support, or traction (9,95,150,170–174). Low-quality evidence showed that when harms were reported for exercise, they were often related to muscle soreness and increased pain, and no serious harms were reported. All reported harms associated with yoga were mild to moderate (119). Low-quality evidence showed that none of the RCTs reported any serious adverse events with massage, although 2 RCTs reported soreness during or after massage therapy (175,176). Adverse events associated with spinal manipulation included muscle soreness or transient increases in pain (134). There were few adverse events reported and no clear differences between MCE and controls. Transcutaneous electrical nerve stimulation was associated with an increased risk for skin site reaction but not serious adverse events (177). Two RCTs (178,179) showed an increased risk for skin flushing with heat compared with no heat or placebo, and no serious adverse events were reported. There were no data on cold therapy. Evidence was insufficient to determine harms of electrical muscle stimulation, LLLT, percutaneous electrical nerve stimulation, interferential therapy, short-wave diathermy, and taping.”
The first thing that strikes me is the brevity of the section. Surely, guidelines of this nature must include a full discussion of the risks of the treatments in question!
The second thing that is noteworthy is the fact that the authors confirm the fact I have been banging on about for years: clinical trials of alternative therapies far too often fail to mention adverse effects. I have often pointed out that the failure to report adverse effects in clinical trials is an unacceptable violation of medical ethics. By contrast, the guideline authors seem not to feel strongly about this omission.
The third thing that is noteworthy is that the guidelines evaluate the harms of the treatments purely on the basis of the adverse effects reported in the clinical trials and systematic reviews included in their efficacy assessments. This is nonsensical for at least two reasons:
- The guideline authors themselves are aware that the trials very often fail to mention adverse effects.
- For any assessment of harm, one has to go far beyond the evidence of clinical trials, because trials tend to be too small to pick up rare adverse effects, and because they are always conducted under optimally controlled conditions where adverse effects are less likely to occur than in real life.
Together, these features of the assessment of harms explain why the guideline authors arrive at conclusions which are oddly misguided; I would even feel that they resemble a white-wash. Here are two of the most overt misjudgements:
- no harms associated with acupuncture,
- only trivial harm associated with spinal manipulations.
The best evidence we have today shows that acupuncture leads to mild adverse effects in about 10% of all cases and is also associated with very severe complications (e.g. pneumothorax, cardiac tamponade, infections, deaths) in an unknown number of patients. More details can be found for instance here, here, here and here.
And the best evidence available shows that spinal manipulation leads to moderately severe adverse effects in ~50% of all cases. In addition, we know of hundreds of cases of very severe complications resulting in stroke, permanent neurological deficits or deaths. More details can be found for instance here, here, here and here.
In the introduction, I stated that this small section of the guidelines is crucial.
The reason is simple: any responsible therapeutic decision has to be based not just on the efficacy of the treatment in question but on its risk/benefit balance. The evidence shows that the risks of some alternative therapies can be considerable, a fact that is almost totally neglected in the guidelines. Therefore, the recommendations of the new guidelines by the American College of Physicians entitled ‘Noninvasive Treatments for Acute, Subacute, and Chronic Low Back Pain: A Clinical Practice Guideline From the American College of Physicians’ are in several aspects not entirely correct and need to be reconsidered.
The website of the HOMEOPATHY HUB gives us intriguing access to the brain of a homeopath. It tells us that “protecting patient choice is at the heart of everything we do. Homeopathy, which is the second largest system of medicine in the world, is a form of treatment which plays a vital role in the lives of hundreds of thousands of people across the UK. There is, however, a movement to try and withdraw homeopathy from the public and make homeopathic medicines difficult to secure. Our intention is to be a central “hub” for accurate information on current campaigns to retain access to homeopathy and details on how you can get involved and make your voice heard. Without public and patient support we will not be successful.”
Here are a few of the above statements that I find doubtful:
- protecting patient choice – choice requires reliable information; as we will see, this is not provided here;
- second largest system of medicine in the world – really?
- plays a vital role – where is the evidence for that claim?
- movement to try and withdraw homeopathy from the public and make homeopathic medicines difficult to secure – nobody works towards this aim, some people are trying to stop wasting public funds on useless therapies, but that’s quite different, I find;
The HOMEOPATHY HUB recently alerted its readers to the fact that the Charity Commission (CC) is currently conducting a public consultation on whether organisations promoting the use of complementary and alternative medicines (CAM) should have charitable status (https://www.gov.uk/government/consultations/consultation-on-complementary-and-alternative-medicines) and urged its readers to defend homeopathy by responding to the CC offering a “few helpful points” to raise. These 7 points give, I think, a good insight into the thinking of homeopaths. I therefore copy them here and add a few of my own comments below:
- there are many types of evidence that should be considered when evaluating the effectiveness of a therapy. These include scientific studies, patient feedback and the clinical experience of doctors who have trained in a CAM discipline. Within Homeopathy there is considerable evidence which can be found (https://www.hri-research.org)
- many conventional therapies and drugs have inconclusive evidence or prove to be useful in only some cases, for example SSRIs (anti-depressants). Inconsistent evidence is often the result of the complexity of both the medical condition being treated and the therapy being used. It is not indicative of a therapy that doesn’t work
- removing all therapies or interventions that have inconsistent or inconclusive evidence would seriously limit the public and the medical profession’s ability to help treat and ease patients suffering.
- all over the world there are doctors, nurses, midwives, vets and other healthcare professional who integrate CAM therapies into their daily practice because they see effectiveness. They would not use these therapies if they did not see their patients benefitting from them. For example in the UK, within the NHS hospital setting, outcome studies demonstrate effectiveness of homeopathy. (http://www.britishhomeopathic.org/evidence/results-from-the-homeopathic-hospitals/)
- practitioners of many CAM therapies belong to registering bodies which expect their members to comply to the highest professional standards in regards to training and practice
- In the UK the producers and suppliers of CAM treatments (homeopathy, herbal medicine etc) are strictly regulated
- as well as providing valuable information to the growing number of people seeking to use CAM as part of their healthcare, CAM charities frequently fund treatment for those people, particularly the elderly and those on a low income, whose health has benefitted from these therapies but who cannot afford them. This meets the charity’s criterion of providing a public benefit.
- “Patient feedback and the clinical experience of doctors” may be important but is not what can be considered evidence of therapeutic effectiveness.
- Yes, in medicine evidence is often inconsistent; this is why we need to rely on proper assessments of the totality of the reliable data. If that fails to be positive (as is the case for homeopathy and several other forms of alternative medicine), we are well advised not to employ the treatment in question in routine healthcare.
- Removing all treatments for which the best evidence fails to show effectiveness – such as homeopathy – would greatly improve healthcare and reduces cost. It is one of the aims of EBM and an ethical imperative.
- Yes, some healthcare professionals do use useless therapies. They urgently need to be educated in the principles of EBM. Outcome studies have normally no control groups and therefore are no adequate tools for testing the effectiveness of medical interventions.
- The highest professional standards in regards to training and practice of nonsense will still result in nonsense.
- The proper regulation of nonsense can only generate proper nonsense.
- Yes, CAM charities frequently fund bogus treatments; hopefully (and with the help of readers of this blog), the CC will put an end to this soon.
I think these 7 points by the HOMEOPATHY HUB are a very poor defence of homeopathy. In fact, they are so bad that it is not worth analysing more closely than I did above. Yet, they do provide us with an insight into the homeopathic mind-set and show how illogical, misguided and wrong the arguments of homeopathy enthusiasts really are.
I do encourage you to give your response to the CC – it wound be hard to use better arguments than the homeopaths!!!
You may recall, we have dealt with the JCAM many times before; for instance here, here, here and here. Now they have come out with another remarkable paper. This study – no, the authors called it a ‘pilot study’ – was to compare the efficacy of Emotional Freedom Techniques (EFT) with that of Cognitive-Behavioral Therapy (CBT) in reducing adolescent anxiety. Sixty-three American high-ability students in grades 6–12, ages 10–18 years, who scored in the moderate to high ranges for anxiety on the Revised Children’s Manifest Anxiety Scale-2 (RCMAS-2) were randomly assigned to one of three groups:
- CBT (n = 21),
- EFT (n = 21),
- or waitlist control (n = 21).
EFT is an alternative therapy that incorporates acupoint stimulation. Students assigned to the CBT or EFT treatment groups received three individual sessions of the identified protocols from trained graduate counseling, psychology, or social work students enrolled at a large northeastern research university. The RCMAS-2 was used to assess preintervention and postintervention anxiety levels in participants.
EFT participants showed significant reduction in anxiety levels compared with the waitlist control group with a moderate to large effect size. CBT participants did not differ significantly from the EFT or control.
The authors concluded that EFT is an efficacious intervention to significantly reduce anxiety for high-ability adolescents.
They also state in their abstract that EFT is an evidence-based treatment for anxiety…
Are you happy with these conclusions?
Are you convinced that this trial lends itself to establish efficacy of anything?
Are you impressed with the trial design, the sample size, etc?
Are you sure that EFT is plausible, credible or evidence-based in any way?
If you look up EFT, you will find that there is a surprising amount of papers on it. Most of them have one thing in common: they were published in highly dubious journals. The field does not inspire trust or competence. The authors of the study state that EFT is an easily implemented strategy that uses such techniques as awareness building, exposure, reframing of interpretation, and systematic desensitization, while teaching the participant to self-stimulate protocol-identified acupoints (i.e., acupuncture points) by tapping. The effectiveness of acupuncture for treating anxiety has been well documented. Rather than using acupuncture needles, EFT relies on the manual stimulation of the acupoints. A recent meta-analysis indicated that interventions using acupoint stimulation had a moderate effect size (Hedge’s g = −0.66 95% CI [−0.99, −0.33]) in reducing symptoms. In EFT, the client stimulates the protocol-identified acupoints by tapping on them. Preliminary studies have suggested that tapping and other alternative ways of stimulating acupuncture points to be as effective as acupuncture needling. The EFT protocol and identified acupoints that were used in this study are the ones recommended for research purposes by the Association for Comprehensive Energy Psychology…
Wikipedia tells us that “Emotional Freedom Techniques (EFT) is a form of counseling intervention that draws on various theories of alternative medicine including acupuncture, neuro-linguistic programming, energy medicine, and Thought Field Therapy (TFT). It is best known through Gary Craig’s EFT Handbook, published in the late 1990s, and related books and workshops by a variety of teachers. EFT and similar techniques are often discussed under the umbrella term “energy psychology”. Advocates claim that the technique may be used to treat a wide variety of physical and psychological disorders, and as a simple form of self-administered therapy. The Skeptical Inquirer describes the foundations of EFT as “a hodgepodge of concepts derived from a variety of sources, [primarily] the ancient Chinese philosophy of chi, which is thought to be the ‘life force’ that flows throughout the body.” The existence of this life force is “not empirically supported”. EFT has no benefit as a therapy beyond the placebo effect or any known-effective psychological techniques that may be provided in addition to the purported “energy” technique. It is generally characterized as pseudoscience and it has not garnered significant support in clinical psychology.”
A recent systematic review of EFT concluded that “there were too few data available comparing EFT to standard-of-care treatments such as cognitive behavioral therapy, and further research is needed to establish the relative efficacy of EFT to established protocols.”
Notwithstanding these and many other verdicts on EFT, we now are asked to agree with the new study that EFT IS EFFICACIOUS.
Is this a joke?
They want us to believe this on the basis of a PILOT STUDY? Such studies are not even supposed to test efficacy! (Yet the authors of the trial state that this study was designed to meet the American Psychological Association (APA) Division 12 quality control criteria and the Consolidated Standards for Reporting Trials (CONSORT) criteria. I have to admit, they could have fooled me!)
No, it is not a joke, it is yet another nonsense from the ‘The Journal of Complementary and Alternative Medicine’ which, in my view, should henceforth be called THE JOURNAL OF ALTERNATIVE FACTS (JAF).
It would be wrong to call the Czech Republic the promised land for homeopathy. For instance, the only research paper by Czech authors related to the subject that I could locate was published in the Journal ‘Homeopathy‘ and, on even superficial reading, it has little to do with homeopathy. Here is the abstract:
We discovered a previously unknown phenomenon in liquid water, which develops over time when water is left to stand undisturbed, and which made precise gravimetric measurement impossible. We term this property autothixotropy (weak gel-like behaviour developing spontaneously over time) and propose a possible explanation. The results of quantitative measurements, performed by two different methods, are presented. We also report the newly discovered phenomenon of autothixotropy-hysteresis and describe the dependence of autothixotropy on the degree of molecular translative freedom. A very important conclusion is that the presence of very low concentration of salt ions, these phenomena do not occur in deionized water. Salt ions may be the determinative condition for the occurrence of the phenomena.
In fact, historically, homeopathy had had a hard time in this country. Until World War II only very few doctors practiced homeopathy on Czech territory. Dr. Quin, founder of British homeopathy, practiced a short time in the small town of Tisnov. A Catholic homeopathic hospital existed at Kromeriz since 1860. During the communist era of 1948-89, homeopathy was prohibited, and, until 1991, no books about homeopathy were available in the Czech language. More recently, about 20 titles were published by the Alternativa Publishing house. The Czech Homeopathic Medical Chamber is an organisation that only permits MDs and currently has about 1000 members. The Czech Medical Homeopathic Society has only about 300 members.
After the fall of the ‘iron curtain’, homeopathy evidently became more popular. It has recently been reported that the number of homeopathic remedies sold in the Czech Republic rose by over 50% during the past 15 years. Last year, Czechs bought homeopathic preparations for over 170 million crowns, which is 10% more than a year ago. “The patients most frequently use homeopathics against the problems associated with common viral diseases,” said Ales Krebs, deputy chairman of the Czech Pharmacy Chamber. The homeopathic flu remedy Oscillococcinum seems to be one of the most popular homeopathic preparation in the Czech Republic. Yet Czech chemists say that it is ‘absolute nonsense’.
Most physicians seem to be equally cynical about homeopathy and its practitioners: “Homeopathics are perfect drugs. The manufacturing is dirt cheap and they sell for 60 crowns. They cannot be forged because the fakes have the same effect as the original product,” Czech doctors joke about the growing interest in homeopathy. Stepan Svacina, chairman of the Czech Medical Society, says: “The doctor can use a placebo in a psychological therapy. It does not matter whether this may be a homeopathic preparation or jumping on one leg.” Another doctor is quoted as stating that “Advocates of homeopathy often argue with doctors’ conspiracy with pharmaceutical makers, but they themselves certainly do not offer their methods for free as a sort of philanthropy.”
The cost for a first consultation with a Czech homeopath ranges between 100 to 3,000 crowns. The patient pays another 800-1,000 crowns for each next examination. ($1 = 24.846 crowns)
In 2014, the Czech Republic Ministry of Health issued a press-release stating that…although the Ministry for Health of the Czech Republic does not perceive the evidence base for homeopathy to be strong enough yet, this does not prevent doctors from utilising this if it is desired and appropriate…
Because the use of homeopathy cannot ever be considered to be ‘appropriate’, this declaration could arguably be interpreted by those who insist on evidence as a new prohibition of homeopathy in the Czech Republic.
“When orthodox medicine has nothing more to offer” is the title of an article by Dr Elizabeth Thompson, a UK medical homeopath. The article was written years ago, but it is still an excellent example for disclosing the dangerously false and deeply unethical reasoning used by many alternative practitioners. The notion that all sorts of disproven treatments like homeopathy are justified when orthodox medicine has nothing more to offer is so very prevalent that I decided to do this post analysing it.
In the following, you see the most relevant sections of Dr Thompson’s original article (in normal print) and my brief comments (in brackets and in bold):
…Some people come when conventional treatments can no longer offer them anything to save their lives. This is a frightening time for them and although the homeopathic approach may not offer a cure at this late stage of their illness (Is she implying that, in some cases, homeopathy can cure cancer?), it can often offer hope of a different kind. (Surely, one does not need homeopathy for giving patients hope). Sometimes it helps people to outlive the prognosis given to them by months or even years. (A prognosis is not a precise time of death; it is based on statistics and therefore depicts a likelihood, not a certainty. Thus patients outlive their prognosis all the time regardless of treatments.) Sometimes it helps them need less (less than what? there is no control group and therefore the statement seems nonsensical) in the way of conventional medicine including pain killers and offers them continuing support despite progressive disease (is she trying to say that in conventional medicine patients with progressive disease do not get continuing support?).
As a doctor working in both conventional and complementary cancer care I have learned the importance of integrating these two perspectives (the integration of unproven therapies into EBM can only render the latter less effective). Ideally the doctor practising homeopathy would work as an integral part of a much wider team which would include family members, nurses, general practitioners, oncologists, surgeons, palliative physicians and other complementary therapists (the concept of a multi-disciplinary team for cancer is one from conventional medicine where it has long been routine). It is disappointing sometimes to see that other healthcare professionals can be unsupportive of a person’s desire to use complementary therapies and for some people the knowledge that the team is not working together can cause doubt and insecurity (for the majority of patients, however, it might be reassuring to know that their oncology-team is evidence-based).
Some patients come at the beginning of their diagnosis wanting to support their bodies with gentler (homeopathic remedies are not gentler, they are ineffective) approaches and help themselves recover from some difficult and powerful treatments such as chemotherapy and radiotherapy (Why are they being told that alternative therapies are effective in achieving these aims when there is no good evidence to show that this is true? Isn’t that unethical?). As well as using homeopathic medicines (no good evidence of effectiveness!!!), the GHH also has experience in using Mistletoe which is given by injection and has been shown to stimulate the group of white cells whose numbers can be depleted during chemotherapy and radiotherapy (also no good evidence that it works clinically!!!).
Other patients come when they have finished most of their treatments but may still not be feeling well despite being given the all clear by their doctors (same again: no good evidence!!!)…
One wonderful aspect of the homeopathic approach is that it can be a very important opportunity to help someone re-evaluate their life and their health (We don’t need to prescribe placebos for that, this aim is better reached by employing a clinical psychologist).
Sometimes hurts in the past have never been healed and sitting with someone as they describe difficult experiences can be itself therapeutic. Combining this therapeutic listening time with substances from nature that gently stimulate the body’s own healing potential (where is the evidence for that claim?) can be an approach that through patient demand and research (what research?) we can demonstrate is really worth offering to many more people…
END OF QUOTE
This text shows in an exemplary fashion how desperate patients can be convinced to make dramatically wrong choices. If you read Dr Thompson’s text without my comments, it probably sounds fairly reasonable to many people. I can understand why patients and carers end up thinking that homeopathy or other disproven therapies are reasonable options WHEN ORTHODOX MEDICINE HAS NOTHING MORE TO OFFER.
But the claim of homeopaths and others that mainstream medicine has, in certain cases, nothing more to offer is demonstrably wrong. Supportive and palliative care are established and important parts of conventional medicine. To deny this fact amounts to a lie! The implied scenario where a patient is told by her oncology team: “sorry but we cannot do anything else for you”, does quite simply not exist. The argument is nothing else but a straw-man – and a vicious one at that.
Moreover, the subsequent argument of homeopaths, “as ‘they’ have given you up, we now offer you our effective homeopathic remedies”, is not supported by good evidence. In other words, one lie is added to another. To call this unethical, would be the understatement of the year, I think.
Guest post by Pete Attkins
Commentator “jm” asked a profound and pertinent question: “What DOES it take for people to get real in this world, practice some common sense, and pay attention to what’s going on with themselves?” This question was asked in the context of asserting that personal experience always trumps the results of large-scale scientific experiments; and asserting that alt-med experts are better able to provide individulized healthcare than 21st Century orthodox medicine.
What does common sense and paying attention lead us to conclude about the following? We test a six-sided die for bias by rolling it 100 times. The number 1 occurs only once and the number 6 occurs many times, never on its own, but in several groups of consecutive sixes.
I think it is reasonable to say that common sense would, and should, lead everyone to conclude that the die is biased and not fit for its purpose as a source of random numbers.
In other words, we have a gut feeling that the die is untrustworthy. Gut instincts and common sense are geared towards maximizing our chances of survival in our complex and unpredictable world — these are innate and learnt behaviours that have enabled humans to survive despite the harshness of our ever changing habitat.
Only very recently in the long history of our species have we developed specialized tools that enable us to better understand our harsh and complex world: science and critical thinking. These tools are difficult to master because they still haven’t been incorporated into our primary and secondary formal education systems.
The vast majority of people do not have these skills therefore, when a scientific finding flies in the face of our gut instincts and/or common sense, it creates an overwhelming desire to reject the finding and classify the scientist(s) as being irrational and lacking basic common sense. It does not create an intense desire to accept the finding then painstakingly learn all of the science that went into producing the finding.
With that in mind, let’s rethink our common sense conclusion that the six-sided die is biased and untrustworthy. What we really mean is that the results have given all of us good reason to be highly suspicious of this die. We aren’t 100% certain that this die is biased, but our gut feeling and common sense are more than adequate to form a reasonable mistrust of it and to avoid using it for anything important to us. Reasons to keep this die rather than discard it might be to provide a source of mild entertainment or to use its bias for the purposes of cheating.
Some readers might be surprised to discover at this point that the results I presented from this apparently heavily-biased die are not only perfectly valid results obtained from a truly random unbiased die, they are to be fully expected. Even if the die had produced 100 sixes in that test, it would not confirm that the die is biased in any way whatsoever. Rolling a truly unbiased die once will produce one of six possible outcomes. Rolling the same die 100 times will produce one unique sequence out of the 6^100 (6.5 x 10^77) possible sequences: all of which are equally valid!
Gut feeling plus common sense rightfully informs us that the probability of a random die producing one hundred consecutive sixes is so incredibly remote that nobody will ever see it occur in reality. This conclusion is also mathematically sound: if there were 6.5 x 10^77 people on Earth, each performing the same test on truly random dice, there is no guarantee that anyone would observe a sequence of one hundred consecutive sixes.
When we observe a sequence such as 2 5 1 4 6 3 1 4 3 6 5 2… common sense informs us that the die is very likely random. If we calculate the arithmetic mean to be very close to 3.5 then common sense will lead us to conclude that the die is both random and unbiased enough to use it as a reliable source of random numbers.
Unfortunately, this is a perfect example of our gut feelings and common sense failing us abysmally. They totally failed to warn us that the 2 5 1 4 6 3 1 4 3 6 5 2… sequence we observed had exactly the same (im)probability of occurring as a sequence of one hundred 6s or any other sequence that one can think of that doesn’t look random to a human observer.
The 100-roll die test is nowhere near powerful enough to properly test a six-sided die, but this test is more than adequately powered to reveal some of our cognitive biases and some of the deficits in our personal mastery of science and critical thinking.
To properly test the die we need to provide solid evidence that it is both truly random and that its measured bias tends towards zero as the number of rolls tends towards infinity. We could use the services of one testing lab to conduct billions of test rolls, but this would not exclude errors caused by such things as miscalibrated equipment and experimenter bias. It is better to subdivide the testing across multiple labs then carefully analyse and appropriately aggregate the results: this dramatically reduces errors caused by equipment and humans.
In medicine, this testing process is performed via systematic reviews of multiple, independent, double-blind, placebo-controlled trials — every trial that is insufficiently powered to add meaningfully to the result is rightfully excluded from the aggregation.
Alt-med relies on a diametrically opposed testing process. It performs a plethora of only underpowered tests; presents those that just happen to show a positive result (just as a random die could’ve produced); and sweeps under the carpet the overwhelming number of tests that produced a negative result. It publishes only the ‘successes’, not its failures. By sweeping its failures under the carpet it feels justified in making the very bold claim: Our plethora of collected evidence shows clearly that it mostly ‘works’ and, when it doesn’t, it causes no harm.
One of the most acidic tests for a hypothesis and its supporting data (which is a mandatory test in a few branches of critical engineering) is to substitute the collected data for random data that has been carefully crafted to emulate the probability mass functions of the collected datasets. This test has to be run multiple times for reasons that I’ve attempted to explain in my random die example. If the proposer of the hypothesis is unable to explain the multiple failures resulting from this acid test then it is highly likely that the proposer either does not fully understand their hypothesis or that their hypothesis is indistinguishable from the null hypothesis.
Complementary treatments have become a popular (and ‘political correct’) option to keep desperate cancer patients happy. But how widely accepted is their use in oncology units? A brand-new article tried to find the answer to this question.
The principal aim of this survey was to map centres across Europe prioritizing those that provide public health services and operating within the national health system in integrative oncology (IO). A cross-sectional descriptive survey design was used to collect data. A questionnaire was elaborated concerning integrative oncology therapies to be administered to all the national health system oncology centres or hospitals in each European country. These institutes were identified by convenience sampling, searching on oncology websites and forums. The official websites of these structures were analysed to obtain more information about their activities and contacts.
Information was received from 123 (52.1 %) out of the 236 centres contacted until 31 December 2013. Forty-seven out of 99 responding centres meeting inclusion criteria (47.5 %) provided integrative oncology treatments, 24 from Italy and 23 from other European countries. The number of patients seen per year was on average 301.2 ± 337. Among the centres providing these kinds of therapies, 33 (70.2 %) use fixed protocols and 35 (74.5 %) use systems for the evaluation of results. Thirty-two centres (68.1 %) had research in progress or carried out until the deadline of the survey. The complementary and alternative medicines (CAMs) more frequently provided to cancer patients were acupuncture 26 (55.3 %), homeopathy 19 (40.4 %), herbal medicine 18 (38.3 %) and traditional Chinese medicine 17 (36.2 %); anthroposophic medicine 10 (21.3 %); homotoxicology 6 (12.8 %); and other therapies 30 (63.8 %). Treatments are mainly directed to reduce adverse reactions to chemo-radiotherapy (23.9 %), in particular nausea and vomiting (13.4 %) and leucopenia (5 %). The CAMs were also used to reduce pain and fatigue (10.9 %), to reduce side effects of iatrogenic menopause (8.8 %) and to improve anxiety and depression (5.9 %), gastrointestinal disorders (5 %), sleep disturbances and neuropathy (3.8 %).
As so often with surveys of this nature, the high non-response rate creates a problem: it is not unreasonable to assume that those centres that responded had an interest in IO, while those that failed to respond tended to have none. Thus the figures reported here for the usage of alternative therapies might be far higher than they actually are. One can only hope that this is the case. The idea that 40% of all cancer patients receive homeopathy, for instance, is hardly one that is in accordance with the principles of evidence-based practice.
The list of medical reasons for using largely unproven treatments is interesting, I think. I am not aware of lots of strong evidence to show that any of the treatments in question would generate more good than harm for any of the conditions in question.
What follows from all of this is worrying, in my view: thousands of desperate cancer patients are being duped into having bogus treatments paid for by their national health system. This, I think, begs the question whether these most vulnerable patients do not deserve better.
Getting good and experienced lecturers for courses is not easy. Having someone who has done more research than most working in the field and who is internationally known, might therefore be a thrill for students and an image-boosting experience of colleges. In the true Christmas spirit, I am today making the offer of being of assistance to the many struggling educational institutions of alternative medicine .
A few days ago, I tweeted about my willingness to give free lectures to homeopathic colleges (so far without response). Having thought about it a bit, I would now like to extend this offer. I would be happy to give a free lecture to the students of any educational institution of alternative medicine. I suggest to
- do a general lecture on the clinical evidence of the 4 major types of alternative medicine (acupuncture, chiropractic, herbal medicine, homeopathy) or
- give a more specific lecture with in-depth analyses of any given alternative therapy.
I imagine that most of the institutions in question might be a bit anxious about such an idea, but there is no need to worry: I guarantee that everything I say will be strictly and transparently evidence-based. I will disclose my sources and am willing to make my presentation available to students so that they can read up the finer details about the evidence later at home. In other words, I will do my very best to only transmit the truth about the subject at hand.
Nobody wants to hire a lecturer without having at least a rough outline of what he will be talking about – fair enough! Here I present a short summary of the lecture as I envisage it:
- I will start by providing a background about myself, my qualifications and my experience in researching and lecturing on the matter at hand.
- This will be followed by a background on the therapies in question, their history, current use etc.
- Next I would elaborate on the main assumptions of the therapies in question and on their biological plausibility.
- This will be followed by a review of the claims made for the therapies in question.
- The main section of my lecture would be to review the clinical evidence regarding the efficacy of therapies in question. In doing this, I will not cherry-pick my evidence but rely, whenever possible, on authoritative systematic reviews, preferably those from the Cochrane Collaboration.
- This, of course, needs to be supplemented by a review of safety issues.
- If wanted, I could also say a few words about the importance of the placebo effect.
- I also suggest to discuss some of the most pertinent ethical issues.
- Finally, I would hope to arrive at a few clear conclusions.
You see, all is entirely up to scratch!
Perhaps you have some doubts about my abilities to lecture? I can assure you, I have done this sort of thing all my life, I have been a professor at three different universities, and I will probably manage a lecture to your students.
A final issue might be the costs involved. As I said, I would charge neither for the preparation (this can take several days depending on the exact topic), nor for the lecture itself. All I would hope for is that you refund my travel (and, if necessary over-night) expenses. And please note: this is time-limited: approaches will be accepted until 1 January 2015 for lectures any time during 2015.
I can assure you, this is a generous offer that you ought to consider seriously – unless, of course, you do not want your students to learn the truth!
(In which case, one would need to wonder why not)
Most of the underlying assumptions of alternative medicine (AM) lack plausibility. Whenever this is the case, so the argument put forward by an international team of researchers in a recent paper, there are difficulties involved in obtaining a valid statistical significance in clinical studies.
Using a mostly statistical approach, they argue that, since the prior probability of a research hypothesis is directly related to its scientific plausibility, the commonly used frequentist statistics, which do not account for this probability, are unsuitable for studies exploring matters in various degree disconnected from science. Any statistical significance obtained in this field should be considered with great caution and may be better applied to more plausible hypotheses (like placebo effect) than the specific efficacy of the intervention.
The researchers conclude that, since achieving meaningful statistical significance is an essential step in the validation of medical interventions, AM practices, producing only outcomes inherently resistant to statistical validation, appear not to belong to modern evidence-based medicine.
To emphasize their arguments, the researchers make the following additional points:
- It is often forgotten that frequentist statistics, commonly used in clinical trials, provides only indirect evidence in support of the hypothesis examined.
- The p-value inherently tends to exaggerate the support for the hypothesis tested, especially if the scientific plausibility of the hypothesis is low.
- When the rationale for a clinical intervention is disconnected from the basic principles of science, as in case of complementary alternative medicines, any positive result obtained in clinical studies is more reasonably ascribable to hypotheses (generally to placebo effect) other than the hypothesis on trial, which commonly is the specific efficacy of the intervention.
- Since meaningful statistical significance as a rule is an essential step to validation of a medical intervention, complementary alternative medicine cannot be considered evidence-based.
Further explanations can be found in the discussion of the article where the authors argue that the quality of the hypothesis tested should be consistent with sound logic and science and therefore have a reasonable prior probability of being correct. As a rule of thumb, assuming a “neutral” attitude towards the null hypothesis (odds = 1:1), a p-value of 0.01 or, better, 0.001 should suffice to give a satisfactory posterior probability of 0.035 and 0.005 respectively.
In the area of AM, hypotheses often are entirely inconsistent with logic and frequently fly in the face of science. Four examples can demonstrate this instantly and sufficiently, I think:
- Homeopathic remedies which contain not a single ‘active’ molecule are not likely to generate biological effects.
- Healing ‘energy’ of Reiki masters has no basis in science.
- Meridians of acupuncture are pure imagination.
- Chiropractic subluxation have never been shown to exist.
Positive results from clinical trials of implausible forms of AM are thus either due to chance, bias or must be attributed to more credible causes such as the placebo effect. Since the achievement of meaningful statistical significance is an essential step in the validation of medical interventions, unless some authentic scientific support to AM is provided, one has to conclude that AM cannot be considered as evidence-based.
Such arguments are by no means new; they have been voiced over and over again. Essentially, they amount to the old adage: IF YOU CLAIM THAT YOU HAVE A CAT IN YOUR GARDEN, A SIMPLE PICTURE MAY SUFFICE. IF YOU CLAIM THERE IS A UNICORN IN YOUR GARDEN, YOU NEED SOMETHING MORE CONVINCING. An extraordinary claim requires an extraordinary proof! Put into the context of the current discussion about AM, this means that the usual level of clinical evidence is likely to be very misleading as long as it totally neglects the biological plausibility of the prior hypothesis.
Proponents of AM do not like to hear such arguments. They usually insist on what we might call a ‘level playing field’ and fail to see why their assumptions require not only a higher level of evidence but also a reasonable scientific hypothesis. They forget that the playing field is not even to start with; to understand the situation better, they should read this excellent article. Perhaps its elegant statistical approach will convince them – but I would not hold my breath.