This seems to be the question that occupies the minds of several homeopaths.
So was I!
Let me explain.
In 1997, Linde et al published their now famous meta-analysis of clinical trials of homeopathy which concluded that “The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo. However, we found insufficient evidence from these studies that homeopathy is clearly efficacious for any single clinical condition. Further research on homeopathy is warranted provided it is rigorous and systematic.”
This paper had several limitations which Linde was only too happy to admit. The authors therefore conducted a re-analysis which, even though published in an excellent journal, is rarely cited by homeopaths. Linde et al stated in their re-analysis of 2000: “there was clear evidence that studies with better methodological quality tended to yield less positive results.” It was this phenomenon that prompted me and my colleague Max Pittler to publish a ‘letter to the editor’ which now – 15 years later – seems the stone of homeopathic contention.
A blog-post by a believer in homeopathy even asks the interesting question: Did Professor Ernst Sell His Soul to Big Pharma? It continues as follows:
Edzard Ernst is an anti-homeopath who spent his career attacking traditional medicine. In 1993 he became Professor of Complementary Medicine at the University of Exeter. He is often described as the first professor of complementary medicine, but the title he assumed should have fooled no-one. His aim was to discredit medical therapies, notably homeopathy, and he then published some 700 papers in ‘scientific’ journals to do so.
Now, Professor Robert Hahn, in his blog, has made an assessment of the quality of his work… In the interests of the honesty and integrity in science, it is an important assessment. It shows, in his view, how science has been taken over by ideology (or as I would suggest, more accurately, the financial interests of Big Corporations, in this case, Big Pharma). The blog indicates that in order to demonstrate that homeopathy is ineffective, over 95% of scientific research into homeopathy has to be discarded or removed!
So for those people who, like myself, cannot read the original German, here is an English translation of the blog…
“I have never seen a science writer so blatantly biased as Edzard Ernst: his work should not be considered of any worth at all, and discarded” finds Sweden’s Professor Robert Hahn, a leading medical scientist, physician, and Professor of Anaesthesia and Intensive Care at the University of Linköping, Sweden.
Hahn determined therefore to analyze for himself the ‘research’ which supposedly demonstrated homeopathy to be ineffective, and reached the shocking conclusion that:
“only by discarding 98% of homeopathy trials and carrying out a statistical meta-analysis on the remaining 2% negative studies, can one ‘prove’ that homeopathy is ineffective”.
In other words, all supposedly negative homeopathic meta-analyses which opponents of homeopathy have relied on, are scientifically bogus…
Who can you trust? We can begin by disregarding Edzard Ernst. I have read several other studies that he has published, and they are all untrustworthy. His work should be discarded…
In the case of homeopathy, one should stick with what the evidence reveals. And the evidence is that only by removing 95-98% of all studies is the effectiveness of homeopathy not demonstrable…
So, now you are wondering, I am sure: HOW MUCH DID HE GET FOR SELLING HIS SOUL TO BIG PHARMA?
No? You are wondering 1) who this brilliant Swedish scientist, Prof Hahn, is and 2) what article of mine he is criticising? Alright, I will try to enlighten you.
Here I can rely on a comment posted on my blog some time ago by someone who can read Swedish (thank you Bjorn). He commented about Hahn as follows:
A renowned director of medical research with well over 300 publications on anesthesia and intensive care and 16 graduated PhD students under his mentorship, who has been leading a life on the side, blogging and writing about spiritualism, and alternative medicine and now ventures on a public crusade for resurrecting the failing realm of homeopathy!?! Unbelievable!
I was unaware of this person before, even if I have lived and worked in Sweden for decades.
I have spent the evening looking up his net-track and at his blog at roberthahn.nu (in Swedish).
I will try to summarise some first impressions:
Hahn is evidently deeply religious and there is the usual, unmistakably narcissistic aura over his writings and sayings. He is religiously confident that there is more to this world than what can be measured and sensed. In effect, he seems to believe that homeopathy (as well as alternative medical methods in general) must work because there are people who say they have experienced it and denying the possibility is akin to heresy (not his wording but the essence of his writing).
He has, along with his wife, authored at least three books on spiritual matters with titles such as (my translations) “Clear replies from the spiritual world” and “Connections of souls”.
He has a serious issue with skeptics and goes on at length about how they are dishonest bluffers[sic] who willfully cherry-pick and misinterpret evidence to fit their preconceived beliefs.
He feels that desperate patients should generally be allowed the chance that alternative methods may offer.
He believes firmly in former-life memories, including his own, which he claims he has found verification for in an ancient Italian parchment.
His main arguments for homeopathy are Claus Linde’s meta analyses and the sheer number of homeopathic research that he firmly believes shows it being superior to placebo, a fact that (in his opinion) shows it has a biological effect. Shang’s work from 2005 he dismisses as seriously flawed.
He also points to individual research like this as credible proof of the biologic effect of remedies.
He somewhat surprisingly denies recommending homeopathy despite being convinced of its effect and maintains that he wants better, more problem oriented and disease specific studies to clarify its applicability. (my interpretation)
If it weren’t for his track record of genuine, acknowledged medical research and him being a renowned authority in a genuine, scientific medical field, this man would be an ordinary, religiously devout quack.
What strikes me as perhaps telling of a consequence of his “exoscientific” activity, is that Hahn, who holds the position of research director at a large city trauma and emergency hospital is an “adjungerad professor”, which is (usually) a part time, time limited, externally financed professorial position, while any Swedish medical doctor with his very extensive formal merits would very likely hold a full professorship at an academic institution.
END OF QUOTE
MY 2000 PAPER THAT SEEMS TO IRRITATE HAHN
This was a short ‘letter to the editor’ by Ernst and Pittler published in the J Clin Epidemiol commenting on the above-mentioned re-analysis by Linde et al which was published in the same journal. As its text is not available on-line, I re-type parts of it here:
In an interesting re-analysis of their meta-analysis of clinical trials of homeopathy, Linde et al conclude that there is no linear relationship between quality scores and study outcome. We have simply re-plotted their data and arrive at a different conclusion. There is an almost perfect correlation between the odds ratio and the Jadad score between the range of 1-4… [some technical explanations follow which I omit]…Linde et al can be seen as the ultimate epidemiological proof that homeopathy is, in fact, a placebo.
And that is, as far as I can see, the whole mysterious story. I cannot even draw a conclusion – all I can do is to ask a question:
DOES ANYONE UNDERSTAND WHAT THEY ARE GOING ON ABOUT?
Being constantly on the look-out for new, good quality articles on alternative therapy which suggest that a treatment might actually work, I was excited to find not just one or two but four recent publications on an old favourite of mine: massage therapy.
The first paper described a study aimed to investigate the effect of whole body massage on the vital signs, Glasgow Coma Scale (GCS) scores and arterial blood gases (ABG) in trauma ICU patients.
In a randomized, double-blind trial, 108 trauma ICU patients received whole body massage or routine care only. The patients vital signs; systolic blood pressure (SBP), diastolic blood pressure (DBP), respiratory rate (RR), pulse rate (PR), Temperature (T), GCS score and ABG parameters were measured in both groups before the intervention and 1 hour and 3 hours after the intervention. The patient in experimental group received full body massage in 45 minute by a family member.
Significant differences were observed between experimental and control groups in SBP 1 hour and 3 hours after intervention, DBP, RR and PR 1 hour after intervention, and GCS 1 hour and 3 hours after intervention. Significant differences were also observed between experimental and control groups in O2 saturation, PH and pO2. No significant differences between experimental and control groups were noted in Temperature, pCO2 and HCO3.
The authors concluded that massage therapy is a safe and effective treatment in intensive care units to reduce patient’s physical and psychological problems. Therefore the use of massage therapy is recommended to clinical practice as a routine method.
The second paper reported a clinical trial on 66 male and female nurses working in intensive care units of Isfahan University of Medical Sciences, Iran.
Patients were randomly divided into experimental and control groups. The Occupational Stress Inventory (OSI) (Osipow and Spokane, 1987) was completed by participants of the two groups before, immediately after, and 2 weeks after the intervention. Swedish massage was performed on participants of the experimental group for 25 min in each session, twice a week for 4 weeks.
Results showed a significant difference in favour of the massage therapy in overall mean occupation stress scores between experimental and control groups two weeks after the intervention.
The authors concluded that it is recommended that massage, as a valuable noninvasive method, be used for nurses in intensive care units to reduce their stress, promote mental health, and prevent the decrease in quality of nursing work life.
The third paper described a randomized controlled trial evaluating the effects of post-operative massage in patients undergoing abdominal colorectal surgery.
One hundred twenty-seven patients were randomized to receive a 20-min massage or social visit and relaxation session on postoperative days 2 and 3. Vital signs and psychological well-being (pain, tension, anxiety, satisfaction with care, relaxation) were assessed before and after each intervention.
Post-operative massage significantly improved the patients’ perception of pain, tension, and anxiety, but overall satisfaction was unchanged.
The authors concluded that massage may be beneficial during postoperative recovery for patients undergoing abdominal colorectal surgery. Further studies are warranted to optimize timing and duration and to determine other benefits in this clinical setting.
The fourth paper reported a systematic review was to evaluate the effectiveness of massage on the short- and long-term outcomes of pre-term infants.
Literature searches were conducted using the PRISMA framework. Validity of included studies was assessed using criteria defined by the Cochrane Collaboration. Assessments were carried out independently by two reviewers with a third reviewer to resolve differences.
Thirty-four studies met the inclusion criteria, 3 were quasi-experimental, 1 was a pilot study, and the remaining 30 were RCTs. The outcomes that could be used in the meta-analysis and found in more than three studies suggested that massage improved daily weight gain by 0.53 g, and resulted in a significant improvement in mental scores by 7.89 points. There were no significant effects on length of hospital stay, caloric intake, or weight at discharge. Other outcomes were not analyzed either because the units of measurement varied between studies, or because means and standard deviations were not provided by the authors. The quality of the studies was variable with methods of randomization and blinding of assessment unclear in 18 of the 34 trials.
The authors concluded that massage therapy could be a comforting measure for infants in the NICU to improve weight gain and enhance mental development. However, the high heterogeneity, the weak quality in some studies, and the lack of a scientific association between massage and developmental outcomes preclude making definite recommendations and highlight the need for further RCTs to contribute to the existing body of knowledge.
I am not saying that these articles are flawless, nor that I agree with all of their conclusion. What I am trying to indicate is that we finally have here an alternative therapy that is promising.
When I worked in Germany and later in Austria, massage was considered to be entirely mainstream. It was only after I had moved to the UK when I realised that, in English-speaking countries, it is mostly considered to be alternative. Perhaps this classification is wrong?
Perhaps we should differentiate according to what type of massage we are talking about. In the realm of alternative medicine – and not just there, I suppose – this seems good advice indeed.
The above papers are about classical massage therapy, but there are some types pf massage which are less than conventional: aura-massage, Marma massage, Indian head massage, shiatsu etc. etc. the list seems endless. These are alternative in more than one sense, and they have one thing in common: there is, as far as I can see, no good evidence to show that they do anything to human health.
My conclusion therefore is that, even with something as common as massage therapy, we need to be careful not to be roped in by the charlatans.
Of all alternative treatments, aromatherapy (i.e. the application of essential oils to the body, usually by gentle massage or simply inhalation) seems to be the most popular. This is perhaps understandable because it certainly is an agreeable form of ‘pampering’ for someone in need of come TLC. But is aromatherapy more than that? Is it truly a ‘THERAPY’?
A recent systematic review was aimed at evaluating the existing data on aromatherapy interventions as a means of improving the quality of sleep. Electronic literature searches were performed to identify relevant studies published between 2000 and August 2013. Randomized controlled and quasi-experimental trials that included aromatherapy for the improvement of sleep quality were considered for inclusion. Of the 245 publications identified, 13 studies met the inclusion criteria, and 12 studies could be used for a meta-analysis.
The meta-analysis of the 12 studies revealed that the use of aromatherapy was effective in improving sleep quality. Subgroup analysis showed that inhalation aromatherapy was more effective than aromatherapy applied via massage.
The authors concluded that readily available aromatherapy treatments appear to be effective and promote sleep. Thus, it is essential to develop specific guidelines for the efficient use of aromatherapy.
Perfect! Let’s all rush out and get some essential oils for inhalation to improve our sleep (remarkably, the results imply that aroma therapists are redundant!).
Not so fast! As I see it, there are several important caveats we might want to consider before spending our money this way:
- Why did this review focus on such a small time-frame? (Systematic reviews should include all the available evidence of a pre-defined quality.)
- The quality of the included studies was often very poor, and therefore the overall conclusion cannot be definitive.
- The effect size of armoatherapy is small. In 2000, we published a similar review and concluded that aromatherapy has a mild, transient anxiolytic effect. Based on a critical assessment of the six studies relating to relaxation, the effects of aromatherapy are probably not strong enough for it to be considered for the treatment of anxiety. The hypothesis that it is effective for any other indication is not supported by the findings of rigorous clinical trials.
- It seems uncertain which essential oil is best suited for this indication.
- Aromatherapy is not always entirely free of risks. Another of our reviews showed that aromatherapy has the potential to cause adverse effects some of which are serious. Their frequency remains unknown. Lack of sufficiently convincing evidence regarding the effectiveness of aromatherapy combined with its potential to cause adverse effects questions the usefulness of this modality in any condition.
- There are several effective ways for improving sleep when needed; we need to know how aromatherapy compares to established treatments for that indication.
All in all, I think stronger evidence is required that aromatherapy is more that pampering.
One could define alternative medicine by the fact that it is used almost exclusively for conditions for which conventional medicine does not have an effective and reasonably safe cure. Once such a treatment has been found, few patients would look for an alternative.
Alzheimer’s disease (AD) is certainly one such condition. Despite intensive research, we are still far from being able to cure it. It is thus not really surprising that AD patients and their carers are bombarded with the promotion of all sorts of alternative treatments. They must feel bewildered by the choice and all too often they fall victim to irresponsible quacks.
Acupuncture is certainly an alternative therapy that is frequently claimed to help AD patients. One of the first websites that I came across, for instance, stated boldly: acupuncture improves memory and prevents degradation of brain tissue.
But is there good evidence to support such claims? To answer this question, we need a systematic review of the trial data. Fortunately, such a paper has just been published.
The objective of this review was to assess the effectiveness and safety of acupuncture for treating AD. Eight electronic databases were searched from their inception to June 2014. Randomized clinical trials (RCTs) with AD treated by acupuncture or by acupuncture combined with drugs were included. Two authors extracted data independently.
Ten RCTs with a total of 585 participants were included in a meta-analysis. The combined results of 6 trials showed that acupuncture was better than drugs at improving scores on the Mini Mental State Examination (MMSE) scale. Evidence from the pooled results of 3 trials showed that acupuncture plus donepezil was more effective than donepezil alone at improving the MMSE scale score. Only 2 trials reported the incidence of adverse reactions related to acupuncture. Seven patients had adverse reactions related to acupuncture during or after treatment; the reactions were described as tolerable and not severe.
The Chinese authors of this review concluded that acupuncture may be more effective than drugs and may enhance the effect of drugs for treating AD in terms of improving cognitive function. Acupuncture may also be more effective than drugs at improving AD patients’ ability to carry out their daily lives. Moreover, acupuncture is safe for treating people with AD.
Anyone reading this and having a friend or family member who is affected by AD will think that acupuncture is the solution and warmly recommend trying this highly promising option. I would, however, caution to remain realistic. Like so very many systematic reviews of acupuncture or other forms of TCM that are currently flooding the medical literature, this assessment of the evidence has to be taken with more than just a pinch of salt:
- As far as I can see, there is no biological plausibility or mechanism for the assumption that acupuncture can do anything for AD patients.
- The abstract fails to mention that the trials were of poor methodological quality and that such studies tend to generate false-positive findings.
- The trials had small sample sizes.
- They were mostly not blinded.
- They were mostly conducted in China, and we know that almost 100% of all acupuncture studies from that country draw positive conclusions.
- Only two trials reported about adverse effects which is, in my view, a sign of violation of research ethics.
As I already mentioned, we are currently being flooded with such dangerously misleading reviews of Chinese primary studies which are of such dubious quality that one could do probably nothing better than to ignore them completely.
Isn’t that a bit harsh? Perhaps, but I am seriously worried that such papers cause real harm:
- They might motivate some to try acupuncture and give up conventional treatments which can be helpful symptomatically.
- They might prompt some families to spend sizable amounts of money for no real benefit.
- They might initiate further research into this area, thus drawing money away from research into much more promising avenues.
IT IS HIGH TIME THAT RESEARCHERS START THINKING CRITICALLY, PEER-REVIEWERS DO THEIR JOB PROPERLY, AND JOURNAL EDITORS STOP PUBLISHING SUCH MISLEADING ARTICLES.
Nonspecific neck pain is extremely common, often disabling, and very costly for us all. If we believe those who earn their money with them, effective treatments for the condition abound. One of these therapies is osteopathy. But does osteopathic manipulation/mobilisation really work?
The objective of a recent review (the link I originally put in here does not work, I will supply a new one as soon as the article becomes available on Medline) was to find out. Specifically, the authors wanted to assess the effectiveness of osteopathic manipulative treatment (OMT) in the management of chronic nonspecific neck pain regarding pain, functional status, and adverse events.
Electronic literature searches unrestricted by language were performed in March 2014. A manual search of reference lists and personal communication with experts identified additional studies. Only randomized clinical trials (RCTs) were included, and studies of specific neck pain or single treatment techniques were excluded. Primary outcomes were pain and functional status, and secondary outcome was adverse events.
Studies were independently reviewed using a standardized data extraction form. Mean difference (MD) or standard mean difference (SMD) with 95% confidence intervals (CIs) and overall effect size were calculated for primary outcomes. GRADE was used to assess quality of the evidence.
Of 299 identified articles, 18 were evaluated and 15 excluded. The three included RCTs had low risk of bias. The results show that moderate-quality evidence suggested OMT had a significant and clinically relevant effect on pain relief (MD: -13.04, 95% CI: -20.64 to -5.44) in chronic nonspecific neck pain, and moderate-quality evidence suggested a non-significant difference in favour of OMT for functional status (SMD: -0.38, 95% CI: -0.88 to -0.11). No serious adverse events were reported.
The authors concluded that, based on the three included studies, the review suggested clinically relevant effects of OMT for reducing pain in patients with chronic nonspecific neck pain. Given the small sample sizes, different comparison groups, and lack of long-term measurements in the few available studies, larger, high-quality randomized controlled trials with robust comparison groups are recommended.
Yet again I am taken aback by several things simultaneously:
- the extreme paucity of RCTs, particularly considering that neck pain is one of the main indication for osteopaths,
- the rather uncritical text by the authors,
- the nonsensical conclusions.
Let me offer my own conclusions which are, I hope, a little more realistic:
GIVEN THE PAUCITY OF THE RCTs AND THEIR SMALL SAMPLE SIZES, IT IS NOT POSSIBLE TO CLAIM THAT OMT FOR NONSPECIFIC NECK PAIN IS AN EVIDENCE-BASED APPROACH
Distant healing is one of the most bizarre yet popular forms of alternative medicine. Healers claim they can transmit ‘healing energy’ towards patients to enable them to heal themselves. There have been many trials testing the effectiveness of the method, and the general consensus amongst critical thinkers is that all variations of ‘energy healing’ rely entirely on a placebo response. A recent and widely publicised paper seems to challenge this view.
This article has, according to its authors, two aims. Firstly it reviews healing studies that involved biological systems other than ‘whole’ humans (e.g., studies of plants or cell cultures) that were less susceptible to placebo-like effects. Secondly, it presents a systematic review of clinical trials on human patients receiving distant healing.
All the included studies examined the effects upon a biological system of the explicit intention to improve the wellbeing of that target; 49 non-whole human studies and 57 whole human studies were included.
The combined weighted effect size for non-whole human studies yielded a highly significant (r = 0.258) result in favour of distant healing. However, outcomes were heterogeneous and correlated with blind ratings of study quality; 22 studies that met minimum quality thresholds gave a reduced but still significant weighted r of 0.115.
Whole human studies yielded a small but significant effect size of r = .203. Outcomes were again heterogeneous, and correlated with methodological quality ratings; 27 studies that met threshold quality levels gave an r = .224.
From these findings, the authors drew the following conclusions: Results suggest that subjects in the active condition exhibit a significant improvement in wellbeing relative to control subjects under circumstances that do not seem to be susceptible to placebo and expectancy effects. Findings with the whole human database suggests that the effect is not dependent upon the previous inclusion of suspect studies and is robust enough to accommodate some high profile failures to replicate. Both databases show problems with heterogeneity and with study quality and recommendations are made for necessary standards for future replication attempts.
In a press release, the authors warned: the data need to be treated with some caution in view of the poor quality of many studies and the negative publishing bias; however, our results do show a significant effect of healing intention on both human and non-human living systems (where expectation and placebo effects cannot be the cause), indicating that healing intention can be of value.
My thoughts on this article are not very complimentary, I am afraid. The problems are, it seems to me, too numerous to discuss in detail:
- The article is written such that it is exceedingly difficult to make sense of it.
- It was published in a journal which is not exactly known for its cutting edge science; this may seem a petty point but I think it is nevertheless important: if distant healing works, we are confronted with a revolution in the understanding of nature – and surely such a finding should not be buried in a journal that hardly anyone reads.
- The authors seem embarrassingly inexperienced in conducting and publishing systematic reviews.
- There is very little (self-) critical input in the write-up.
- A critical attitude is necessary, as the primary studies tend to be by evangelic believers in and amateur enthusiasts of healing.
- The article has no data table where the reader might learn the details about the primary studies included in the review.
- It also has no table to inform us in sufficient detail about the quality assessment of the included trials.
- It seems to me that some published studies of distant healing are missing.
- The authors ignored all studies that were not published in English.
- The method section lacks detail, and it would therefore be impossible to conduct an independent replication.
- Even if one ignored all the above problems, the effect sizes are small and would not be clinically important.
- The research was sponsored by the ‘Confederation of Healing Organisations’ and some of the comments look as though the sponsor had a strong influence on the phraseology of the article.
Given these reservations, my conclusion from an analysis of the primary studies of distant healing would be dramatically different from the one published by the authors: DESPITE A SIZABLE AMOUNT OF PRIMARY STUDIES ON THE SUBJECT, THE EFFECTIVENESS OF DISTANT HEALING REMAINS UNPROVEN. AS THIS THERAPY IS BAR OF ANY BIOLOGICAL PLAUSIBILITY, FURTHER RESEARCH IN THIS AREA SEEMS NOT WARRANTED.
The discussion whether acupuncture is more than a placebo is as long as it is heated. Crucially, it is also quite tedious, tiresome and unproductive, not least because no resolution seems to be in sight. Whenever researchers develop an apparently credible placebo and the results of clinical trials are not what acupuncturists had hoped for, the therapists claim that the placebo is, after all, not inert and the negative findings must be due to the fact that both placebo and real acupuncture are effective.
Laser acupuncture (acupoint stimulation not with needle-insertion but with laser light) offers a possible way out of this dilemma. It is relatively easy to make a placebo laser that looks convincing to all parties concerned but is a pure and inert placebo. Many trials have been conducted following this concept, and it is therefore highly relevant to ask what the totality of this evidence suggests.
A recent systematic review did just that; specifically, it aimed to evaluate the effects of laser acupuncture on pain and functional outcomes when it is used to treat musculoskeletal disorders.
Extensive literature searches were used to identify all RCTs employing laser acupuncture. A meta-analysis was performed by calculating the standardized mean differences and 95% confidence intervals, to evaluate the effect of laser acupuncture on pain and functional outcomes. Included studies were assessed in terms of their methodological quality and appropriateness of laser parameters.
Forty-nine RCTs met the inclusion criteria. Two-thirds (31/49) of these studies reported positive effects. All of them were rated as being of high methodological quality and all of them included sufficient details about the lasers used. Negative or inconclusive studies mostly failed to demonstrate these features. For all diagnostic subgroups, positive effects for both pain and functional outcomes were more consistently seen at long-term follow-up rather than immediately after treatment.
The authors concluded that moderate-quality evidence supports the effectiveness of laser acupuncture in managing musculoskeletal pain when applied in an appropriate treatment dosage; however, the positive effects are seen only at long-term follow-up and not immediately after the cessation of treatment.
Surprised? Well, I am!
This is a meta-analysis I always wanted to conduct and never came round to doing. Using the ‘trick’ of laser acupuncture, it is possible to fully blind patients, clinicians and data evaluators. This eliminates the most obvious sources of bias in such studies. Those who are convinced that acupuncture is a pure placebo would therefore expect a negative overall result.
But the result is quite clearly positive! How can this be? I can see three options:
- The meta-analysis could be biased and the result might therefore be false-positive. I looked hard but could not find any significant flaws.
- The primary studies might be wrong, fraudulent etc. I did not see any obvious signs for this to be so.
- Acupuncture might be more than a placebo after all. This notion might be unacceptable to sceptics.
I invite anyone who sufficiently understands clinical trial methodology to scrutinise the data closely and tell us which of the three possibilities is the correct one.
Chinese proprietary herbal medicines (CPHMs) are a well-established and a hugely profitable part of Traditional Chinese Medicine (TCM) with a long history in China and elsewhere; they are used for all sorts of conditions, not least for the treatment of common cold. Many CPHMs have been listed in the ‘China national essential drug list’ (CNEDL), the official reference published by the Chinese Ministry of Health. One would hope that such a document to be based on reliable evidence – but is it?
The aim of a recent review was to provide an assessment on the potential benefits and harms of CPHMs for common cold listed in the CNEDL.
The authors of this assessment were experts from the Chinese ‘Centre for Evidence-Based Medicine’ and one well-known researcher of alternative medicine from the UK. They searched CENTRAL, MEDLINE, EMBASE, SinoMed, CNKI, VIP, China Important Conference Papers Database, China Dissertation Database, and online clinical trial registry websites from their inception to 31 March 2013 for clinical studies of CPHMs listed in the CNEDL for common cold.
Of the 33 CPHMs listed in the 2012 CNEDL for the treatment of common cold, only 7 had any type of clinical trial evidence at all. A total of 6 randomised controlled trials (RCTs) and 7 case series (CSs) could be included in the assessments.
All these studies had been conducted in China and published in Chinese. All of them were burdened with poor study design and low methodological quality, and all had to be graded as being associated with a very high risk of bias.
The authors concluded that the use of CPHMs for common cold is not supported by robust evidence. Further rigorous well designed placebo-controlled, randomized trials are needed to substantiate the clinical claims made for CPHMs.
I should state that it is, in my view, most laudable that the authors draw such a relatively clear, negative conclusion. This does certainly not happen often with papers originating from China, and George Lewith, the UK collaborator in this article, is also not known for his critical attitude towards alternative medicine. But there are other, less encouraging issues here to mention.
In the discussion section of their paper, the authors mention that the CNEDL has been approved by the Chinese Ministry of Public Health and is currently regarded as the accepted reference point for the medicines used in China. They also explain that the CNEDL was officially launched and implemented in August 2009. The CNEDL is now up-dated every 3 years, and its 2012 edition contains 520 medicines, including 203 CPHMs. The CPHMs listed in CNEDL cover 137 herbal remedies for internal medicine, 11 for surgery, 20 for gynaecology, 7 for ophthalmology, 13 for otorhinolaryngology and 15 for orthopaedics and traumatology.
Moreover, the authors inform us that about 3,100 medical and clinical experts had been recruited to evaluate the safety, effectiveness and costs of CPHMs. The selection process of medicines into CNEDL was strictly in accordance with the principle that they ‘must be preventive and curative, safe and effective, affordable, easy to use, think highly of both Chinese and Western medicine’. A detailed procedure for evaluation is, however, not available because the files are confidential.
The authors finally state that their paper demonstrates that the selection of CPHMs into the CNEDL is less likely to be ‘evidence-based’ and revealed the sharp contrast between the policy and priority given to by the Chinese government to Traditional Chinese Medicine(TCM).
This surely must be a benign judgement, if there ever was one! I would say that the facts disclosed in this review show that TCM seems to exist in a strange universe where commercial interests are officially allowed to reign supreme over patients’ interests and public health.
If we go on the internet, we find no end of positive claims for TM. The official TM website, for instance, claims that more than 350 peer-reviewed research studies on the TM technique have been published in over 160 scientific journals. These studies were conducted at many US and international universities and research centers, including Harvard Medical School, Stanford Medical School, Yale Medical School, and UCLA Medical School.
This may well be true – but do those studies amount to more than a heap of beans? Let’s find out.
The objective of our Cochrane review was to determine the effectiveness of TM for the primary prevention of cardiovascular disease (CVD). We searched the following electronic databases: the Cochrane Central Register of Controlled Trials (CENTRAL) (2013, Issue 10); MEDLINE (Ovid) (1946 to week three November 2013); EMBASE Classic and EMBASE (Ovid) (1947 to week 48 2013); ISI Web of Science (1970 to 28 November 2013); and Database of Abstracts of Reviews of Effects (DARE) and Health Technology Assessment Database and Health Economics Evaluations Database (November 2013). We also searched the Allied and complementary Medicine Database (AMED) (inception to January 2014) and IndMed (inception to January 2014). We hand searched trial registers and reference lists of reviews and articles and contacted experts in the field. We applied no language restrictions.
We included randomised controlled trials (RCTs) of at least three months’ duration involving healthy adults or adults at high risk of CVD. Trials examined TM only and the comparison group was no intervention or minimal intervention. We excluded trials that involved multi-factorial interventions. Outcomes of interest were clinical CVD events (cardiovascular mortality, all-cause mortality and non-fatal events) and major CVD risk factors (e.g. blood pressure and blood lipids, occurrence of type 2 diabetes, quality of life, adverse events and costs). Two authors independently selected trials for inclusion, extracted data and assessed the risk of bias.
We identified 4 RCTs with a total of 430 participants for inclusion in this review. The included trials were small, short term (three months) and at risk of bias. In all studies, TM was practised for 15 to 20 minutes twice a day. None of the included studies reported all-cause mortality, cardiovascular mortality or non-fatal endpoints as trials were short term, but one study reported survival rate three years after the trial was completed. In view of the considerable statistical heterogeneity between the results of the studies for the only outcomes reported, systolic blood pressure (I2 = 72%) and diastolic blood pressure (I2 = 66%), we decided not to undertake a meta-analysis. None of the four trials reported blood lipids, occurrence of type 2 diabetes, adverse events, costs or quality of life.
We concluded that there are few trials with limited outcomes examining the effectiveness of TM for the primary prevention of CVD. Due to the limited evidence to date, we could draw no conclusions as to the effectiveness of TM for the primary prevention of CVD. There was considerable heterogeneity between trials and the included studies were small, short term and at overall serious risk of bias. More and larger long-term, high-quality trials are needed.
Even though I am a co-author of this review, I am not entirely sure that the last sentence of our conclusion is totally correct. The TM movement has, in my view, all the characteristics of a cult with all its the dangers that cults entail. This means, I think, we ought to be cautious about TM and sceptical about their research and results. At the risk of provoking harsh criticism, I would even say we should be distrustful of their aims and methods.
As promised, I will try with this post to explain my reservations regarding the new meta-analysis suggesting that individualised homeopathic remedies are superior to placebos. Before I start, however, I want to thank all those who have commented on various issues; it is well worth reading the numerous and diverse comments.
To remind us of the actual meta-analysis, it might be useful to re-publish its abstract (the full article is also available online):
A rigorous and focused systematic review and meta-analysis of randomised controlled trials (RCTs) of individualised homeopathic treatment has not previously been undertaken. We tested the hypothesis that the outcome of an individualised homeopathic treatment approach using homeopathic medicines is distinguishable from that of placebos.
The review’s methods, including literature search strategy, data extraction, assessment of risk of bias and statistical analysis, were strictly protocol-based. Judgment in seven assessment domains enabled a trial’s risk of bias to be designated as low, unclear or high. A trial was judged to comprise ‘reliable evidence’ if its risk of bias was low or was unclear in one specified domain. ‘Effect size’ was reported as odds ratio (OR), with arithmetic transformation for continuous data carried out as required; OR > 1 signified an effect favouring homeopathy.
Thirty-two eligible RCTs studied 24 different medical conditions in total. Twelve trials were classed ‘uncertain risk of bias’, three of which displayed relatively minor uncertainty and were designated reliable evidence; 20 trials were classed ‘high risk of bias’. Twenty-two trials had extractable data and were subjected to meta-analysis; OR = 1.53 (95% confidence interval (CI) 1.22 to 1.91). For the three trials with reliable evidence, sensitivity analysis revealed OR = 1.98 (95% CI 1.16 to 3.38).
Medicines prescribed in individualised homeopathy may have small, specific treatment effects. Findings are consistent with sub-group data available in a previous ‘global’ systematic review. The low or unclear overall quality of the evidence prompts caution in interpreting the findings. New high-quality RCT research is necessary to enable more decisive interpretation.
Since my team had published an RCTs of individualised homeopathy, it seems only natural that my interest focussed on why the study (even though identified by Mathie et al) had not been included in the meta-analysis. Our study had provided no evidence that adjunctive homeopathic remedies, as prescribed by experienced homeopathic practitioners, are superior to placebo in improving the quality of life of children with mild to moderate asthma in addition to conventional treatment in primary care.
I was convinced that this trial had been rigorous and thus puzzled why, despite receiving ‘full marks’ from the reviewers, they had not included it in their meta-analysis. I thus wrote to Mathie, the lead author of the meta-analysis, and he explained: For your trial (White et al. 2003), under domain V of assessment, we were unable to extract data for meta-analysis, and so it was attributed high risk of bias, as specified by the Cochrane judgmental criteria. Our designated main outcome was the CAQ, for which we needed to know (or could at least estimate) a mean and SD for both the baseline and the end-point of the study. Since your paper reported only the change from baseline in Table 3 or in the main text, it is not possible to derive the necessary end-point for analysis.
It took a while and several further emails until I understood: our study did report both the primary (Table 2 quality of life) and secondary outcome measure (Table 3 severity of symptoms). The primary outcome measure was reported in full detail such that a meta-analysis would have been possible. The secondary outcome measure was also reported but not in full detail, and the data provided by us would not lend themselves to meta-analyses. By electing not our primary but our secondary outcome measure for their meta-analysis, Mathie et al were able to claim that they were unable to use our study and reject it for their meta-analysis.
Why did they do that?
The answer is simple: in their methods section, they specify that they used outcome measures “based on a pre-specified hierarchical list in order of greatest to least importance, recommended by the WHO“. This, I would argue is deeply flawed: the most important outcome measure of a study is usually the one for which the study was designed, not the one that some guys at the WHO feel might be important (incidentally, the WHO list was never meant to be applied to meta-analyses in that way).
By following rigidly their published protocol, the authors of the meta-analysis managed to exclude our negative trial. Thus they did everything right – or did they?
Well, I think they committed several serious mistakes.
- Firstly, they wrote the protocol, which forced them to exclude our study. Following a protocol is not a virtue in itself; if the protocol is nonsensical it even is the opposite. Had they proceeded as is normal in such cases and used our primary outcome measure in their meta-analyses, it is most likely that their overall results would not have been in favour of homeopathy.
- Secondly, they awarded our study a malus point for the criterium ‘selective outcome reporting’. This is clearly a wrong decision: we did report the severity-outcome, albeit not in sufficient detail for their meta-analysis. Had they not committed this misjudgment, our RCT would have been the only one with an ‘A’ rating. This would have very clearly highlighted the nonsense of excluding the best-rated trial from meta-analysis.
There are several other oddities as well. For instance, Mathie et al judge our study to be NOT free of vested interest. I asked Mathie why they had done this and was told it is because we accepted free trial medication from a homeopathic pharmacy. I would argue that my team was far less plagued by vested interest than the authors of their three best (and of course positive) trials who, as I happen to know, are consultants for homeopathic manufacturers.
And all of this is just in relation to our own study. Norbert Aust has uncovered similar irregularities with other trials and I take the liberty of quoting his comments posted previously again here:
I have reason to believe that this review and metaanalysis in biased in favor of homeopathy. To check this, I compared two studies (1) Jacobs 1994 about the treatment of childhood diarrhea in Nicaragua, (2) Walach 1997 about homeopathic threatment of headaches. The Jacobs study is one of the three that provided ‘reliable evidence’, Walach’s study earned a poor C2.2 rating and was not included in the meta-analyses. Jacobs’ results were in favour of homeopathy, Walach’s not.
For the domains where the rating of Walach’s study was less than that of the Jacobs study, please find citations from the original studies or my short summaries for the point in question.
Domain I: Sequence generation:
“The remedy selected was then mailed to a notary public who held a stock of placebos. The notary threw a dice and mailed either the homeopathic remedy or an appropriate placebo. The notary was provided with a blank randomisation list.”
Rating: UNCLEAR (Medium risk of bias)
“For each of these medications, there was a box of tubes in sequentially numbered order which had been previously randomized into treatment or control medication using a random numbers table in blocks of four”
Rating: YES (Low risk of bias)
Domain IIIb: Blinding of outcome assessor
“The notary was provided with a blank randomization list which was an absolutely unique document. It was only handed out after the biometrician (WG) had deposited all coded original data as a printout at the notary’s office. (…) Data entry was performed blindly by personnel not involved in the study. ”
Rating: UNCLEAR (Medium risk of bias)
“All statistical analyses were done before breaking the randomisation code, using the program …”
Rating: YES (Low risk of bias)
Domain V: Selective outcome reporting
Study protocol was published in 1991 prior to enrollment of participants, all primary outcome variables were reported with respect to all participants and the endpoints.
Rating: NO (high risk of bias)
No prior publication of protocol, but a pilot study exists. However this was published in 1993 only after the trial was performed in 1991. Primary outcome defined (duration of diarrhea), reported but table and graph do not match, secondary outcome (number of unformed stools on day 3) seems defined post hoc, for this is the only one point in time, this outcome yielded a significant result.
Rating: YES (low risk of bias)
Domain VI: Other sources of bias:
Rating: NO (high risk of bias), no details given
Imbalance of group properties (size, weight and age of children), that might have some impact on course of disease, high impact of parallel therapy (rehydration) by far exceeding effect size of homeopathic treatment
Rating: YES (low risk of bias), no details given
In a nutshell: I fail to see the basis for the different ratings in the studies themselves. I assume bias of the authors of the review.
So, what about the question posed in the title of this article? The meta-analysis is clearly not a ‘proof of concept’. But is it proof for misconduct? I asked Mathie and he answered as follows: No, your statement does not reflect the situation at all. As for each and every paper, we selected the main outcome measure for your trial using the objective WHO classification approach (in which quality of life is clearly of lower rank than severity). This is all clearly described in our prospective protocol. Under no circumstances did we approach this matter retrospectively, in the way you are implying.
Some nasty sceptics might have assumed that the handful of rigorous studies with negative results were well-known to most researchers of homeopathy. In this situation, it would have been hugely tempting to write the protocol such that these studies must be excluded. I am thrilled to be told that the authors of the current new meta-analysis (who declared all sorts of vested interests at the end of the article) resisted this temptation.