A comprehensive review of the evidence relating to acupuncture entitled “The Acupuncture Evidence Project: A Comparative Literature Review” has just been published. The document aims to provide “an updated review of the literature with greater rigour than was possible in the past.” That sounds great! Let’s see just how rigorous the assessment is.
The review was conducted by John McDonald who no stranger to this blog; we have mentioned him here, for instance. To call him an unbiased, experienced, or expert researcher would, in my view, be more than a little optimistic.
The review was financed by the ‘Australian Acupuncture and Chinese Medicine Association Ltd.’ – call me a pessimist, but I do wonder whether this bodes well for the objectivity of the findings.
The research seems to have been assisted by a range of experts: Professor Caroline Smith, National Institute of Complementary Medicine, Western Sydney University, provided advice regarding evidence levels for assisted reproduction trials; Associate Professor Zhen Zheng, RMIT University identified the evidence levels for postoperative nausea and vomiting and post-operative pain; Dr Suzanne Cochrane, Western Sydney University; Associate Professor Chris Zaslawski, University of Technology Sydney; and Associate Professor Zhen Zheng, RMIT University provided prepublication commentary and advice. I fail to see anyone in this list who is an expert in EBM or who is even mildly critical of acupuncture and the many claims that are being made for it.
The review has not been published in a journal. This means, it has not been peer-reviewed. As we will see shortly, there is reason to doubt that it could pass the peer-review process of any serious journal.
There is an intriguing declaration of conflicts of interest: “Dr John McDonald was a co-author of three of the research papers referenced in this review. Professor Caroline Smith was a co-author of six of the research papers referenced in this review, and Associate Professor Zhen Zheng was co-author of one of the research papers in this review. There were no other conflicts of interest.” Did they all forget to mention that they earn their livelihoods through acupuncture? Or is that not a conflict?
I do love the disclaimer: “The authors and the Australian Acupuncture and Chinese Medicine Association Ltd (AACMA) give no warranty that the information contained in this publication and within any online updates available on the AACMA website are correct or complete.” I think they have a point here.
But let’s not be petty, let’s look at the actual review and how well it was done!
Systematic reviews must first formulate a precise research question, then disclose the exact methodology, reveal the results and finally discuss them critically. I am afraid, I miss almost all of these essential elements in the document in question.
The methods section includes statements which puzzle me (my comments are in bold):
- A total of 136 systematic reviews, including 27 Cochrane systematic reviews were included in this review, along with three network meta-analyses, nine reviews of reviews and 20 other reviews. Does that indicate that non-systematic reviews were included too? Yes, it does – but only, if they reported a positive result, I presume.
- Some of the included systematic reviews included studies which were not randomised controlled trials. In this case, they should have not been included at all, in my view.
- … evidence from individual randomised controlled trials has been included occasionally where new high quality randomised trials may have changed the conclusions from the most recent systematic review. ‘Occasionally’ is the antithesis of systematic. This discloses the present review as being non-systematic and therefore worthless.
- Some systematic reviews have not reported an assessment of quality of evidence of included trials, and due to time constraints, this review has not attempted to make such an assessment. Say no more!
It is almost needless to mention that the findings (presented in a host of hardly understandable tables) suggest that acupuncture is of proven or possible effectiveness/efficacy for a very wide array of conditions. It also goes without saying that there is no critical discussion, for instance, of the fact that most of the included evidence originated from China, and that it has been shown over and over again that Chinese acupuncture research never seems to produce negative results.
So, what might we conclude from all this?
I don’t know about you, but for me this new review is nothing but an orgy in deceit and wishful thinking!
Gastro-oesophageal reflux disease (GORD) is a common, benign condition. It can be treated by changing eating habits or drugs. Many alternative therapies are also on offer, for instance, acupuncture. But does it work? Let’s find out.
The objective of this meta-analysis was to explore the effectiveness of acupuncture for the treatment of gastro-oesophageal reflux disease (GORD). Four English and four Chinese databases were searched through June 2016. Randomised controlled trials investigating the effectiveness of manual acupuncture or electroacupuncture (MA/EA) for GORD versus or as an adjunct to Western medicine (WM) were selected.
A total of 12 trials involving 1235 patients were included. The results demonstrated that patients receiving MA/EA combined with WM had a superior global symptom improvement compared with those receiving WM alone with no significant heterogeneity. Recurrence rates of those receiving MA/EA alone were lower than those receiving WM with low heterogeneity, while global symptom improvement (six studies) and symptom scores (three studies) were similar. Descriptive analyses suggested that acupuncture also improves quality of life in patients with GORD.
The authors concluded that this meta-analysis suggests that acupuncture is an effective and safe treatment for GORD. However, due to the small sample size and poor methodological quality of the included trials, further studies are required to validate our conclusions.
I am glad the authors used the verb ‘suggest’ in their conclusions. In fact, even this cautious terminology is too strong, in my view. Here are 9 reasons why:
- The hypothesis that acupuncture is effective for GORD lacks plausibility.
- All the studies were of poor or very poor methodological quality.
- All but one were from China, and we know that all acupuncture trials from this country are positive, thus casting serious doubt on their validity.
- Six trials had the infamous ‘A+B versus B’ design which never generates a negative result.
- There was evidence of publication bias, i. e. negative trials had disappeared and were thus not included in the meta-analysis.
- None of the trials made an attempt to control for placebo effects by using a sham-control procedure.
- None used patient-blinding.
- The safety of a therapy cannot be assessed on the basis of 12 trials
- Seven studies failed to report adverse effects, thus violating research ethics.
Considering these facts, I think that a different conclusion would have been more appropriate: this meta-analysis provides no good evidence for the assumption that acupuncture is an effective and safe treatment for GORD.
It used to be called ‘good bedside manners’. The term is an umbrella for a range of attitudes and behaviours including compassion, empathy and conveying positive messages. What could be more obvious than the assumption that good bedside manners are better than bad ones?
But as sceptics, we need to doubt obvious assumptions! Where is the evidence? we need to ask. So, where is the evidence that positive messages have any clinical effects? A meta-analysis has tackled the issue, and the results are noteworthy.
The researchers aimed to estimate the efficacy of positive messages for pain reduction. They included RCTs of the effects of positive messages. Their primary outcome measures were differences in patient- or observer reported pain between groups who were given positive messages and those who were not. Of the 16 RCTs (1703 patients) that met the inclusion criteria, 12 trials had sufficient data for meta-analysis. The pooled standardized effect size was −0.31 (95% CI −0.61 to −0.01, P = 0.04, I² = 82%). The effect size remained positive but not statistically significant after we excluded studies considered to have a high risk of bias (standard effect size −0.17, 95% CI −0.54 to 0.19, P = 0.36, I² = 84%). The authors concluded that care of patients with chronic or acute pain may be enhanced when clinicians deliver positive messages about possible clinical outcomes. However, we have identified several limitations of the present study that suggest caution when interpreting the results. We recommend further high quality studies to confirm (or falsify) our result.
The 1st author of this paper published a comment in which he stated that our recent mega-study with 12 randomized trials confirmed that doctors who use positive language reduce patient pain by a similar amount to drugs. Other trials show that positive messages can:
• help Parkinson’s patients move their hands faster,
• increase ‘peak flow’ (a measure of how much air is breathed) in asthma patients,
• improve the diameter of arteries in heart surgery patients, and
• reduce the amount of pain medication patients use.
The way a positive message seems to help is biological. When a patient anticipates a good thing happening (for example that their pain will go away), this activates parts of the brain that help the body make its own drugs like endorphins. A positive doctor may also help a patient relax which can also improve health.
I am not sure that this is entirely correct. When the authors excluded the methodologically weak and therefore unreliable studies, the effect was no longer significant. That is to say, it was likely due to chance.
And what about the other papers cited above? I am not sure about them either. Firstly, they do not necessarily show that positive messages are effective. Secondly, there is just one study for each claim, and one swallow does not make a summer; we would need independent replications.
So, am I saying that being positive as a clinician is ineffective? No! I am saying that the evidence is too flimsy to be sure. And possibly, this means that the effect of positive messages is smaller than we all thought.
How often have I pointed out that most studies of chiropractic (and other alternative therapies) are overtly unethical because they fail to report adverse events? And if you think this is merely my opinion, you are mistaken. This new analysis by a team of chiropractors aimed to describe the extent of adverse events reporting in published RCTs of Spinal Manipulative Therapy (SMT), and to determine whether the quality of reporting has improved since publication of the 2010 Consolidated Standards Of Reporting Trials (CONSORT) statement.
The Physiotherapy Evidence Database and the Cochrane Central Register of Controlled Trials were searched for RCTs involving SMT. Domains of interest included classifications of adverse events, completeness of adverse events reporting, nomenclature used to describe the events, methodological quality of the study, and details of the publishing journal. Data were analysed using descriptive statistics. Frequencies and proportions of trials reporting on each of the specified domains above were calculated. Differences in proportions between pre- and post-CONSORT trials were calculated with 95% confidence intervals using standard methods, and statistical comparisons were analysed using tests for equality of proportions with continuity correction.
Of 7,398 records identified in the electronic searches, 368 articles were eligible for inclusion in this review. Adverse events were reported in 140 (38.0%) articles. There was a significant increase in the reporting of adverse events post-CONSORT (p=.001). There were two major adverse events reported (0.3%). Only 22 articles (15.7%) reported on adverse events in the abstract. There were no differences in reporting of adverse events post-CONSORT for any of the chosen parameters.
The authors concluded that although there has been an increase in reporting adverse events since the introduction of the 2010 CONSORT guidelines, the current level should be seen as inadequate and unacceptable. We recommend that authors adhere to the CONSORT statement when reporting adverse events associated with RCTs that involve SMT.
We conducted a very similar analysis back in 2012. Specifically, we evaluated all 60 RCTs of chiropractic SMT published between 2000 and 2011 and found that 29 of them did not mention adverse effects at all. Sixteen RCTs reported that no adverse effects had occurred (which I find hard to believe since reliable data show that about 50% of patients experience adverse effects after consulting a chiropractor). Complete information on incidence, severity, duration, frequency and method of reporting of adverse effects was included in only one RCT. Conflicts of interests were not mentioned by the majority of authors. Our conclusion was that adverse effects are poorly reported in recent RCTs of chiropractic manipulations.
The new paper suggests that the situation has improved a little, yet it is still wholly unacceptable. To conduct a clinical trial and fail to mention adverse effects is not, as the authors of the new article suggest, against current guidelines; it is a clear and flagrant violation of medical ethics. I blame the authors of such papers, the reviewers and the journal editors for behaving dishonourably and urge them to get their act together.
The effects of such non-reporting are obvious: anyone looking at the evidence (for instance via systematic reviews) will get a false-positive impression of the safety of SMT. Consequently, chiropractors are able to claim that very few adverse effects have been reported in the literature, therefore our hallmark therapy SMT is demonstrably safe. Those who claim otherwise are quite simply alarmist.
As the data suggesting that homeopathy is effective for improving health is – to put it mildly – less than convincing, a frantic search is currently on amongst homeopaths and their followers to identify a specific condition for which the evidence is stronger than for all conditions pooled into one big analysis. If they could show that it works for just one disease, they could celebrate this finding and henceforth use it for refuting doubters stating that highly diluted homeopathic remedies are pure placebos. One such condition is allergic rhinitis; there have been several trials suggesting that homeopathy might be effective for it, and therefore it is only logical that homeopathy-promoters want to summarise these data in order to silence sceptics once and for all.
A new paper ought to be seen in this vein. It is systematic review by the Mathie group with the stated aim “to evaluate the efficacy and effectiveness of homeopathic intervention in the treatment of seasonal or perennial allergic rhinitis (AR).”
Randomized controlled trials evaluating all forms of homeopathic treatment for AR were included in a systematic review (SR) of studies published up to and including December 2015. Two authors independently screened potential studies, extracted data, and assessed risk of bias. Primary outcomes included symptom improvement and total quality-of-life score. Treatment effect size was quantified as mean difference (continuous data), or by risk ratio (RR) and odds ratio (dichotomous data), with 95% confidence intervals (CI). Meta-analysis was performed after assessing heterogeneity and risk of bias.
Eleven studies were eligible for SR. All trials were placebo-controlled except one. Six trials used the treatment approach known as isopathy, but they were unsuitable for meta-analysis due to problems of heterogeneity and data extraction. The overall standard of methods and reporting was poor: 8/11 trials were assessed as “high risk of bias”; only one trial, on isopathy for seasonal AR, possessed reliable evidence. Three trials of variable quality (all using Galphimia glauca for seasonal AR) were included in the meta-analysis: nasal symptom relief at 2 and 4 weeks (RR = 1.48 [95% CI 1.24-1.77] and 1.27 [95% CI 1.10-1.46], respectively) favoured homeopathy compared with placebo; ocular symptom relief at 2 and 4 weeks also favoured homeopathy (RR = 1.55 [95% CI 1.33-1.80] and 1.37 [95% CI 1.21-1.56], respectively). The single trial with reliable evidence had a small positive treatment effect without statistical significance. A homeopathic and a conventional nasal spray produced equivalent improvements in nasal and ocular symptoms.
The authors concluded that the low or uncertain overall quality of the evidence warrants caution in drawing firm conclusions about intervention effects. Use of either Galphimia glauca or a homeopathic nasal spray may have small beneficial effects on the nasal and ocular symptoms of AR. The efficacy of isopathic treatment of AR is unclear.
Extracts of Galphimia glauca (GG) have been used traditionally in South America for the treatment of allergic conditions, with some reports suggesting effectiveness. A 1997 meta-analysis of 11 clinical trials (most of them of very poor quality) of homeopathic GG suggested this therapy to be effective in the treatment of AR. In 2011, I published a review (FACT 2011, 16 200-203) focussed exclusively on the remarkable set of RCTs of homeopathic Galphimia glauca (GG). My conclusions were as follows: three of the four currently available placebo-controlled RCTs of homeopathic GG suggest this therapy is an effective symptomatic treatment for hay fever. There are, however, important caveats. Most essentially, independent replication would be required before GG can be considered for the routine treatment of hay fever. Since then, no new studies have emerged.
I am citing this for two main reasons:
- There is nothing homeopathic about the principle of using GG for allergic conditions; according to homeopathic theory GG extracts would need to cause allergies for GG to have potential as a homeopathic allergy remedy. Arguably, the GG trials should therefore have been excluded from this meta-analysis for not following the homeopathic principal of ‘like cures like’.
- All the RCTs of GG were done by the same German research group. There is not a single independent replication of their findings!
Seen from this perspective, the conclusion by Mathie et al, that the use of either Galphimia glauca … may have small beneficial effects on the nasal and ocular symptoms of AR, seems more than a little over-optimistic.
On this blog, we have had (mostly unproductive) discussions with homeopath so often that sometimes they sound like a broken disk. I don’t want to add to this kerfuffle; what I hope to do today is to summarise a certain line of argument which, from the homeopaths’ point of view, seems entirely logical. I do this in the form of a fictitious conversation between a scientist (S) and a classical homeopath (H). My aim is to make the reader understand homeopaths better so that, future debates might be better informed.
HERE WE GO:
S: I have studied the evidence from studies of homeopathy in some detail, and I have to tell you, it fails to show that homeopathy works.
H: This is not true! We have plenty of evidence to prove that patients get better after seeing a homeopath.
S: Yes, but this is not because of the remedy; it is due to non-specific effect like the empathetic consultation with a homeopath. If one controls for these factors in adequately designed trials, the result usually is negative.
I will re-phrase my claim: the evidence fails to show that highly diluted homeopathic remedies are more effective than placebos.
H: I disagree, there are positive studies as well.
S: Let’s not cherry pick. We must always consider the totality of the reliable evidence. We now have a meta-analysis published by homeopaths that demonstrates the ineffectiveness of homeopathy quite clearly.
H: This is because homeopathy was not used correctly in the primary trials. Homeopathy must be individualised for each unique patient; no two cases are alike! Remember: homeopathy is based on the principle that like cures like!!!
S: Are you saying that all other forms of using homeopathy are wrong?
H: They are certainly not adhering to what Hahnemann told us to do; therefore you cannot take their ineffectiveness as proof that homeopathy does not work.
S: This means that much, if not most of homeopathy as it is used today is to be condemned as fake.
H: I would not go that far, but it is definitely not the real thing; it does not obey the law of similars.
S: Let’s leave this to one side for the moment. If you insist on individualised homeopathy, I must tell you that this approach can also be tested in clinical trials.
H: I know; and there is a meta-analysis which proves that it is effective.
S: Not quite; it concluded that medicines prescribed in individualised homeopathy may have small, specific treatment effects. Findings are consistent with sub-group data available in a previous ‘global’ systematic review. The low or unclear overall quality of the evidence prompts caution in interpreting the findings. New high-quality RCT research is necessary to enable more decisive interpretation.
If you call this a proof of efficacy, I would have to disagree with you. The effect was tiny and at least two of the best studies relevant to the subject were left out. If anything, this paper is yet another proof that homeopathy is useless!
H: You simply don’t understand homeopathy enough to say that. I tried to tell you that the remedy must be carefully chosen to fit each unique patient. This is a very difficult task, and sometimes it is not successful – mainly because the homeopaths employed in clinical trials are not skilled enough to find it. This means that, in these studies, we will always have a certain failure rate which, in turn, is responsible for the small average effect size.
S: But these studies are always conducted by experienced homeopaths, and only the very best, most experienced homeopaths were chosen to cooperate in them. Your argument that the trials are negative because of the ineffectiveness of the homeopaths – rather than the ineffectiveness of homeopathy – is therefore nonsense.
H: This is what you say because you don’t understand homeopathy!
S: No, it is what you say because you don’t understand science. How else would you prove that your hypothesis is correct?
H: Simple! Just look at individual cases from the primary studies within this meta-analysis . You will see that there are always patients who did improve. These cases are the proof we need. The method of the RCT is only good for defining average effects; this is not what we should be looking at, and it is certainly not what homeopaths are interested in.
S: Are you saying that the method of the RCT is wrong?
H: It is not always wrong. Some RCTs of homeopathy are positive and do very clearly prove that homeopathy works. These are obviously the studies where homeopathy has been applied correctly. We have to make a meta-analysis of such trials, and you will see that the result turns out to be positive.
S: So, you claim that all the positive studies have used the correct method, while all the negative ones have used homeopathy incorrectly.
H: If you insist to put it like that, yes.
S: I see, you define a trial to have used homeopathy correctly by its result. Essentially you accept science only if it generates the outcome you like.
H: Yes, that sounds odd to you – because you don’t understand enough of homeopathy.
S: No, what you seem to insist on is nothing short of double standards. Or would you accept a drug company claiming: some patients did feel better after taking our new drug, and this is proof that it works?
H: You see, not understanding homeopathy leads to serious errors.
S: I give up.
The question whether spinal manipulative therapy (SMT) is effective for acute low back pain is still discussed controversially. Chiropractors (they use SMT more regularly than other professionals) try everything to make us believe it does work, while the evidence is far less certain. Therefore, it is worth considering the best and most up-to-date data.
The aim of this paper was to systematically review studies of the effectiveness and harms of SMT for acute (≤6 weeks) low back pain. The research question was straight forward: Is the use of SMT in the management of acute (≤6 weeks) low back pain associated with improvements in pain or function?
A through literature search was conducted to locate all relevant papers. Study quality was assessed using the Cochrane Back and Neck (CBN) Risk of Bias tool. The evidence was assessed using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) criteria. The main outcome measures were pain (measured by either the 100-mm visual analog scale, 11-point numeric rating scale, or other numeric pain scale), function (measured by the 24-point Roland Morris Disability Questionnaire or Oswestry Disability Index [range, 0-100]), or any harms measured within 6 weeks.
Of 26 eligible RCTs identified, 15 RCTs (1711 patients) provided moderate-quality evidence that SMT has a statistically significant association with improvements in pain (pooled mean improvement in the 100-mm visual analog pain scale, −9.95 [95% CI, −15.6 to −4.3]). Twelve RCTs (1381 patients) produced moderate-quality evidence that SMT has a statistically significant association with improvements in function (pooled mean effect size, −0.39 [95% CI, −0.71 to −0.07]). Heterogeneity was not explained by type of clinician performing SMT, type of manipulation, study quality, or whether SMT was given alone or as part of a package of therapies. No RCT reported any serious adverse event. Minor transient adverse events such as increased pain, muscle stiffness, and headache were reported 50% to 67% of the time in large case series of patients treated with SMT.
The authors concluded that among patients with acute low back pain, spinal manipulative therapy was associated with modest improvements in pain and function at up to 6 weeks, with transient minor musculoskeletal harms. However, heterogeneity in study results was large.
This meta-analysis has been celebrated by chiropractors around the world as a triumph for their hallmark therapy, SMT. But there have also been more cautionary voices – not least from the lead author of the paper. Patients undergoing spinal manipulation experienced a decline of 1 point in their pain rating, says Dr. Paul Shekelle, an internist with the West Los Angeles Veterans Affairs Medical Center and the Rand Corporation who headed the study. That’s about the same amount of pain relief as from NSAIDs, over-the-counter nonsteroidal anti-inflammatory medication, such as ibuprofen. The study also found spinal manipulation modestly improved function. On average, patients reported greater ease and comfort engaging in two day-to-day activities — such as finding they could walk more quickly, were having less difficulty turning over in bed or were sleeping more soundly.
It’s not clear exactly how spinal manipulation relieves back pain. But it may reposition the small joints in the spine in a way that causes less pain, according to Dr. Richard Deyo, an internist and professor of evidence-based medicine at the Oregon Health and Science University. Deyo wrote an editorial published along with the study. Another possibility, Deyo says, is that spinal manipulation may restore some material in the disk between the vertebrae, or it may simply relax muscles, which could be important. There may also be mind-body interaction that comes from the “laying of hands” or a trusting relationship between patients and their health care provider, he says.
Deyo notes that there are many possible treatments for lower back pain, including oral medicine, injected medicine, corsets, traction, surgery, acupuncture and massage therapy. But of about 200 treatment options, “no single treatment is clearly superior,” he says.
In another comment by Paul Ingraham the critical tone was much clearer: “Claiming it as a victory is one of the best examples I’ve ever seen of making lemonade out of science lemons! But I can understand the mistake, because the review itself does seem positive at first glance: the benefits of SMT are disingenuously summarized as “statistically significant” in the abstract, with no mention of clinical significance (effect size; see Statistical Significance Abuse). So the abstract sounds like good news to anyone but the most wary readers, while deep in the main text the same results are eventually conceded to be “clinically modest.” But even even that seems excessively generous: personally, I need at least a 2-point improvement in pain on a scale of 10 to consider it a “modest” improvement! This is not a clearly positive review: it shows weak evidence of minor efficacy, based on “significant unexplained heterogeneity” in the results. That is, the results were all over the place — but without any impressive benefits reported by any study — and the mixture can’t be explained by any obvious, measurable factor. This probably means there’s just a lot of noise in the data, too many things that are at least as influential as the treatment itself. Or — more optimistically — it could mean that SMT is “just” disappointingly mediocre on average, but might have more potent benefits in a minority of cases (that no one seems to be able to reliably identify). Far from being good news, this review continues a strong trend (eg Rubinstein 2012) of damning SMT with faint praise, and also adds evidence of backfiring to mix. Although fortunately “no RCT reported any serious adverse event,” it seems that minor harms were legion: “increased pain, muscle stiffness, and headache were reported 50% to 67% of the time in large case series of patients treated with SMT.” That’s a lot of undesirable outcomes. So the average patient has a roughly fifty-fifty chance of up to roughly maybe a 20% improvement… or feeling worse to some unknown degree! That does not sound like a good deal to me. It certainly doesn’t sound like good medicine.”
END OF QUOTE
As I have made clear in many previous posts, I do fully agree with these latter statements and would add just three points:
- We know that many of the SMT studies completely neglect reporting adverse effects. Therefore it is hardly surprising that no serious complications were on record. Yet, we know that they do occur with sad regularity.
- None of the studies controlled for placebo effects. It is therefore possible – I would say even likely – that a large chunk of the observed benefit is not due to SMT per se but to a placebo response.
- It seems more than questionable whether the benefits of SMT outweigh its risks.
This new systematic review by proponents of homeopathy (and supported by a grant from the Manchester Homeopathic Clinic) tested the null hypothesis that “the main outcome of treatment using a non-individualised (standardised) homeopathic medicine is indistinguishable from that of placebo“. An additional aim was to quantify any condition-specific effects of non-individualised homeopathic treatment. In reporting this paper, I will stay very close to the published text hoping that this avoids both misunderstandings and accusations of bias on my side:
Literature search strategy, data extraction and statistical analysis followed the methods described in a pre-published protocol. A trial comprised ‘reliable evidence’ if its risk of bias was low or it was unclear in one specified domain of assessment. ‘Effect size’ was reported as standardised mean difference (SMD), with arithmetic transformation for dichotomous data carried out as required; a negative SMD indicated an effect favouring homeopathy.
The authors excluded the following types of trials: studies of crossover design; of radionically prepared homeopathic medicines; of homeopathic prophylaxis; of homeopathy combined with other (complementary or conventional) intervention; for other specified reasons. The final explicit exclusion criterion was that there was obviously no blinding of participants and practitioners to the assigned intervention.
Forty-eight different clinical conditions were represented in 75 eligible RCTs; 49 were classed as ‘high risk of bias’ and 23 as ‘uncertain risk of bias’; the remaining three trials displayed sufficiently low risk of bias to be designated reliable evidence. Fifty-four trials had extractable data: pooled SMD was -0.33 (95% confidence interval (CI) -0.44, -0.21), which was attenuated to -0.16 (95% CI -0.31, -0.02) after adjustment for publication bias. The three trials with reliable evidence yielded a non-significant pooled SMD: -0.18 (95% CI -0.46, 0.09). There was no single clinical condition for which meta-analysis produced reliable evidence.
A meta-regression was performed to test specifically for within-group differences for each sub-group. The results showed that there were no significant differences between studies that were and were not:
- included in previous meta-analyses (p = 0.447);
- pilot studies (p = 0.316);
- greater than the median sample (p = 0.298);
- potency ≥ 12C (p = 0.221);
- imputed for meta-analysis (p = 0.384);
- free from vested interest (p = 0.391);
- acute/chronic (p = 0.796);
- different types of homeopathy (p = 0.217).
After removal of ‘C’-rated trials, the pooled SMD still favoured homeopathy for all sub-groups, but was statistically non-significant for 10 of the 18 (included in previous meta-analysis; pilot study; sample size > median; potency ≥12C; data imputed; free of vested interest; not free of vested interest; combination medicine; single medicine; chronic condition). There remained no significant differences between sub-groups—with the exception of the analysis for sample size > median (p = 0.028).
Meta-analyses were possible for eight clinical conditions, each analysis comprising two to 5 trials. A statistically significant pooled SMD, favouring homeopathy, was observed for influenza (N = 2), irritable bowel syndrome (N = 2), and seasonal allergic rhinitis (N = 5). Each of the other five clinical conditions (allergic asthma, arsenic toxicity, infertility due to amenorrhoea, muscle soreness, post-operative pain) showed non-significant findings. Removal of ‘C’-rated trials negated the statistically significant effect for seasonal allergic rhinitis and left the non-significant effect for post-operative pain unchanged; no higher-rated trials were available for additional analysis of arsenic toxicity, infertility due to amenorrhoea or irritable bowel syndrome. There were no ‘C’-rated trials to remove for allergic asthma, influenza, or muscle soreness. Thus, influenza was the only clinical condition for which higher-rated trials indicated a statistically significant effect; neither of its contributing trials, however, comprised reliable evidence.
The authors concluded that the quality of the body of evidence is low. A meta-analysis of all extractable data leads to rejection of our null hypothesis, but analysis of a small sub-group of reliable evidence does not support that rejection. Reliable evidence is lacking in condition-specific meta-analyses, precluding relevant conclusions. Better designed and more rigorous RCTs are needed in order to develop an evidence base that can decisively provide reliable effect estimates of non-individualised homeopathic treatment.
I am sure that this paper will lead to lively discussions in the comments section of this blog. I will therefore restrict my comments to a bare minimum.
In my view, this new meta-analysis essentially yield a negative result and confirms most previous, similar reviews.
- It confirms Linde’s conclusion that “insufficient evidence from these studies that homeopathy is clearly efficacious for any single clinical condition”.
- It confirms Linde’s conclusion that “there was clear evidence that studies with better methodological quality tended to yield less positive results”.
- It confirms Kleinjen’s conclusion that “most trials are of low methodological quality”.
- It also confirms the results of the meta-analysis by Shang et al (much-maligned by homeopaths) than “finding is compatible with the notion that the clinical effects of homoeopathy are placebo effects.”
- Finally, it confirms the conclusion of the analysis of the Australian National Health and Medical Research Council: “Homeopathy should not be used to treat health conditions that are chronic, serious, or could become serious. People who choose homeopathy may put their health at risk if they reject or delay treatments for which there is good evidence for safety and effectiveness. People who are considering whether to use homeopathy should first get advice from a registered health practitioner. Those who use homeopathy should tell their health practitioner and should keep taking any prescribed treatments.”
Another not entirely unimportant point that often gets missed in these discussions is this: even if we believe (which I do not) the most optimistic interpretation of these (and similar data) by homeopaths, we ought to point out that there is no evidence whatsoever that homeopathy cures anything. At the very best it provides marginal symptomatic relief. Yet, the claim of homeopaths that we hear constantly is that homeopathy is a causal and curative therapy.
The first author of the new meta-analysis is an employee of the Homeopathy Research Institute. We might therefore forgive him that he he repeatedly insists on dwelling on largely irrelevant (i. e. based on unreliable primary studies) findings. It seems obvious that firm conclusions can only be based on reliable data. I therefore disregard those analyses and conclusions that include such studies.
In the discussion, the authors of the new meta-analysis confirm my interpretation this by stating that they “reject the null hypothesis (non-individualised homeopathy is indistinguishable from placebo) on the basis of pooling all studies, but fail to reject the null hypothesis on the basis of the reliable evidence only.” And, in the long version of their conclusions, we find this remarkable statement: “Our meta-analysis of the current reliable evidence base therefore fails to reject the null hypothesis that the outcome of treatment using a non-individualised homeopathic medicine is not distinguishable from that using placebo.” A most torturous way of stating the obvious: the more reliable data show no difference between homeopathy and placebo.
Tui Na is a massage technique that is based on the Taoist principles of TCM. It involves a range of manipulations usually performed by an operator’s finger, hand, elbow, knee, or foot applied to muscle or soft tissue at specific parts of the body. According to one website of TCM-proponents “Tui Na makes use of various hand techniques in combination with acupuncture and other manipulation techniques. To enhance the healing process, the practitioner may recommend the use of Chinese herbs. Many of the techniques used in this massage resemble that of a western massage like gliding, kneading, vibration, tapping, friction, pulling, rolling, pressing and shaking. In Tui Na massage, the muscles and tendons are massaged with the help of hands, and an acupressure technique is applied to directly affect the flow of Qi at different acupressure points of the body, thus facilitating the healing process. It removes the blockages and keeps the energy moving through the meridians as well as the muscles. A typical session of Tui Na massage may vary from thirty minutes to an hour. The session timings may vary depending on the patient’s needs and condition. The best part of the therapy is that it relaxes as well as energizes the person. The main benefit of Tui Na massage is that it focuses on the specific problem, whether it is an acute or a chronic pain associated with the joints, muscles or a skeletal system. This technique is very beneficial in reducing the pain of neck, shoulders, hips, back, arms, highs, legs and ankle disorders. It is a very effective therapy for arthritis, pain, sciatica and muscle spasms. Other benefits of this massage therapy include alleviation of the stress related disorders like insomnia, constipation, headaches and other disorders related to digestive, respiratory and reproductive systems. The greatest advantage of Tui Na is that it focuses on maintaining overall balance with both physical and mental health. Any one who wants to avoid the side effects of drugs or a chemical based treatment can adopt this effective massage technique to alleviate their pain. Tui Na massage therapy is now becoming a more common therapy method due to its focus on specific problems rather than providing a general treatment.”
This clearly begs the question IS IT EFFECTIVE?
This systematic review assessed the evidence of Tui Na for cervical radiculopathy. Seven databases were searched. Randomised controlled trials (RCTs) incorporating Tui Na alone or Tui Na combined with conventional treatment were included. Five studies involving 448 patients were found. The pooled analysis from the 3 trials indicated that Tui Na alone showed a significant lowering immediate effects on pain score with moderate heterogeneity compared to cervical traction. The meta-analysis from 2 trials revealed significant immediate effects of Tui Na plus cervical traction in improving pain score with no heterogeneity compared to cervical traction alone. None of the RCTs mentioned adverse effects. There was very low quality or low quality evidence to support the results.
The authors concluded that “Tui Na alone or Tui Na plus cervical traction may be helpful to cervical radiculopathy patients, but supportive evidence seems generally weak. Future clinical studies with low risk of bias and adequate follow-up design are recommended.”
In my view, this is a misleading conclusion. A correct one would have been: THE CURRENT EVIDENCE IS INSUFFICIENT TO DRAW ANY CONCLUSIONS ABOUT THE EFFECTIVENESS OF TUI NA.
Here are some of the most obvious reasons:
- there are far too few studies for a firm conclusion,
- the included RCTs lack scientific rigour,
- all trials originate from China where reliability seems to be a serious problem,
- traction is not a useful therapy for radiculopathy,
- the primary studies violate research ethics by not reporting adverse effects.
Personally, I am getting very tired of conclusions stating ‘…XY MAY BE EFFECTIVE/HELPFUL/USEFUL/WORTH A TRY…’ It is obvious that the therapy in question MAY be effective, otherwise one would surely not conduct a systematic review. If a review fails to produce good evidence, it is the authors’ ethical, moral and scientific obligation to state this clearly. If they don’t, they simply misuse science for promotion and mislead the public. Strictly speaking, this amounts to scientific misconduct.
Drug and alcohol dependencies are notoriously difficult to treat effectively. Patients and their families are often desperate and willing to try anything. This seems like an ideal ground for acupuncturists who are, in my experience, experts in putting up smokescreens hiding the true value of their treatment.
The best way to determine the value of any intervention is probably conducting a systematic review of the evidence from rigorous clinical trials. Today we are in the fortunate position to have not just one of those articles; but do they really tell us the truth?
This brand-new systematic review investigated the effects of acupuncture on alcohol-related symptoms and behaviors in patients with this disorder. The PubMed database was searched until 23 August 2016, and reference lists from review studies were also reviewed. The inclusion criteria were the following: (1) being published in a peer-reviewed English-language journal, (2) use of randomized controlled trials (RCTs), (3) assessing the effects of acupuncture on psychological variables in individuals with a primary alcohol problem, and (4) reporting statistics that could be converted to effect sizes.
Seventeen studies were identified for a full-text inspection, and seven (243 patients) of these met our inclusion criteria. The outcomes assessed at the last post-treatment point and any available follow-up data were extracted from each of the studies. Five studies treated patients by inserting a needle into several acupoints in each ear. Two studies stimulated body points with or without ear stimulation. Four studies treated control patients with a placebo needle or under a completely different type of intervention, such as relaxation or transdermal stimulation, whereas the remaining studies inserted needles into nonspecific points. The patients were treated for 2 weeks to 3 months, and the treatment duration per session was 15–45 min. The results of the meta-analysis demonstrated that an acupuncture intervention had a stronger effect on reducing alcohol-related symptoms and behaviours than did the control intervention. A beneficial but weak effect of acupuncture treatment was also found in the follow-up data.
The authors concluded that although our analysis showed a significant difference between acupuncture and the control intervention in patients with alcohol use disorder, this meta-analysis is limited by the small number of studies included. Thus, a larger cohort study is required to provide a firm conclusion.
I am used to reading poor research papers, but this one is like a new dimension. Here are just the most obvious flaws:
- by searching just one database, the likelihood of missing studies is huge,
- by excluding non-English papers, the review automatically becomes non-systematic,
- the included studies differed vastly in many respects and can therefore not be pooled.
As it happens, a further meta-analysis has just been published. Here is its abstract:
Acupuncture has been widely used as a treatment for alcohol dependence. An updated and rigorously conducted systematic review is needed to establish the extent and quality of the evidence on the effectiveness of acupuncture as an intervention for reducing alcohol dependence. This review aimed to ascertain the effectiveness of acupuncture for reducing alcohol dependence as assessed by changes in either craving or withdrawal symptoms.
In this systematic review, a search strategy was designed to identify randomised controlled trials (RCTs) published in either the English or Chinese literature, with a priori eligibility criteria. The following English language databases were searched from inception until June 2015: AMED, Cochrane Library, EMBASE, MEDLINE, PsycINFO, and PubMed; and the following Chinese language databases were similarly searched: CNKI, Sino-med, VIP, and WanFang. Methodological quality of identified RCTs was assessed using the Jadad Scale and the Cochrane Risk of Bias tool.
Fifteen RCTs were included in this review, comprising 1378 participants. The majority of the RCTs were rated as having poor methodological rigour. A statistically significant effect was found in the two primary analyses: acupuncture reduced alcohol craving compared with all controls (SMD = −1.24, 95% CI = −1.96 to −0.51); and acupuncture reduced alcohol withdrawal symptoms compared with all controls (SMD = −0.50, 95% CI = −0.83 to −0.17). In secondary analyses: acupuncture reduced craving compared with sham acupuncture (SMD = −1.00, 95% CI = −1.79 to −0.21); acupuncture reduced craving compared with controls in RCTs conducted in Western countries (SMD = −1.15, 95% CI = −2.12 to −0.18); and acupuncture reduced craving compared with controls in RCTs with only male participants (SMD = −1.68, 95% CI = −2.62 to −0.75).
This study showed that acupuncture was potentially effective in reducing alcohol craving and withdrawal symptoms and could be considered as an additional treatment choice and/or referral option within national healthcare systems.
This Meta-analysis is only a little better than the first, I am afraid. What its conclusions do not sufficiently reflect, in my view, is the fact that the quality of the primary studies was mostly very poor – too poor to draw conclusions from (other than ‘acupuncture research is usually lousy’; see figure below). Therefore, I fail to see how the authors could draw the relatively firm and positive conclusions cited above. In my view, they should have stated something like this: DUE TO THE RISK OF BIAS IN MANY TRIALS, THE EFFECTIVENESS OF ACUPUNCTURE REMAINS UNPROVEN.
The authors of the first meta-analysis open the discussion by proudly declaring that “the present study is the first meta-analysis to examine the effect of acupuncture treatment on patients with alcohol use disorder and to provide data on the magnitude of this effect on alcohol-related clinical symptoms and behaviours.” They discretely overlook this meta-analysis from 2009 (and several others which even their rudimentary search would have identified):
Nineteen electronic databases, including English, Korean, Japanese, and Chinese databases, were systematically searched for RCTs of acupuncture for alcohol dependence up to June 2008 with no language restrictions. The methodological qualities of eligible studies were assessed using the criteria described in the Cochrane Handbook.
Eleven studies, which comprised a total of 1,110 individual cases, were systematically reviewed. Only 2 of 11 trials reported satisfactorily all quality criteria. Four trials comparing acupuncture treatment and sham treatments reported data for alcohol craving. Three studies reported that there were no significant differences. Among 4 trials comparing acupuncture and no acupuncture with conventional therapies, 3 reported significant reductions. No differences between acupuncture and sham treatments were found for completion rates (Risk Ratio = 1.07, 95% confidence interval, CI = 0.91 to 1.25) or acupuncture and no acupuncture (Risk Ratio = 1.15, 95% CI = 0.79 to 1.67). Only 3 RCTs reported acupuncture-related adverse events, which were mostly minimal.
The results of the included studies were equivocal, and the poor methodological quality and the limited number of the trials do not allow any conclusion about the efficacy of acupuncture for treatment of alcohol dependence. More research and well-designed, rigorous, and large clinical trials are necessary to address these issues.
One does not need to be an expert in interpreting meta-analyses, I think, to see that this paper is more rigorous than the new ones (which incidentally were published in the very dubious journals). And this is why I trust the conclusions of this last-named meta-analysis more than those of the new one: the efficacy of acupuncture remains unproven. And this means that we should not employ or promote it for routine care.