George Lewith has died on 17 March, aged 67. He was one of the most productive researchers of alternative medicine in the UK; specifically he was interested in acupuncture. If you search this blog, you find several posts that mention him or are entirely dedicated to his work. Undeniably, my own views and research were often very much at odds with those of Lewith.
Wikipedia informs us that Lewith graduated from Trinity College, Cambridge in medicine and biochemistry. He then went on to Westminster Medical School to complete his clinical studies and began working clinically in 1974. In 1977 Lewith became a member of the Royal College of Physicians. Then, in 1980, he became a member of the Royal College of General Practitioners and, later in 1999, was elected a fellow of the Royal College of Physicians.
He was a Professor of Health Research in the Department of Primary Care at the University of Southampton and a director of the International Society for Complementary Medicine Research. Lewith has obtained a significant number of institutional peer reviewed fellowships at doctoral and post-doctoral level and has been principal investigator or collaborator in research grants totally over £5 million during the last decade.
Between 1980 and 2010, Lewith was a partner at the Centre for Complementary and Integrated Medicine, a private practice providing complementary treatments with clinics in London and Southampton.
A tribute by the British Acupuncture Council is poignant, in my view: “George was a friend not only to all of the acupuncture profession, be it traditional, medical or physiotherapist – he was a member of all three professional bodies – but to the whole of complementary medicine. As well as being a research leader he was also politically savvy, working tirelessly up front and behind the scenes to try to bring acupuncture and CAM further into the mainstream. Nobody did more.”
I find it poignant because it hints at the many differences I (and many others) had with Lewith during the last 25 years that I knew him. George was foremost a proponent of acupuncture. His 1985 book – the first of many – advocated treating many internal diseases with acupuncture!
George did not strike me as someone who had the ability or even the ambition to use science for finding the truth and for falsifying hypotheses; in my view, he employed it to confirm his almost evangelic belief. In the pursuit of this all-important aim he did indeed spend a lot of time and energy pulling strings, including on the political level. George was undoubtedly successful but the question has to be asked to what extent this was due to his tireless work ‘behind the scenes’.
In my view, George was a typical example of someone who first and foremost was an advocate and a researcher second. During my time in Exeter, I have met numerous co-workers who had the same problem. Almost without exception, I found that it is impossible to turn such a person into a decent scientist. The advocacy of alternative medicine always got in the way of objectivity and rationality, qualities that are, of course, essential for good science.
George’s very first publication on acupuncture was a ‘letter to the editor’ published in the BMJ. In it, he announced that he is planning to conduct a trial of acupuncture and stated that “acupuncture will be compared to an equally magical placebo”. Yes, George always had the ability to make me laugh; and this is why I will miss him.
One phenomenon that can be noted more frequently than any other in alternative medicine research is that studies arrive at wrong or misleading conclusions. This is more than a little disappointing, not least because it is the conclusion of a trial that is often picked up by health writers and others who in turn mislead the public. On this blog, we must have seen hundreds of examples of this irritating phenomenon. Here is yet another one. This study, a randomized, parallel, open-label exploratory trial, evaluated and compared the effects of systemic manual acupuncture, periauricular electroacupuncture and distal electroacupuncture for treating patients with tinnitus. It included patients who suffered from idiopathic tinnitus for more than two weeks were recruited. They were divided into three groups:
- systemic manual acupuncture group (MA),
- periauricular electroacupuncture group (PE),
- distal electroacupuncture group (DE).
Nine acupoints (TE 17, TE21, SI19, GB2, GB8, ST36, ST37, TE3 and TE9), two periauricular acupoints (TE17 and TE21), and four distal acupoints (TE3, TE9, ST36, and ST37) were selected. The treatment sessions were performed twice weekly for a total of 8 sessions over 4 weeks. Outcome measures were the tinnitus handicap inventory (THI) score and the loud and uncomfortable visual analogue scales (VAS). Demographic and clinical characteristics of all participants were compared between the groups upon admission using one-way analysis of variance (ANOVA). One-way ANOVA was used to evaluate the THI, VAS loud, and VAS uncomfortable scores. The least significant difference test was used as a post-hoc test. In total, 39 subjects were eligible for analysis. No differences in THI and VAS loudness scores were observed between groups. The VAS uncomfortable scores decreased significantly in MA and DE compared with those in PE. Within the group, all three treatments showed some effect on THI, VAS loudness scores and VAS uncomfortable scores after treatment except DE in THI. The authors concluded that there was no statistically significant difference between systemic manual acupuncture, periauricular electroacupuncture and distal electroacupuncture in tinnitus. However, all three treatments had some effect on tinnitus within the group before and after treatment. Systemic manual acupuncture and distal electroacupuncture have some effect on VAS. Neither of the three treatments tested in this study have been previously proven to work. Therefore, it is quite simply nonsensical to compare them. Comparative studies are indicated only with therapies that have a solid evidence-base. They are called ‘superiority trials’ and require a different statistical approach as well as much larger sample sizes. In other words, this study was an unethical waste of resources from the outset. With this in mind, there is only one conclusion that fits the data: there was no statistically significant difference between the three types of acupuncture. The data are therefore in keeping with the notion that all three are placebos. Alternatively one might conclude more clearly for those who are otherwise resistant to learning a lesson: POORLY DESIGNED CLINICAL TRIALS ARE UNETHICAL AND NEVER LEND THEMSELVES TO MEANINGFUL CONCLUSIONS.
A new study published in JAMA investigated the long-term effects of acupuncture compared with sham acupuncture and being placed in a waiting-list control group for migraine prophylaxis. The trial was a 24-week randomized clinical trial (4 weeks of treatment followed by 20 weeks of follow-up). Participants were randomly assigned to 1) true acupuncture, 2) sham acupuncture, or 3) a waiting-list control group. The trial was conducted from October 2012 to September 2014 in outpatient settings at three clinical sites in China. Participants 18 to 65 years old were enrolled with migraine without aura based on the criteria of the International Headache Society, with migraine occurring 2 to 8 times per month.
Participants in the true acupuncture and sham acupuncture groups received treatment 5 days per week for 4 weeks for a total of 20 sessions. Participants in the waiting-list group did not receive acupuncture but were informed that 20 sessions of acupuncture would be provided free of charge at the end of the trial. Participants used diaries to record migraine attacks. The primary outcome was the change in the frequency of migraine attacks from baseline to week 16. Secondary outcome measures included the migraine days, average headache severity, and medication intake every 4 weeks within 24 weeks.
A total of 249 participants 18 to 65 years old were enrolled, and 245 were included in the intention-to-treat analyses. Baseline characteristics were comparable across the 3 groups. The mean (SD) change in frequency of migraine attacks differed significantly among the 3 groups at 16 weeks after randomization; the mean (SD) frequency of attacks decreased in the true acupuncture group by 3.2 (2.1), in the sham acupuncture group by 2.1 (2.5), and the waiting-list group by 1.4 (2.5); a greater reduction was observed in the true acupuncture than in the sham acupuncture group (difference of 1.1 attacks; 95% CI, 0.4-1.9; P = .002) and in the true acupuncture vs waiting-list group (difference of 1.8 attacks; 95% CI, 1.1-2.5; P < .001). Sham acupuncture was not statistically different from the waiting-list group (difference of 0.7 attacks; 95% CI, −0.1 to 1.4; P = .07).
The authors concluded that among patients with migraine without aura, true acupuncture may be associated with long-term reduction in migraine recurrence compared with sham acupuncture or assigned to a waiting list.
Note the cautious phraseology: “… acupuncture may be associated with long-term reduction …”
The authors were, of course, well advised to be so atypically cautious:
- Comparisons to the waiting list group are meaningless for informing us about the specific effects of acupuncture, as they fail to control for placebo-effects.
- Comparisons between real and sham acupuncture must be taken with a sizable pinch of salt, as the study was not therapist-blind and the acupuncturists may easily have influenced their patients in various ways to report the desired result (the success of patient-blinding was not reported but would have gone some way to solving this problem).
- The effect size of the benefit is tiny and of doubtful clinical relevance.
My biggest concern, however, is the fact that the study originates from China, a country where virtually 100% of all acupuncture studies produce positive (or should that be ‘false-positive’?) findings and data fabrication has been reported to be rife. These facts do not inspire trustworthiness, in my view.
So, does acupuncture work for migraine? The current Cochrane review included 22 studies and its authors concluded that the available evidence suggests that adding acupuncture to symptomatic treatment of attacks reduces the frequency of headaches. Contrary to the previous findings, the updated evidence also suggests that there is an effect over sham, but this effect is small. The available trials also suggest that acupuncture may be at least similarly effective as treatment with prophylactic drugs. Acupuncture can be considered a treatment option for patients willing to undergo this treatment. As for other migraine treatments, long-term studies, more than one year in duration, are lacking.
So, maybe acupuncture is effective. Personally, I am not convinced and certainly do not think that the new JAMA study significantly strengthened the evidence.
You may recall, we have dealt with the JCAM many times before; for instance here, here, here and here. Now they have come out with another remarkable paper. This study – no, the authors called it a ‘pilot study’ – was to compare the efficacy of Emotional Freedom Techniques (EFT) with that of Cognitive-Behavioral Therapy (CBT) in reducing adolescent anxiety. Sixty-three American high-ability students in grades 6–12, ages 10–18 years, who scored in the moderate to high ranges for anxiety on the Revised Children’s Manifest Anxiety Scale-2 (RCMAS-2) were randomly assigned to one of three groups:
- CBT (n = 21),
- EFT (n = 21),
- or waitlist control (n = 21).
EFT is an alternative therapy that incorporates acupoint stimulation. Students assigned to the CBT or EFT treatment groups received three individual sessions of the identified protocols from trained graduate counseling, psychology, or social work students enrolled at a large northeastern research university. The RCMAS-2 was used to assess preintervention and postintervention anxiety levels in participants.
EFT participants showed significant reduction in anxiety levels compared with the waitlist control group with a moderate to large effect size. CBT participants did not differ significantly from the EFT or control.
The authors concluded that EFT is an efficacious intervention to significantly reduce anxiety for high-ability adolescents.
They also state in their abstract that EFT is an evidence-based treatment for anxiety…
Are you happy with these conclusions?
Are you convinced that this trial lends itself to establish efficacy of anything?
Are you impressed with the trial design, the sample size, etc?
Are you sure that EFT is plausible, credible or evidence-based in any way?
If you look up EFT, you will find that there is a surprising amount of papers on it. Most of them have one thing in common: they were published in highly dubious journals. The field does not inspire trust or competence. The authors of the study state that EFT is an easily implemented strategy that uses such techniques as awareness building, exposure, reframing of interpretation, and systematic desensitization, while teaching the participant to self-stimulate protocol-identified acupoints (i.e., acupuncture points) by tapping. The effectiveness of acupuncture for treating anxiety has been well documented. Rather than using acupuncture needles, EFT relies on the manual stimulation of the acupoints. A recent meta-analysis indicated that interventions using acupoint stimulation had a moderate effect size (Hedge’s g = −0.66 95% CI [−0.99, −0.33]) in reducing symptoms. In EFT, the client stimulates the protocol-identified acupoints by tapping on them. Preliminary studies have suggested that tapping and other alternative ways of stimulating acupuncture points to be as effective as acupuncture needling. The EFT protocol and identified acupoints that were used in this study are the ones recommended for research purposes by the Association for Comprehensive Energy Psychology…
Wikipedia tells us that “Emotional Freedom Techniques (EFT) is a form of counseling intervention that draws on various theories of alternative medicine including acupuncture, neuro-linguistic programming, energy medicine, and Thought Field Therapy (TFT). It is best known through Gary Craig’s EFT Handbook, published in the late 1990s, and related books and workshops by a variety of teachers. EFT and similar techniques are often discussed under the umbrella term “energy psychology”. Advocates claim that the technique may be used to treat a wide variety of physical and psychological disorders, and as a simple form of self-administered therapy. The Skeptical Inquirer describes the foundations of EFT as “a hodgepodge of concepts derived from a variety of sources, [primarily] the ancient Chinese philosophy of chi, which is thought to be the ‘life force’ that flows throughout the body.” The existence of this life force is “not empirically supported”. EFT has no benefit as a therapy beyond the placebo effect or any known-effective psychological techniques that may be provided in addition to the purported “energy” technique. It is generally characterized as pseudoscience and it has not garnered significant support in clinical psychology.”
A recent systematic review of EFT concluded that “there were too few data available comparing EFT to standard-of-care treatments such as cognitive behavioral therapy, and further research is needed to establish the relative efficacy of EFT to established protocols.”
Notwithstanding these and many other verdicts on EFT, we now are asked to agree with the new study that EFT IS EFFICACIOUS.
Is this a joke?
They want us to believe this on the basis of a PILOT STUDY? Such studies are not even supposed to test efficacy! (Yet the authors of the trial state that this study was designed to meet the American Psychological Association (APA) Division 12 quality control criteria and the Consolidated Standards for Reporting Trials (CONSORT) criteria. I have to admit, they could have fooled me!)
No, it is not a joke, it is yet another nonsense from the ‘The Journal of Complementary and Alternative Medicine’ which, in my view, should henceforth be called THE JOURNAL OF ALTERNATIVE FACTS (JAF).
Mike Cummings recently stated on this blog “I’m not into blog banter.” Is he perhaps referring to some ‘alternative facts’? The truth seems to be that he blogs happily, regularly and – I am sad to say – disgracefully. This is a quote from his new post about the discussions regarding an acupuncture trial which was in the press a few days ago, (and also has been discussed on this blog):
START OF QUOTE
So there has been a big response to this paper press released by BMJ on behalf of the journal Acupuncture in Medicine. The response has been influenced by the usual characters – retired professors who are professional bloggers and vocal critics of anything in the realm of complementary medicine. They thrive on oiling up and flexing their EBM muscles for a baying mob of fellow sceptics (see my ‘stereotypical mental image’ here). Their target in this instant is a relatively small trial on acupuncture for infantile colic. Deserving of being press released by virtue of being the largest to date in the field, but by no means because it gave a definitive answer to the question of the efficacy of acupuncture in the condition. We need to wait for an SR where the data from the 4 trials to date can be combined. On this occasion I had the pleasure of joining a short segment on the Today programme on BBC Radio 4 led by John Humphreys. My protagonist was the ever-amusing David Colquhoun (DC), who spent his short air-time complaining that the journal was even allowed to be published in the first place. You can learn all about DC care of Wikipedia – he seems to have a surprisingly long write up for someone whose profession career was devoted to single ion channels, perhaps because a significant section of the page is devoted to his activities as a quack-busting blogger. So why would BBC Radio 4 invite a retired basic scientist and professional sceptic blogger to be interviewed alongside one of the journal editors – a clinician with expertise in acupuncture (WMA)? At no point was it made manifest that only one of the two had ever been in a position to try to help parents with a baby that they think cries excessively. Of course there are a lot of potential causes of excessive crying, but I am sure DC would agree that it is unlikely to be attributable to a single ion channel…
END OF QUOTE
I encourage everyone to read Cummings post in full; it’s full of surprises. Here I just want to comment very briefly why I find his post disgraceful (the Cummings quotes are in bold followed by my comments):
….usual characters….. Disrespectful to the point of being derogatory, in my view
….retired professors….. Not true, non-retired professionals commented as well
….professional bloggers…. Meaning people who earn their income by blogging? Certainly not true!
….vocal critics of anything in the realm of complementary medicine…. Critic not of ‘anything’ but merely of things that are false or misleading like the trial in question
….a baying mob of fellow sceptics…. Unquestionably meant to be insulting and arguably libelous
….Deserving of being press released by virtue of being the largest to date in the field…. Large is not necessarily a virtue that merits a press-release, particularly, if it is not matched with quality
….We need to wait for an SR where the data from the 4 trials to date can be combined…. More than doubtful that we ‘need to wait’. The 4 trials in question are all very weak and therefore cannot provide a firm answer via a systematic review
….the ever-amusing David Colquhoun…. Derogatory to the extreme
….why would BBC Radio 4 invite a retired basic scientist and professional sceptic blogger…. The answer could be because he understands science and has vast experience exposing false claims
….only one of the two had ever been in a position to try to help parents with a baby that they think cries excessively…. This does not necessarily mean that such a person understands science, and Cummings might even be an example for this
….is unlikely to be attributable to a single ion channel…. Even Cummings’ attempts at humour are quite appalling.
The comments of Dr Mike Cummings MB ChB Dip Med Ac, I am afraid, befit an ill-educated acupuncturist who feels personally hurt because his views have been challenged and who is not quite bright enough to have a rational discussion about his favourite subject, particularly with someone who has a superior grasp of the issues at hand (which are clearly not ‘how to stick a needle in a baby’). However, Cummings is not a simple acupuncturist; he happens to be a member of the medical profession, a medical director of the British Medical Acupuncture Society and (as he seems keen to point out) a journal editor. With these credentials, he should, in my view, be able to argue a bit more intelligently, truthfully and a lot more gracefully.
One could almost think he wants to give acupuncture a bad name!!!
Acupuncture is often recommended as a treatment for shoulder pain, but its effectiveness is far from proven. A new study has just been published; but does it change this uncertainty?
A total of 227 patients with subacromial pain syndrome were recruited to this RCT. The patients were allocated to three groups who received either A) group exercise, B) group exercise plus acupuncture or C) group exercise plus electro-acupuncture. The primary outcome measure was the Oxford Shoulder Score. Follow-up was post treatment, and at 6 and 12 months. Data were analysed on intention-to-treat principles with imputation of missing values.
Treatment groups were similar at baseline. All treatment groups demonstrated improvements over time. Between-group estimates were, however, small and non-significant.
The authors concluded that neither acupuncture nor electro-acupuncture were found to be more beneficial than exercise alone in the treatment of subacromial pain syndrome.
Well, that was to be expected!… I hear the rationalists amongst us exclaim.
Actually, I am not so sure.
One could easily have expected that the acupuncture groups (B and C) show a significant advantage over group A.
Because acupuncture is a ‘theatrical placebo’, a ritual that impresses patients and thus impacts on results, particularly on subjective outcomes like pain. If the results had shown a benefit for acupuncture + exercise (groups B and C) versus exercise alone (group A), what would we have made of it? Acupuncture fans would surely have claimed that it is evidence confirming acupuncture’s effectiveness. Sceptics, on the other hand, would have rightly insisted that it demonstrates nothing of the sort – it merely confirms that placebo effects can affect clinical outcomes such as pain.
As it turned out, however, this trial results happened to indicate that these placebo-effects can be so small that they fail to reach the level of statistical significance.
I think there is one noteworthy message here: RCTs with such a design (no adequate control for placebo effects) can easily generate false-positive results (in this case, this did not happen, but it was nevertheless a possible outcome). Such studies are popular but utterly useless: they don’t advance our knowledge one single iota. If that is so, we should not waste our resources on them because, in the final analysis, this is not ethical. In other words, we must stop funding research that has little or no chance of advancing our knowledge.
This new RCT was embargoed until today; so, I had to wait until I was able to publish my comments. Here are the essentials of the study:
The Swedish investigators compared the effect of two types of acupuncture versus no acupuncture in infants with colic in public child health centres (CHCs). The study was designed as a multicentre, randomised controlled, single-blind, three-armed trial (ACU-COL) comparing two styles of acupuncture with no acupuncture, as an adjunct to standard care. Among 426 infants whose parents sought help for colic and registered their child’s fussing/crying in a diary, 157 fulfilled the criteria for colic and 147 started the intervention.
Parallel to usual care, study participants visited the study CHC twice a week for 2 weeks. Thus, all infants received usual care plus 4 extra visits to a CHC, during which parents met a nurse for 20–30 min and were able to discuss their infant’s symptoms. Together these were considered to represent gold standard care. The nurse listened, and gave evidence-based advice and calming reassurance. Breastfeeding mothers were encouraged to continue breastfeeding. At each visit, the study nurse carried the infant to a separate treatment room where they were left alone with the acupuncturist for 5 min.
The acupuncturist treated the baby according to group allocation and recorded the treatment procedures and any adverse events. Disposable stainless steel 0.20×13 mm Vinco needles (Helio, Jiangsu Province, China) were used. Infants allocated to group A received standardised MA at LI4. One needle was inserted to a depth of approximately 3 mm unilaterally for 2–5 s and then withdrawn without stimulation. Infants allocated to group B received semi-standardised individualised acupuncture, mimicking clinical TCM practice. Following a manual, the acupuncturists were able to choose one point, or any combination of Sifeng, LI4 and ST36, depending on the infant’s symptoms, as reported in the diary. A maximum of five insertions were allowed per treatment. Needling at Sifeng consisted of 4 insertions, each to a depth of approximately 1 mm for 1 s. At LI4 and ST36, needles were inserted to a depth of approximately 3 mm, uni- or bilaterally. Needles could be retained for 30 seconds. De qi was not sought, therefore stimulation was similarly minimal in groups A and B. Infants in group C spent 5 min alone with the acupuncturist without receiving acupuncture.
The effect of the two types of acupuncture was similar and both were superior to gold standard care alone. Relative to baseline, there was a greater relative reduction in time spent crying and colicky crying by the second intervention week (p=0.050) and follow-up period (p=0.031), respectively, in infants receiving either type of acupuncture. More infants receiving acupuncture cried <3 hours/day, and thereby no longer fulfilled criteria for colic, in the first (p=0.040) and second (p=0.006) intervention weeks. No serious adverse events were reported.
The authors concluded that acupuncture appears to reduce crying in infants with colic safely.
Notice that the investigators are cautious and state in the abstract that “acupuncture appears to reduce crying…” Their conclusions from the actual article are, however, quite different; here they state the following:
Among those initially experiencing excessive infant crying, the majority of parents reported normal values once the infant’s crying had been evaluated in a diary and a diet free of cow’s milk had been introduced. Therefore, objective measurement of crying and exclusion of cow’s milk protein are recommended as first steps, to avoid unnecessary treatment. For those infants that continue to cry >3 hours/day, acupuncture may be an effective treatment option. The two styles of MA tested in ACU-COL had similar effects; both reduced crying in infants with colic and had no serious side effects. However, there is a need for further research to find the optimal needling locations, stimulation and treatment intervals.
Such phraseology is much more assertive and seems to assume acupuncture caused specific therapeutic effects. Yet, I think, this assumption is not warranted.
In fact, I believe, the study shows almost the opposite of what the authors conclude. Both minimal and TCM acupuncture seemed to reduce the symptoms of colic compared to no acupuncture at all. I think, this confirms previous research showing that acupuncture is a ‘theatrical placebo’. The study was designed without an adequate placebo group. It would have been easy to use some form of sham acupuncture in the control group. Why did the authors not do that? Heaven knows, but one might speculate that they were aiming for a positive result – and what better way to ensure it than with a ‘no treatment’ control group?
There are, of course, numerous other flaws. For instance, Prof David Colquhoun FRS, Professor of Pharmacology at University College London, criticised the study because of its lousy statistics:
START OF QUOTE
“It is truly astonishing that, in the 21st century, the BMJ still publishes a journal devoted to a form of pre-scientific medicine which after more than 3000 trials has still not been able to produce convincing evidence of efficacy1. Like most forms of alternative medicine, acupuncture has been advocated for a vast range of problems, and there is little evidence that it works for any of them. Colic has not been prominent in these claims. What parent would think that sticking needles into their baby would stop it crying? The idea sounds bizarre. It is. This paper certainly doesn’t show that it works.
“The statistical analysis in the paper is incompetent. This should have been detected by the referees, but wasn’t. For a start, the opening statement, ‘A two-sided P value ≤0.05 was considered statistically significant’ is simply unacceptable in the light of all recent work about reproducibility. Still worse, Table 1 uses the description ‘statistical tendency towards significance (p=0.051–0.1)’.
“Worst of all, Table 1 reports 24 different P values, of which three are (just) below 0.05. Yet no correction has been used for multiple comparisons. This is very bad practice. It’s highly unlikely that, if the proper correction had been done, any of the results would have given a type 1 error rate below 5%.
“Even were it not for this, most of the ‘significant’ P values are marginal (only slightly less than 0.05). It is now well known that the type 1 error rate gives an optimistic view. What matters is the false positive rate – the chance that a ‘significant’ result is a false positive. A p-value close to 0.05 implies that there is at least a 30% chance that they are false positives. If one thought, a priori, that the chance of colic being cured by sticking needles into a baby was less than 50%, the false positive rate could easily be greater than 80%2. It is now recognised that this misinterpretation of p-values is a major contributor to the crisis of reproducibility.
“Other problems concern the power calculation. A priori calculations of power are well-known to be overoptimistic, because small trials usually overestimate the effect size. In this case the initial estimated sample size was not attained, and a rather mysterious recalculation of power was used.
“Another small problem: the discussion points out that ‘the majority of infants in this cohort did not have colic’.
“The nature of the control group is not very clear. An appropriate control might have been to cuddle the baby – this was used in a study in which another implausible treatment, chiropractic, was shown not to work. This appears not to have been done.
“Lastly, p-values are reported in the text without mention of effect sizes. This is contrary to all statistical advice.
“In conclusion, the design of the trial is reasonable (apart from the control group) but the statistical analysis is appalling. It’s very likely that there aren’t any real effects of acupuncture at all. This paper serves more to muddy the waters than to add useful information. It’s a model for the sort of mistakes that have led to the crisis in reproducibility. The BMJ should not be publishing this sort of stuff, and the referees seem to have no understanding of statistics.”
END OF QUOTE
Despite these rather obvious – some would say fatal – flaws, the editor of ACUPUNCTURE IN MEDICINE (AIM) thought this trial to be so impressively rigorous that he issued a press-release about it. This, I think, is particularly telling, perhaps even humorous: it shows what kind of a journal AIM is, and also provides an insight into the state of acupuncture research in general.
The long and short of it is that conclusions about specific therapeutic effects of acupuncture are not permissible. We know that colicky babies respond even to minimal attention, and this trial confirms that even a little additional TLC in the form of acupuncture will generate an effect. The observed outcome is most likely unrelated to acupuncture.
If you want to scientifically investigate this question, it might be a good idea NOT to start with the following sentence: “Auricular acupuncture (AA) is effective in the treatment of preoperative anxiety”. Yet, this is exactly what the authors did in their recent publication.
The aim of this new study was to investigate whether AA can reduce exam anxiety as compared to placebo and no intervention. Forty-four medical students were randomized to receive AA, placebo, or no intervention in a crossover manner. Subsequently they completed three comparable oral anatomy exams with an interval of one month between the exams/interventions.
A licensed acupuncturist with more than five years of experience with this technique applied AA at the acupuncture points MA-IC1 (Lung), MA-TF1 (ear Shenmen), MA-SC (Kidney), MA-AT1 (Subcortex) and MA-TG (Adrenal gland) bilaterally. Indwelling fixed ‘New Pyonex’ needles embedded in a skin-coloured adhesive tape were used for AA. The participants were instructed by the acupuncturist to stimulate the auricular needles for 3–5 minutes, if they felt anxious. For the placebo procedure, ‘New Pyonex’ placebo needles were attached to five sites on the helix of the auricle bilaterally. ‘New Pyonex’ placebo needles have the same appearance as AA needles but consist of self-adhesive tape only. In order to avoid potential physiologic effects of acupressure, the participants were not instructed to stimulate the attached ‘New Pyonex’ placebo needles. AA and placebo needles were left in situ until the next day and were removed out of sight of the participants after the exam by the investigator, who was not involved in acupuncture procedure
Levels of anxiety were measured using a visual analogue scale before and after each intervention as well as before each exam. Additional measures included the State-Trait-Anxiety Inventory, duration of sleep at night, blood pressure, heart rate and the extent of participant blinding.
All included participants finished the study. Anxiety levels were reduced after AA and placebo intervention compared to baseline and the no intervention condition (p < 0.003). Moreover, AA was also better at reducing anxiety than placebo in the evening before the exam (p = 0.018). Participants were able to distinguish between AA and placebo intervention.
The authors concluded that both auricular acupuncture and placebo procedure were shown to be effective in reducing levels of exam anxiety in medical students. The superiority of verum AA over placebo AA and no intervention is considered to be due to stimulation of cranial nerves, but may have been increased in effect by insufficient participant blinding.
Here are just three of the major concerns I have about this study:
- The trial design seems odd: a crossover study can only work well, if there is a stable baseline. This may not be the case with three consecutive exams; the anxiety experienced by students is bound to get less as time goes by. I think anyone who has passed a series of exams will confirm that there is a large degree of habituation.
- It seems inadequate to employ just one acupuncturist; it means that the trial might end up testing not acupuncture per se but the skills of the therapist.
- The placebo used for this study cannot possibly have fooled anyone into believing that it was real AA; volunteers were not even instructed to ‘stimulate’ the placebo devices. The difference to the ‘real thing’ must have been very clear to all involved. This means that the control for placebo-effects was woefully incomplete. In turn, this means that the observed outcomes are most likely due to residual bias.
In view of these concerns, allow me to re-phrase the authors’ conclusions:
THE RESULTS OF THIS POORLY-DESIGNED STUDY ARE DIFFICULT TO INTERPRET. MOST LIKELY THEY SHOW THAT ACUPUNCTURE IS NOT EFFECTIVE BUT MERELY WORKS THROUGH A PLACEBO-RESPONSE.
The ACUPUNCTURE NOW FOUNDATION (ANF) has featured on this blog before. Today I want to re-introduce them because I just came across one of their articles which I found remarkable. In it, they define what many of us have often wondered about: the most important myth about acupuncture.
Is it acupuncture’s current popularity, its long history, its mode of action, its efficacy, its safety?
No, here is the answer directly from the ANF:
The most important myth that needs to be put to rest is the idea promoted by a small group of vocal critics that acupuncture is nothing more than a placebo. Many cite the fact that studies showing acupuncture to be highly effective were of low quality and that several higher quality studies show that, while acupuncture was clinically effective, it usually does not outperform “sham” acupuncture. But those studies are dominated by the first quality issue cited above; studies with higher methodological rigor where the “real” acupuncture was so poorly done as to not be a legitimate comparison. Yet despite the tendency toward poor quality acupuncture in studies with higher methodological standards, a benchmark study was done that showed “real” acupuncture clearly outperforming “sham” acupuncture in four different chronic pain conditions.3 When you add this study together with the fact veterinary acupuncture is used successfully in many different animals, the idea of acupuncture only being placebo must now be considered finally disproven. This is further supported by studies which show that the underlying physiological pathways activated by acupuncture sometimes overlap, but can be clearly differentiated from, those activated by placebo responses.
Yes, I was too.
The myth, according to the ANF, essentially is that sceptics do not understand the scientific evidence. And these blinkered sceptics even go as far as ignoring the findings from what the ANF consider to be a ‘benchmark study’! Ghosh, that’s nasty of them!!!
But, no – the benchmark study (actually, it was not a ‘study’ but a meta-analysis of studies) has been discussed fully on this blog (and in many other places too). Here is what I wrote in 2012 when it was first published:
An international team of acupuncture trialists published a meta-analysed of individual patient data to determine the analgesic effect of acupuncture compared to sham or non-acupuncture control for the following 4 chronic pain conditions: back and neck pain, osteoarthritis, headache, and shoulder pain. Data from 29 RCTs, with an impressive total of 17 922 patients, were included.
The results of this new evaluation suggest that acupuncture is superior to both sham and no-acupuncture controls for each of these conditions. Patients receiving acupuncture had less pain, with scores that were 0.23 (95% CI, 0.13-0.33), 0.16 (95% CI, 0.07-0.25), and 0.15 (95% CI, 0.07-0.24) SDs lower than those of sham controls for back and neck pain, osteoarthritis, and chronic headache, respectively; the effect sizes in comparison to no-acupuncture controls were 0.55 (95% CI, 0.51-0.58), 0.57 (95% CI, 0.50-0.64), and 0.42 (95% CI, 0.37-0.46) SDs.
Based on these findings, the authors reached the conclusion that “acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option. Significant differences between true and sham acupuncture indicate that acupuncture is more than a placebo. However, these differences are relatively modest, suggesting that factors in addition to the specific effects of needling are important contributors to the therapeutic effects of acupuncture”.
… even the enthusiastic authors of this article admit that, when compared to sham, the effect size of real acupuncture is too small to be clinically relevant. Therefore one might argue that this meta-analysis confirms what critics have suggested all along: acupuncture is not a useful treatment for clinical routine.
Unsurprisingly, the authors of the meta-analysis do their very best to play down this aspect. They reason that, for clinical routine, the comparison between acupuncture and non-acupuncture controls is more relevant than the one between acupuncture and sham. But this comparison, of course, includes placebo- and other non-specific effects masquerading as effects of acupuncture – and with this little trick (which, by the way is very popular in alternative medicine), we can, of course, show that even sugar pills are effective.
I do not doubt that context effects are important in patient care; yet I do doubt that we need a placebo treatment for generating such benefit in our patients. If we administer treatments which are effective beyond placebo with kindness, time, compassion and empathy, our patients will benefit from both specific and non-specific effects. In other words, purely generating non-specific effects with acupuncture is far from optimal and certainly not in the interest of our patients. In my view, it cannot be regarded as not good medicine, and the authors’ conclusion referring to a “reasonable referral option” is more than a little surprising in my view.
Acupuncture-fans might argue that, at the very minimum, the new meta-analysis does demonstrate acupuncture to be statistically significantly better than a placebo. Yet I am not convinced that this notion holds water: the small residual effect-size in the comparison of acupuncture with sham might not be the result of a specific effect of acupuncture; it could be (and most likely is) due to residual bias in the analysed studies.
The meta-analysis is strongly driven by the large German trials which, for good reasons, were heavily and frequently criticised when first published. One of the most important potential drawbacks was that many participating patients were almost certainly de-blinded through the significant media coverage of the study while it was being conducted. Moreover, in none of these trials was the therapist blinded (the often-voiced notion that therapist-blinding is impossible is demonstrably false). Thus it is likely that patient-unblinding and the absence of therapist-blinding importantly influenced the clinical outcome of these trials thus generating false positive findings. As the German studies constitute by far the largest volume of patients in the meta-analysis, any of their flaws would strongly impact on the overall result of the meta-analysis.
So, has this new meta-analysis finally solved the decades-old question about the effectiveness of acupuncture? It might not have solved it, but we have certainly moved closer to a solution, particularly if we employ our faculties of critical thinking. In my view, this meta-analysis is the most compelling evidence yet to demonstrate the ineffectiveness of acupuncture for chronic pain.
END OF QUOTE
The ANF-text then goes from bad to worse. First they cite the evidence from veterinary acupuncture as further proof of the efficacy of their therapy. Well, the only systematic review in this are is, I think, by my team; and it concluded that there is no compelling evidence to recommend or reject acupuncture for any condition in domestic animals. Some encouraging data do exist that warrant further investigation in independent rigorous trials.
Lastly, the ANF mentions acupuncture’s mode of action which they seem to understand clearly and fully. Congratulations ANF! In this case, you are much better than the many experts in basic science or neurology who almost unanimously view these ‘explanations’ of how acupuncture might work as highly adventurous hypotheses or speculations.
So, what IS the most important myth about acupuncture? I am not sure and – unlike the ANF – I do not feel that I can speak for the rest of the world, but one of the biggest myths FOR ME is how acupuncture fans constantly manage to mislead the public.
Is acupuncture a pseudoscience? An interesting question! It was used as the title of a recent article. Knowing who authored it, the question unfortunately promised to be rhetorical. Dr Mike Cummings is (or was?) the ‘Medical Director at British Medical Acupuncture Society’ – hardly a source of critical or sceptical thinking about acupuncture, I’d say. The vast majority of his recent publications are in ‘ACUPUNCTURE IN MEDICINE’ and his blog post too is for that journal. Nevertheless, his thoughts might be worth considering, and therefore I present the essence of his post below [the footnotes refer to my comments following Cummings’ article]:
…Wikipedia has branded acupuncture as pseudoscience and its benefits as placebo . ‘Acupuncture’ is clearly is not pseudoscience; however, the way in which it is used or portrayed by some may on occasion meet that definition. Acupuncture is a technique that predates the development of the scientific method  … so it is hardly fair to classify this ancient medical technique within that framework . It would be better to use a less pejorative classification within the bracket of history when referring to acupuncture and other ancient East Asian medical techniques . The contemporary use of acupuncture within modern healthcare is another matter entirely, and the fact that it can be associated with pre-scientific medicine does not make it a pseudoscience.
The Wikipedia acupuncture page is extensive and currently runs to 302 references. But how do we judge the quality or reliability of a text or its references? … I would generally look down on blogs, such as this, because they lack … hurdles prior to publication . Open peer review was introduced relatively recently associated with immediate publication. But all this involves researchers and senior academics publishing and reviewing within their own fields of expertise. Wikipedia has a slightly different model built on five pillars. The second of those pillars reads:
Wikipedia is written from a neutral point of view: We strive for articles that document and explain major points of view, giving due weight with respect to their prominence in an impartial tone. We avoid advocacy and we characterize information and issues rather than debate them. In some areas there may be just one well-recognized point of view; in others, we describe multiple points of view, presenting each accurately and in context rather than as “the truth” or “the best view”. All articles must strive for verifiable accuracy, citing reliable, authoritative sources, especially when the topic is controversial or is on living persons. Editors’ personal experiences, interpretations, or opinions do not belong.
Experts within a field may be seen to have a certain POV (point of view), and are discouraged from editing pages directly because they cannot have the desired NPOV (neutral POV). This is a rather unique publication model in my experience, although the editing and comments are all visible and traceable, so there is no hiding… apart from the fact that editors are allowed to be entirely anonymous. Have a look at the talk page behind the main acupuncture page on Wikipedia. You may be shocked by the tone of much of the commentary. It certainly does not seem to comply with the fourth of the five pillars, which urges respect and civility, and in my opinion results primarily from the security of anonymity. I object to the latter, but there is always a balance to be found between freedom of expression (enhanced for some by the safety of anonymity) and cyber bullying (almost certainly fuelled in part by anonymity). That balance requires good moderation, and whilst there was some evidence of moderation on the talk page, it was inadequate to my mind… I might move to drop anonymity from Wikipedia if moderation is wanting.
Anyway my impression, for what it’s worth, is that the acupuncture page on Wikipedia is not written from an NPOV, but rather it appears to be controlled by semi professional anti-CAM pseudosceptics . I have come across these characters  regularly since I was introduced to the value of needling in military general practice. I have a stereotypical mental image: plain or scary looking bespectacled geeks and science nuts , the worst are often particle physicists … Interacting with them is at first intense, but rapidly becomes tedious as they know little of the subject detail , fall back on the same rather simplistic arguments  and ultimately appear to be motivated by eristic discourse rather than the truth .
I am not surprised that they prefer to close the comments, because I imagine that some people might object rather strongly to many of the statements made in this text.
Here are my short comments: I should perhaps stress that I am not the author of nor a contributor to this Wiki (or any other) page.  Is this an attempt to employ the ‘appeal to tradition’ fallacy?  The Wiki page does by no means classify the ancient history of acupuncture as pseudoscience.  I have always felt that classification of science or medicine according to geography is nonsensical; they should not be classified as Western or Asian but as sound or not, effective or not, etc.  As we have often seen on this blog, the ‘hurdles’ (peer-review) are often laughable, particularly in the realm of alternative medicine.  This article is essentially trying to show that the Wiki page is biased. Yet it ends with a bonanza of insults which essentially reveal the profound bias of the author.
IS ACUPUNCTURE PSEUDOSCIENCE? Cummings’ article promised to address this question. Sadly it did nothing of the sort. It turned out to be an incompetent rant about a Wiki page. If anything, Cummings contributed to the neutral reader of his text getting convinced that, indeed, acupuncture IS a pseudoscience! At least Wiki used facts, arguments, evidence etc. and it went a lot further in finding a rational answer to this intriguing question.