I remember reading this paper entitled ‘Comparison of acupuncture and other drugs for chronic constipation: A network meta-analysis’ when it first came out. I considered discussing it on my blog, but then decided against it for a range of reasons which I shall explain below. The abstract of the original meta-analysis is copied below:
The objective of this study was to compare the efficacy and side effects of acupuncture, sham acupuncture and drugs in the treatment of chronic constipation. Randomized controlled trials (RCTs) assessing the effects of acupuncture and drugs for chronic constipation were comprehensively retrieved from electronic databases (such as PubMed, Cochrane Library, Embase, CNKI, Wanfang Database, VIP Database and CBM) up to December 2017. Additional references were obtained from review articles. With quality evaluations and data extraction, a network meta-analysis (NMA) was performed using a random-effects model under a frequentist framework. A total of 40 studies (n = 11032) were included: 39 were high-quality studies and 1 was a low-quality study. NMA showed that (1) acupuncture improved the symptoms of chronic constipation more effectively than drugs; (2) the ranking of treatments in terms of efficacy in diarrhoea-predominant irritable bowel syndrome was acupuncture, polyethylene glycol, lactulose, linaclotide, lubiprostone, bisacodyl, prucalopride, sham acupuncture, tegaserod, and placebo; (3) the ranking of side effects were as follows: lactulose, lubiprostone, bisacodyl, polyethylene glycol, prucalopride, linaclotide, placebo and tegaserod; and (4) the most commonly used acupuncture point for chronic constipation was ST25. Acupuncture is more effective than drugs in improving chronic constipation and has the least side effects. In the future, large-scale randomized controlled trials are needed to prove this. Sham acupuncture may have curative effects that are greater than the placebo effect. In the future, it is necessary to perform high-quality studies to support this finding. Polyethylene glycol also has acceptable curative effects with fewer side effects than other drugs.
END OF 1st QUOTE
This meta-analysis has now been retracted. Here is what the journal editors have to say about the retraction:
After publication of this article , concerns were raised about the scientific validity of the meta-analysis and whether it provided a rigorous and accurate assessment of published clinical studies on the efficacy of acupuncture or drug-based interventions for improving chronic constipation. The PLOS ONE Editors re-assessed the article in collaboration with a member of our Editorial Board and noted several concerns including the following:
- Acupuncture and related terms are not mentioned in the literature search terms, there are no listed inclusion or exclusion criteria related to acupuncture, and the outcome measures were not clearly defined in terms of reproducible clinical measures.
- The study included acupuncture and electroacupuncture studies, though this was not clearly discussed or reported in the Title, Methods, or Results.
- In the “Routine paired meta-analysis” section, both acupuncture and sham acupuncture groups were reported as showing improvement in symptoms compared with placebo. This finding and its implications for the conclusions of the article were not discussed clearly.
- Several included studies did not meet the reported inclusion criteria requiring that studies use adult participants and assess treatments of >2 weeks in duration.
- Data extraction errors were identified by comparing the dataset used in the meta-analysis (S1 Table) with details reported in the original research articles. Errors included aspects of the study design such as the experimental groups included in the study, the number of study arms in the trial, number of participants, and treatment duration. There are also several errors in the Reference list.
- With regard to side effects, 22 out of 40 studies were noted as having reported side effects. It was not made clear whether side effects were assessed as outcome measures for the other 18 studies, i.e. did the authors collect data clarifying that there were no side effects or was this outcome measure not assessed or reported in the original article. Without this clarification the conclusion comparing side effect frequencies is not well supported.
- The network geometry presented in Fig 5 is not correct and misrepresents some of the study designs, for example showing two-arm studies as three-arm studies.
- The overall results of the meta-analysis are strongly reliant on the evidence comparing acupuncture versus lactulose treatment. Several of the trials that assessed this comparison were poorly reported, and the meta-analysis dataset pertaining to these trials contained data extraction errors. Furthermore, potential bias in studies assessing lactulose efficacy in acupuncture trials versus lactulose efficacy in other trials was not sufficiently addressed.
While some of the above issues could be addressed with additional clarifications and corrections to the text, the concerns about study inclusion, the accuracy with which the primary studies’ research designs and data were represented in the meta-analysis, and the reporting quality of included studies directly impact the validity and accuracy of the dataset underlying the meta-analysis. As a consequence, we consider that the overall conclusions of the study are not reliable. In light of these issues, the PLOS ONE Editors retract the article. We apologize that these issues were not adequately addressed during pre-publication peer review.
LZ disagreed with the retraction. YM and XD did not respond.
END OF 2nd QUOTE
Let me start by explaining why I initially decided not to discuss this paper on my blog. Already the first sentence of the abstract put me off, and an entire chorus of alarm-bells started ringing once I read further.
- A meta-analysis is not a ‘study’ in my book, and I am somewhat weary of researchers who employ odd or unprecise language.
- We all know (and I have discussed it repeatedly) that studies of acupuncture frequently fail to report adverse effects (in doing this, their authors violate research ethics!). So, how can it be a credible aim of a meta-analysis to compare side-effects in the absence of adequate reporting?
- The methodology of a network meta-analysis is complex and I know not a lot about it.
- Several things seemed ‘too good to be true’, for instance, the funnel-plot and the overall finding that acupuncture is the best of all therapeutic options.
- Looking at the references, I quickly confirmed my suspicion that most of the primary studies were in Chinese.
In retrospect, I am glad I did not tackle the task of criticising this paper; I would probably have made not nearly such a good job of it as PLOS ONE eventually did. But it was only after someone raised concerns that the paper was re-reviewed and all the defects outlined above came to light.
While some of my concerns listed above may have been trivial, my last point is the one that troubles me a lot. As it also related to dozens of Cochrane reviews which currently come out of China, it is worth our attention, I think. The problem, as I see it, is as follows:
- Chinese (acupuncture, TCM and perhaps also other) trials are almost invariably reporting positive findings, as we have discussed ad nauseam on this blog.
- Data fabrication seems to be rife in China.
- This means that there is good reason to be suspicious of such trials.
- Many of the reviews that currently flood the literature are based predominantly on primary studies published in Chinese.
- Unless one is able to read Chinese, there is no way of evaluating these papers.
- Therefore reviewers of journal submissions tend to rely on what the Chinese review authors write about the primary studies.
- As data fabrication seems to be rife in China, this trust might often not be justified.
- At the same time, Chinese researchers are VERY keen to publish in top Western journals (this is considered a great boost to their career).
- The consequence of all this is that reviews of this nature might be misleading, even if they are published in top journals.
I have been struggling with this problem for many years and have tried my best to alert people to it. However, it does not seem that my efforts had even the slightest success. The stream of such reviews has only increased and is now a true worry (at least for me). My suspicion – and I stress that it is merely that – is that, if one would rigorously re-evaluate these reviews, their majority would need to be retracted just as the above paper. That would mean that hundreds of papers would disappear because they are misleading, a thought that should give everyone interested in reliable evidence sleepless nights!
So, what can be done?
Personally, I now distrust all of these papers, but I admit, that is not a good, constructive solution. It would be better if Journal editors (including, of course, those at the Cochrane Collaboration) would allocate such submissions to reviewers who:
- are demonstrably able to conduct a CRITICAL analysis of the paper in question,
- can read Chinese,
- have no conflicts of interest.
In the case of an acupuncture review, this would narrow it down to perhaps just a handful of experts worldwide. This probably means that my suggestion is simply not feasible.
But what other choice do we have?
One could oblige the authors of all submissions to include full and authorised English translations of non-English articles. I think this might work, but it is, of course, tedious and expensive. In view of the size of the problem (I estimate that there must be around 1 000 reviews out there to which the problem applies), I do not see a better solution.
(I would truly be thankful, if someone had a better one and would tell us)
Psoriasis is one of those conditions that is
- not curable,
- irritating to the point where it reduces quality of life.
In other words, it is a disease for which virtually all alternative treatments on the planet are claimed to be effective. But which therapies do demonstrably alleviate the symptoms?
This review (published in JAMA Dermatology) compiled the evidence on the efficacy of the most studied complementary and alternative medicine (CAM) modalities for treatment of patients with plaque psoriasis and discusses those therapies with the most robust available evidence.
PubMed, Embase, and ClinicalTrials.gov searches (1950-2017) were used to identify all documented CAM psoriasis interventions in the literature. The criteria were further refined to focus on those treatments identified in the first step that had the highest level of evidence for plaque psoriasis with more than one randomized clinical trial (RCT) supporting their use. This excluded therapies lacking RCT data or showing consistent inefficacy.
A total of 457 articles were found, of which 107 articles were retrieved for closer examination. Of those articles, 54 were excluded because the CAM therapy did not have more than 1 RCT on the subject or showed consistent lack of efficacy. An additional 7 articles were found using references of the included studies, resulting in a total of 44 RCTs (17 double-blind, 13 single-blind, and 14 nonblind), 10 uncontrolled trials, 2 open-label nonrandomized controlled trials, 1 prospective controlled trial, and 3 meta-analyses.
Compared with placebo, application of topical indigo naturalis, studied in 5 RCTs with 215 participants, showed significant improvements in the treatment of psoriasis. Treatment with curcumin, examined in 3 RCTs (with a total of 118 participants), 1 nonrandomized controlled study, and 1 uncontrolled study, conferred statistically and clinically significant improvements in psoriasis plaques. Fish oil treatment was evaluated in 20 studies (12 RCTs, 1 open-label nonrandomized controlled trial, and 7 uncontrolled studies); most of the RCTs showed no significant improvement in psoriasis, whereas most of the uncontrolled studies showed benefit when fish oil was used daily. Meditation and guided imagery therapies were studied in 3 single-blind RCTs (with a total of 112 patients) and showed modest efficacy in treatment of psoriasis. One meta-analysis of 13 RCTs examined the association of acupuncture with improvement in psoriasis and showed significant improvement with acupuncture compared with placebo.
The authors concluded that CAM therapies with the most robust evidence of efficacy for treatment of psoriasis are indigo naturalis, curcumin, dietary modification, fish oil, meditation, and acupuncture. This review will aid practitioners in advising patients seeking unconventional approaches for treatment of psoriasis.
I am sorry to say so, but this review smells fishy! And not just because of the fish oil. But the fish oil data are a good case in point: the authors found 12 RCTs of fish oil. These details are provided by the review authors in relation to oral fish oil trials: Two double-blind RCTs (one of which evaluated EPA, 1.8g, and DHA, 1.2g, consumed daily for 12 weeks, and the other evaluated EPA, 3.6g, and DHA, 2.4g, consumed daily for 15 weeks) found evidence supporting the use of oral fish oil. One open-label RCT and 1 open-label non-randomized controlled trial also showed statistically significant benefit. Seven other RCTs found lack of efficacy for daily EPA (216mgto5.4g)or DHA (132mgto3.6g) treatment. The remainder of the data supporting efficacy of oral fish oil treatment were based on uncontrolled trials, of which 6 of the 7 studies found significant benefit of oral fish oil. This seems to support their conclusion. However, the authors also state that fish oil was not shown to be effective at several examined doses and duration. Confused? Yes, me too!
Even more confusing is their failure to mention a single trial of Mahonia aquifolium. A 2013 meta-analysis published in the British Journal of Dermatology included 5 RCTs of Mahonia aquifolium which, according to these authors, provided ‘limited support’ for its effectiveness. How could they miss that?
More importantly, how could the reviewers miss to conduct a proper evaluation of the quality of the studies they included in their review (even in their abstract, they twice speak of ‘robust evidence’ – but how can they without assessing its robustness? [quantity is not remotely the same as quality!!!]). Without a transparent evaluation of the rigour of the primary studies, any review is nearly worthless.
Take the 12 acupuncture trials, for instance, which the review authors included based not on an assessment of the studies but on a dodgy review published in a dodgy journal. Had they critically assessed the quality of the primary studies, they could have not stated that CAM therapies with the most robust evidence of efficacy for treatment of psoriasis …[include]… acupuncture. Instead they would have had to admit that these studies are too dubious for any firm conclusion. Had they even bothered to read them, they would have found that many are in Chinese (which would have meant they had to be excluded in their review [as many pseudo-systematic reviewers, the authors only considered English papers]).
There might be a lesson in all this – well, actually I can think of at least two:
- Systematic reviews might well be the ‘Rolls Royce’ of clinical evidence. But even a Rolls Royce needs to be assembled correctly, otherwise it is just a heap of useless material.
- Even top journals do occasionally publish poor-quality and thus misleading reviews.
Holistic ideas are booming, and they do not stop at dental medicine, where procedures and techniques that take an alleged ‘holistic’ approach are becoming more and more popular. Are these procedures and techniques effective, and do they offer a benefit over their conventional counterparts, or is it rather the providers of such procedures and techniques who benefit from a lack of knowledge and understanding in patients who seek out this so-called alternative dentistry? This paper will take a look at three topics—the concept of projections, material testing approaches, amalgam removal—that form the basis for many procedures and techniques in so-called alternative dentistry, to examine whether they offer a sound foundation for said procedures and techniques, or whether they are merely empty promises. Might they be nothing but marketing tricks?
The concept of projections suggests that conventional medicine does look closely enough at the human body, ignoring as of yet undiscovered energy lines and other mysterious linkages. Material testing approaches claim to detect harmful and allergenic components, the removal of which may be beneficial in case of systemic diseases, possibly even curing them. Beginning on July 1, 2018, the use of amalgam will be strongly restricted all throughout Europe. This easy-to-use material has received much attention for decades, as it contains a large proportion of mercury, which is known for its high neurotoxicity, and is, therefore, suspected of causing illness in the long term.
Normally, we think of projections as requiring a screen, onto which something then can be projected. Teeth, however, are also ideally suited as a dumping ground for the underlying causes of somatic and/or mental diseases, from where they can radiate out as so-called projections. Once these are identified as the true cause of disease, other potential causes such as age-related wear and tear, detrimental behaviors, or harmful eating habits can be readily ignored. This concept of projections may have particularly harmful and negative consequences in patients with tumors, as it may cause feelings of guilt, although in many cases no definite cause of tumor development can be discerned. Projected feelings of guilt, in turn, can be a negative influence on a person’s health.
The so-called “system of meridians” assigns relationship qualities to individual teeth, meaning that there are strict relationships of individual teeth to the body’s organs and individual entities. 
According to this system, an inflammation of the urinary bladder would be related to the number 1 teeth, the incisors. Rheumatism is linked to the number 8 teeth, the wisdom teeth. In between, there are the teeth of the ordinal numbers 2 to 7, distinguished by their locations on the left or right, in the upper or lower jaw, which offer a wealth of opportunities to assign a “guilty tooth” to clinically common physical complaints. However, this mysterious connection is postulated not only for teeth and major organs, but also for joints, vertebral levels, sensory organs, tonsils, and glands, with the relationships neatly organized in ten groups and subgroups. Multiplied by the number of teeth—eight per each of the four quadrants, 32 in total—these afford the “holistic dentist” 320 opportunities for projecting physical complaints ranging from asthma to zonulitis onto a tooth. Those who believe in this system of projections are not deterred by the fact that there is no scientific proof whatsoever for this odd thesis.
On the other hand, it is basic medical knowledge that pathogens may spread hematogenically and affect remote organs. Seeking adequate specialist counsel when dealing with rheumatic diseases, fevers of unclear etiology, or in conjunction with orthopedic joint surgeries, is, therefore, mandated by guidelines and an obvious standard in the practice of medicine. So-called alternative dentistry makes no particular mention of these general facts, but instead focuses on occult-seeming correlations in order to use a mysterious, almost conspiratorial idea of a disease to legitimize the often invasive treatment options it then recommends. Most patients will not realize that these interpretations often mistake synchronicity for causality. For example, most infections of the urinary bladder will resolve over time, regardless of whether any work was done on the upper incisors or not. However, if during the period of healing one of the incisors was treated by a dentist, it is easy enough to associate this treatment with the resolving bladder infection. From a psychological viewpoint, this constitutes a simple manipulation technique, applied to demonstrate the seemingly superior diagnostics of alternative dentistry: a simple, and easily recognized marketing strategy.
When asked what would happen to these doubtful projections in case of an autologous transplantation during which a tooth would move to another tooth’s original place in the jaw, three leading representatives of the so-called alternative dentistry answered in an evasive and even manipulative manner. 
There are reports of invasive therapies, conducted following dubious, often electromedical diagnostic procedures, that not only lead to high costs for the repair of the damage they caused, but also to a lasting mutilation of the patients’ jaws and dentitions. [3-6]
Another supposedly holistic school of thought that is similar to that of the system of meridians exists in some fields of dentistry regarding temporo-mandibular joint dysfunction (TMJD, TMD). These theories suggest that a disbalance in the interaction between jaw bones and masticatory muscles may be responsible for all kinds of diseases. 
According to the German self-appointed “TMJD Umbrella Organization” (CMD-Dachverband e. V.), TMJD is a “multifaceted disease.” The claim is that TMJD may not only cause back pain, vertigo, and tinnitus, but also sleep apnea, snoring, neck and shoulder pain, hip and knee pain, headaches, migraines, visual, mood swings, and even depression. However, there is no scientific evidence for any of these claims. [8,9]
Jens C. Türp of the University Center for Dental Medicine Basel’s Department of Oral Health & Medicine, Division Temporomandibular Disorders and Orofacial Pain, has called this standard diagnosis, offered by TMJD diagnosticians whenever a patient shows signs of nocturnal teeth grinding, “nonsense that makes your hair stand on end.”
“For a variety of general symptoms, it is claimed that they are caused by a TMJD: Tinnitus, ocular pressure, differences in the lengths of a person’s legs, back pain, hip pain, and knee pain, balance disorders, tingling in the fingers and many more. ‘A relationship [with TMJD] has never been proven for any of these symptoms’, says Türp. According to him, true TMJD causes problems with chewing and pain. Affected patients have difficulties opening their mouth wide or closing it fully. The “CMD-Arztsuche” (Find a TMJD Specialist) website recommends ‘a lasting correction of a person’s bite’ as treatment. This should be achieved with the help of ceramic inlays, dental crowns, and implants— all of which are expensive and unnecessary measures, in the opinion of Jens Türp. He treats his TMJD patients–almost always successfully, as he says–with occlusal splints, physiotherapy, and relaxation exercises.” (Translated from German )
In general, any patient should be advised, therefore, to seek a second opinion whenever confronted with a diagnosis requiring invasive treatments.
1. Madsen, H. Studie zur Kieferorthopädie in der Alternativmedizin: Darstellung der Grundlagen und kritische Bewertung. Doctoral dissertation, Poliklinik für Kieferorthopädie der Universität Würzburg. Würzburg 1994
2. Schulte von Drach, M.C. Wenn Zähne fremdgehen. Süddeutsche Zeitung May 15, 2012.
3. Staehle, H.J. Der Patientin wurde das Gebiss verstümmelt. Zahnärztliche Mitteilungen 2000.
4. Dowideit, A. Wenn nach der “Störfeld-Messung” alle Backenzähne fehlen. Welt June 3, 2017.
5. Bertelsen, H.-W. Die Attraktvität “ganzheitlicher” Zahnmedizin – Teil 1: Bohren ohne Reue. skeptiker 2012, 4.
6. Bertelsen, H.-W. Die Attraktivität “ganzheitlicher” Zahnmedizin – Teil 2: Bohren ohne Reue. skeptiker 2013, 4.
7. CMD Dachverband e. V. Craniomandibuläre Dysfunktion – Ursachen & Symptome. http://www.cmd-dachverband.de/fuer-patienten/ursachen-symptome/ (May 11, 2018),
8. Wolf, T. Die richtige Hilfe bei Kieferbeschwerden. Spiegel Online July 7, 2014, 2014.
9. Türp, J.C.; Schindler, H.-J.; Antes, G. Temporomandibular disorders: Evaluation of the usefulness of a self-test questionnaire. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen 2013, 107, 285-290.
10. Albrecht, B. Teure Tricks der Zahnärzte – so schützen Sie sich vor Überbehandlung. stern February 18, 2016.
I have often cautioned my readers about the ‘evidence’ supporting acupuncture (and other alternative therapies). Rightly so, I think. Here is yet another warning.
This systematic review assessed the clinical effectiveness of acupuncture in the treatment of postpartum depression (PPD). Nine trials involving 653 women were selected. A meta-analysis demonstrated that the acupuncture group had a significantly greater overall effective rate compared with the control group. Moreover, acupuncture significantly increased oestradiol levels compared with the control group. Regarding the HAMD and EPDS scores, no difference was found between the two groups. The Chinese authors concluded that acupuncture appears to be effective for postpartum depression with respect to certain outcomes. However, the evidence thus far is inconclusive. Further high-quality RCTs following standardised guidelines with a low risk of bias are needed to confirm the effectiveness of acupuncture for postpartum depression.
What a conclusion!
What a review!
What a journal!
Let’s start with the conclusion: if the authors feel that the evidence is ‘inconclusive’, why do they state that ‘acupuncture appears to be effective for postpartum depression‘. To me this does simply not make sense!
Such oddities are abundant in the review. The abstract does not mention the fact that all trials were from China (published in Chinese which means that people who cannot read Chinese are unable to check any of the reported findings), and their majority was of very poor quality – two good reasons to discard the lot without further ado and conclude that there is no reliable evidence at all.
The authors also tell us very little about the treatments used in the control groups. In the paper, they state that “the control group needed to have received a placebo or any type of herb, drug and psychological intervention”. But was acupuncture better than all or any of these treatments? I could not find sufficient data in the paper to answer this question.
Moreover, only three trials seem to have bothered to mention adverse effects. Thus the majority of the studies were in breach of research ethics. No mention is made of this in the discussion.
In the paper, the authors re-state that “this meta-analysis showed that the acupuncture group had a significantly greater overall effective rate compared with the control group. Moreover, acupuncture significantly increased oestradiol levels compared with the control group.” This is, I think, highly misleading (see above).
Finally, let’s have a quick look at the journal ‘Acupuncture in Medicine’ (AiM). Even though it is published by the BMJ group (the reason for this phenomenon can be found here: “AiM is owned by the British Medical Acupuncture Society and published by BMJ”; this means that all BMAS-members automatically receive the journal which thus is a resounding commercial success), it is little more than a cult-newsletter. The editorial board is full of acupuncture enthusiasts, and the journal hardly ever publishes anything that is remotely critical of the wonderous myths of acupuncture.
My conclusion considering all this is as follows: we ought to be very careful before accepting any ‘evidence’ that is currently being published about the benefits of acupuncture, even if it superficially looks ok. More often than not, it turns out to be profoundly misleading, utterly useless and potentially harmful pseudo-evidence.
Acupunct Med. 2018 Jun 15. pii: acupmed-2017-011530. doi: 10.1136/acupmed-2017-011530. [Epub ahead of print]
Li S, Zhong W, Peng W, Jiang G.
They say that minds are like parachutes – they function only when open. Having an open mind means being receptive to new and different ideas or the opinions of others.
I am regularly accused of lacking this quality. Most recently, an acupuncturist questioned whether acupuncture-sceptics, and I in particular, have an open mind. Subsequently, an interesting dialogue ensued:
edzard on Thursday 02 August 2018 at 12:46
“Perhaps the full conclusion should always be quoted…”
YES, IF NOT, IT’S CALLED ‘BEING ECONOMICAL WITH THE TRUTH’
I am sorry to say that I see no critical evaluation in the post you linked to.
” I’d say it comes across as ‘negative assessment’ rather than ‘critical assessment’.
have you noticed that criticism is often experienced as negative to the person(s) it is aimed at?
This exchange shows how easily I can be provoked to get stroppy and even impolite – I do apologise.
But it also made me wonder: how can anyone be sure to have an open mind?
And how can we decide that a person has a closed mind?
We probably all think we are open minded, but are we correct?
I am not at all sure that I know the answer. It obviously depends a lot on the subject. There are subjects where one hardly needs to keep an open mind and some where it might be advisable to have a closed mind:
- the notion that the earth is flat,
- flying carpets,
- chiropractic subluxation,
- the vital force,
No doubt, there will be people who even disagree with this short list.
Something that intrigues me – and I am here main ly talking about alternative medicine – is the fact that I often get praised by people who say, “I do appreciate your critical stance on therapy X, but on my treatment Y you are clearly biased and unfairly negative!” To me, it is an indication of a closed mind, if criticism is applauded as long as it does not tackle someone’s own belief system.
On the subject of homeopathy, Prof M Baum and I once published a paper entitled ‘Should we maintain an open mind about homeopathy?’ Its introduction explains the problem quite well, I think:
Once upon a time, doctors had little patience with the claims made for alternative medicines. In recent years the climate has changed dramatically. It is now politically correct to have an open mind about such matters; “the patient knows best” and “it worked for me” seem to be the new mantras. Although this may be a reasonable approach to some of the more plausible aspects of alternative medicine, such as herbal medicine or physical therapies that require manipulation, we believe it cannot apply across the board. Some of these alternatives are based on obsolete or metaphysical concepts of human biology and physiology that have to be described as absurd with proponents who will not subject their interventions to scientiﬁc scrutiny or if they do, and are found wanting, suggest that the mere fact of critical evaluation is sufﬁcient to chase the healing process away. These individuals have a conﬂict of interest more powerful than the requirement for scientiﬁc integrity and yet defend themselves by claiming that those wanting to carry out the trials are in the pocket of the pharmaceutical industry and are part of a conspiracy to deny their patients tried and tested palliatives….
END OF QUOTE
And this leads me to try to define 10 criteria indicative for an open mind.
- to be free of conflicts of interest,
- to resist the temptation of applying double standards,
- to have a track record of having changed one’s views in line with the evidence,
- to not cling to overt absurdities,
- to reject conspiracy theories,
- to be able to engage in a meaningful dialogue with people who have different views,
- to avoid fallacious thinking,
- to be willing to learn more on the subject in question.
I would be truly interested to hear, if you have further criteria, or indeed any other thoughts on the subject.
Needle acupuncture in small children is controversial, not least because the evidence that it works is negative or weak, and because small children are unable to consent to the treatment. Yet it is recommended by some acupuncturists for infant colic. This, of course, begs the questions:
- Does the best evidence tell us that acupuncture is effective for infant colic?
- Are acupuncturists who recommend acupuncture for this condition responsible and ethical?
This systematic review and a blinding-test validation based on individual patient data from randomised controlled trials was aimed to assess its efficacy for treating infantile colic. Primary end-points were crying time at mid-treatment, at the end of treatment and at a 1-month follow-up. A 30-min mean difference (MD) in crying time between acupuncture and control was predefined as a clinically important difference. Pearson’s chi-squared test and the James and Bang indices were used to test the success of blinding of the outcome assessors [parents].
The investigators included three randomised controlled trials with data from 307 participants. Only one of the included trials obtained a successful blinding of the outcome assessors in both the acupuncture and control groups. The MD in crying time between acupuncture intervention and no acupuncture control was -24.9 min at mid-treatment, -11.4 min at the end of treatment and -11.8 min at the 4-week follow-up. The heterogeneity was negligible in all analyses. The statistically significant result at mid-treatment was lost when excluding the apparently unblinded study in a sensitivity analysis: MD -13.8 min. The registration of crying during treatment suggested more crying during acupuncture.
The authors concluded that percutaneous needle acupuncture treatments should not be recommended for infantile colic on a general basis.
The authors also provide this further comment: “Our blinding test validated IPD meta-analysis of minimal acupuncture treatments of infantile colic did not show clinically relevant effects in pain reduction as estimated by differences in crying time between needle acupuncture intervention and no acupuncture control. Analyses indicated that acupuncture treatment induced crying in many of the children. Caution should therefore be exercised in recommending potentially painful treatments with uncertain efficacy in infants. The studies are few, the analysis is made on small samples of individuals, and conclusions should be considered in this context. With this limitation in mind, our findings do not support the idea that percutaneous needle acupuncture should be recommended for treatment of infantile colic on a general basis.”
So, returning to the two questions that I listed above – what are the answers?
I think they must be:
Most people probably think of acupuncture as being used mainly as a therapy for pain control. But acupuncture is currently being promoted (and has traditionally been used) for all sorts of conditions. One of them is stroke. It is said to speed up recovery and even improve survival rates after such an event. There are plenty of studies on this subject, but their results are far from uniform. What is needed in this situation, is a rigorous summary of the evidence.
The authors of this Cochrane review wanted to assess whether acupuncture could reduce the proportion of people suffering death or dependency after acute ischaemic or haemorrhagic stroke. They included all randomized clinical trials (RCTs) of acupuncture started within 30 days after stroke onset. Acupuncture had to be compared with placebo or sham acupuncture or open control (no placebo) in people with acute ischaemic or haemorrhagic stroke, or both. Comparisons were made versus (1) all controls (open control or sham acupuncture), and (2) sham acupuncture controls.
The investigators included 33 RCTs with 3946 participants. Outcome data were available for up to 22 trials (2865 participants) that compared acupuncture with any control (open control or sham acupuncture) but for only 6 trials (668 participants) comparing acupuncture with sham acupuncture.
When compared with any control (11 trials with 1582 participants), findings of lower odds of death or dependency at the end of follow-up and over the long term (≥ three months) in the acupuncture group were uncertain and were not confirmed by trials comparing acupuncture with sham acupuncture. In trials comparing acupuncture with any control, findings that acupuncture was associated with increases in the global neurological deficit score and in the motor function score were uncertain. These findings were not confirmed in trials comparing acupuncture with sham acupuncture.Trials comparing acupuncture with any control showed little or no difference in death or institutional care or death at the end of follow-up.The incidence of adverse events (eg, pain, dizziness, faint) in the acupuncture arms of open and sham control trials was 6.2% (64/1037 participants), and 1.4% of these patients (14/1037 participants) discontinued acupuncture. When acupuncture was compared with sham acupuncture, findings for adverse events were uncertain.
The authors concluded that this updated review indicates that apparently improved outcomes with acupuncture in acute stroke are confounded by the risk of bias related to use of open controls. Adverse events related to acupuncture were reported to be minor and usually did not result in stopping treatment. Future studies are needed to confirm or refute any effects of acupuncture in acute stroke. Trials should clearly report the method of randomization, concealment of allocation, and whether blinding of participants, personnel, and outcome assessors was achieved, while paying close attention to the effects of acupuncture on long-term functional outcomes.
This Cochrane review seems to be thorough, but it is badly written (Cochrane reviewers: please don’t let this become the norm!). It contains some interesting facts. The majority of the studies came from China. This review confirmed the often very poor methodological quality of acupuncture trials which I have frequently mentioned before.
In particular, the RCTs originating from China were amongst those that most overtly lacked rigor, also a fact that has been discussed regularly on this blog.
For me, by far the most important finding of this review is that studies which at least partly control for placebo effects fail to show positive results. Depending on where you stand in the never-ending debate about acupuncture, this could lead to two dramatically different conclusions:
- If you are a believer in or earn your living from acupuncture, you might say that these results suggest that the trials were in some way insufficient and therefore they produced false-negative results.
- If you are a more reasonable observer, you might feel that these results show that acupuncture (for acute stroke) is a placebo therapy.
Regardless to which camp you belong, one thing seems to be certain: acupuncture for stroke (and other indications) is not supported by sound evidence. And that means, I think, that it is not responsible to use it in routine care.
Can I tempt you to run a little (hopefully instructive) thought-experiment with you? It is quite simple: I will tell you about the design of a clinical trial, and you will tell me what the likely outcome of this study would be.
Are you game?
Here we go:
Imagine we conduct a trial of acupuncture for persistent pain (any type of pain really). We want to find out whether acupuncture is more than a placebo when it comes to pain-control. Of course, we want our trial to look as rigorous as possible. So, we design it as a randomised, sham-controlled, partially-blinded study. To be really ‘cutting edge’, our study will not have two but three parallel groups:
1. Standard needle acupuncture administered according to a protocol recommended by a team of expert acupuncturists.
2. Minimally invasive sham-acupuncture employing shallow needle insertion using short needles at non-acupuncture points. Patients in groups 1 and 2 are blinded, i. e. they are not supposed to know whether they receive the sham or real acupuncture.
3. No treatment at all.
We apply the treatments for a sufficiently long time, say 12 weeks. Before we start, after 6 and 12 weeks, we measure our patients’ pain with a validated method. We use sound statistical methods to compare the outcomes between the three groups.
WHAT DO YOU THINK THE RESULT WOULD BE?
You are not sure?
Well, let me give you some hints:
Group 3 is not going to do very well; not only do they receive no therapy at all, but they are also disappointed to have ended up in this group as they joined the study in the hope to get acupuncture. Therefore, they will (claim to) feel a lot of pain.
Group 2 will be pleased to receive some treatment. However, during the course of the 6 weeks, they will get more and more suspicious. As they were told during the process of obtaining informed consent that the trial entails treating some patients with a sham/placebo, they are bound to ask themselves whether they ended up in this group. They will see the short needles and the shallow needling, and a percentage of patients from this group will doubtlessly suspect that they are getting the sham treatment. The doubters will not show a powerful placebo response. Therefore, the average pain scores in this group will decrease – but only a little.
Group 1 will also be pleased to receive some treatment. As the therapists cannot be blinded, they will do their best to meet the high expectations of their patients. Consequently, they will benefit fully from the placebo effect of the intervention and the pain score of this group will decrease significantly.
So, now we can surely predict the most likely result of this trial without even conducting it. Assuming that acupuncture is a placebo-therapy, as many people do, we now see that group 3 will suffer the most pain. In comparison, groups 1 and 2 will show better outcomes.
Of course, the main question is, how do groups 1 and 2 compare to each other? After all, we designed our sham-controlled trial in order to answer exactly this issue: is acupuncture more than a placebo? As pointed out above, some patients in group 2 would have become suspicious and therefore would not have experienced the full placebo-response. This means that, provided the sample sizes are sufficiently large, there should be a significant difference between these two groups favouring real acupuncture over sham. In other words, our trial will conclude that acupuncture is better than placebo, even if acupuncture is a placebo.
THANK YOU FOR DOING THIS THOUGHT EXPERIMENT WITH ME.
Now I can tell you that it has a very real basis. The leading medical journal, JAMA, just published such a study and, to make matters worse, the trial was even sponsored by one of the most prestigious funding agencies: the NIH.
Here is the abstract:
Musculoskeletal symptoms are the most common adverse effects of aromatase inhibitors and often result in therapy discontinuation. Small studies suggest that acupuncture may decrease aromatase inhibitor-related joint symptoms.
To determine the effect of acupuncture in reducing aromatase inhibitor-related joint pain.
Design, Setting, and Patients:
Randomized clinical trial conducted at 11 academic centers and clinical sites in the United States from March 2012 to February 2017 (final date of follow-up, September 5, 2017). Eligible patients were postmenopausal women with early-stage breast cancer who were taking an aromatase inhibitor and scored at least 3 on the Brief Pain Inventory Worst Pain (BPI-WP) item (score range, 0-10; higher scores indicate greater pain).
Patients were randomized 2:1:1 to the true acupuncture (n = 110), sham acupuncture (n = 59), or waitlist control (n = 57) group. True acupuncture and sham acupuncture protocols consisted of 12 acupuncture sessions over 6 weeks (2 sessions per week), followed by 1 session per week for 6 weeks. The waitlist control group did not receive any intervention. All participants were offered 10 acupuncture sessions to be used between weeks 24 and 52.
Main Outcomes and Measures:
The primary end point was the 6-week BPI-WP score. Mean 6-week BPI-WP scores were compared by study group using linear regression, adjusted for baseline pain and stratification factors (clinically meaningful difference specified as 2 points).
Among 226 randomized patients (mean [SD] age, 60.7 [8.6] years; 88% white; mean [SD] baseline BPI-WP score, 6.6 [1.5]), 206 (91.1%) completed the trial. From baseline to 6 weeks, the mean observed BPI-WP score decreased by 2.05 points (reduced pain) in the true acupuncture group, by 1.07 points in the sham acupuncture group, and by 0.99 points in the waitlist control group. The adjusted difference for true acupuncture vs sham acupuncture was 0.92 points (95% CI, 0.20-1.65; P = .01) and for true acupuncture vs waitlist control was 0.96 points (95% CI, 0.24-1.67; P = .01). Patients in the true acupuncture group experienced more grade 1 bruising compared with patients in the sham acupuncture group (47% vs 25%; P = .01).
Conclusions and Relevance:
Among postmenopausal women with early-stage breast cancer and aromatase inhibitor-related arthralgias, true acupuncture compared with sham acupuncture or with waitlist control resulted in a statistically significant reduction in joint pain at 6 weeks, although the observed improvement was of uncertain clinical importance.
Do you see how easy it is to deceive (almost) everyone with a trial that looks rigorous to (almost) everyone?
My lesson from all this is as follows: whether consciously or unconsciously, SCAM-researchers often build into their trials more or less well-hidden little loopholes that ensure they generate a positive outcome. Thus even a placebo can appear to be effective. They are true masters of producing false-positive findings which later become part of a meta-analysis which is, of course, equally false-positive. It is a great shame, in my view, that even top journals (in the above case JAMA) and prestigious funders (in the above case the NIH) cannot (or want not to?) see behind this type of trickery.
Alternative practitioners practise highly diverse therapies. They seem to have nothing in common – except perhaps that ALL of them are allegedly stimulating our self-healing powers (and except that most proponents are latently or openly against vaccinations). And it is through these self-healing powers that the treatments in question cure anything and become a true panacea. When questioned what these incredible powers really are, most practitioners would (somewhat vaguely) name the immune system as the responsible mechanism. With this post, I intend to provide a short summary of the evidence on this issue:
Acupuncture: no good evidence to show stimulation of self-healing powers.
Aromatherapy: no good evidence to show stimulation of self-healing powers.
Bioresonance: no good evidence to show stimulation of self-healing powers.
Chiropractic: no good evidence to show stimulation of self-healing powers.
Detox: no good evidence to show stimulation of self-healing powers.
Energy healing: no good evidence to show stimulation of self-healing powers.
Feldenkrais: no good evidence to show stimulation of self-healing powers.
Gua sha: no good evidence to show stimulation of self-healing powers.
Herbal medicine: no good evidence to show stimulation of self-healing powers.
Homeopathy: no good evidence to show stimulation of self-healing powers.
Macrobiotics: no good evidence to show stimulation of self-healing powers.
Naturopathy: no good evidence to show stimulation of self-healing powers.
Osteopathy: no good evidence to show stimulation of self-healing powers.
Power bands: no good evidence to show stimulation of self-healing powers.
Reiki: no good evidence to show stimulation of self-healing powers.
Reflexology: no good evidence to show stimulation of self-healing powers.
Shiatsu: no good evidence to show stimulation of self-healing powers.
Tai chi: no good evidence to show stimulation of self-healing powers.
TCM: no good evidence to show stimulation of self-healing powers.
Vibrational therapy: no good evidence to show stimulation of self-healing powers.
Vaccinations: very good evidence to show stimulation of self-healing powers.
Why do most alternative practitioners show such dogged determination not to change their view of the efficacy of their therapy, even if good evidence shows that it is a placebo? This is the question that I have been pondering for some time. I have seen many doctors change their mind about this or that treatment in the light of new evidence. In fact, I have not seen one who has not done so at some stage. Yet I have never seen an alternative therapist change his/her mind about his/her alternative therapy. Why is that?
You might say that the answers are obvious:
- because they have heavily invested in their therapy, both emotionally and financially;
- because their therapy has ‘stood the test of time’;
- because they believe what they were taught;
- because they are deluded, not very bright, etc.;
- because they need to earn a living.
All of these reasons may apply. But do they really tell the whole story? While contemplating about this question, I thought of something that had previously not been entirely clear to me: they simply KNOW that the evidence MUST be wrong.
Let me try to explain.
Consider an acupuncturist (I could have chosen almost any other type of alternative practitioner) who has many years of experience. He has grown to be a well-respected expert in the world of acupuncture. He sits on various committees and has advised important institutions. He knows the literature and has treated thousands of patients.
This experience has taught him one thing for sure: his patients do benefit from his treatment. He has seen it happening too many times; it cannot be a coincidence. Acupuncture works, no question about it.
And this is also what the studies tell him. Even the most sceptical scientist cannot deny the fact that patients do get better after acupuncture. So, what is the problem?
The problem is that sceptics say that this is due to a placebo effect, and many studies seem to confirm this to be true. Yet, our acupuncturist completely dismisses the placebo explanation.
- Because he has heavily invested in their therapy? Perhaps.
- Because acupuncture has ‘stood the test of time’? Perhaps.
- Because he believes what he has been taught? Perhaps.
- Because he is deluded, not very bright, etc.? Perhaps.
- Because he needs to earn a living? Perhaps.
But there is something else.
He has only ever treated his patients with acupuncture. He has therefore no experience of real medicine, or other therapeutic options. He has no perspective. Therefore, he does not know that patients often get better, even if they receive an ineffective treatment, even if they receive no treatment, and even if they receive a harmful treatment. Every improvement he notes in his patients, he relates to his acupuncture. Our acupuncturist never had the opportunity to learn to doubt cause and effect in his clinical routine. He never had to question the benefits of acupuncture. He never had to select from a pool of therapies the optimal one, because he only ever used acupuncture.
It is this lack of experience that never led him to think critically about acupuncture. He is in a similar situation as physicians were 200 years ago; they only (mainly) had blood-letting, and because some patients improved with it, they had no reason to doubt it. He only ever saw his successes (not that all his patients improved, but those who did not, did not return). He simply KNOWS that acupuncture works, because his own, very limited experience never forced him to consider anything else. And because he KNOWS, the evidence that does not agree with his knowledge MUST be wrong.
I am of course exaggerating and simplifying in order to make a point. And please don’t get me wrong.
I am not saying that doctors cannot be stubborn. And I am not saying that all alternative practitioners have such limited experience and are unable to change their mind in the light of new evidence. However, I am trying to say that many alternative practitioners have a limited perspective and therefore find it impossible to be critical about their own practice.
If I am right, there would be an easy (and entirely alternative) cure to remedy this situation. We should sent our acupuncturist to a homeopath (or any other alternative practitioner whose practice he assumes to be entirely bogus) and ask him to watch what kind of therapeutic success the homeopath is generating. The acupuncturist would soon see that it is very similar to his own. He would then have the choice to agree that highly diluted homeopathic remedies are effective in curing illness, or that the homeopath relies on the same phenomenon as his own practice: placebo.
Sadly, this is not going to happen, is it?