This was essentially the question raised in a correspondence with a sceptic friend. His suspicion was that statistical methods might produce false-positive overall findings, if the research is done by enthusiasts of the so-called alternative medicine (SCAM) in question (or other areas of inquiry which I will omit because they are outside my area of expertise). Consciously or inadvertently, such researchers might introduce a pro-SCAM bias into their work. As the research is done mostly by such enthusiasts; the totality of the evidence would turn out to be heavily skewed in favour of the SCAM under investigation. The end-result would then be a false-positive overall impression about the SCAM which is less based on reality than on the wishful thinking of the investigators.
How can one deal with this problem?
How to minimise the risk of being overwhelmed by false-positive research?
Today, we have several mechanisms and initiatives that are at least partly aimed at achieving just this. For instance, there are guidelines on how to conduct the primary research so that bias is minimised. The CONSORT statements are an example. As many studies pre-date CONSORT, we need a different approach for reviews of clinical trials. The PRISMA guideline or the COCHRANE handbook are attempts to make sure systematic reviews are transparent and rigorous. These methods can work quite well in finding the truth, but one needs to be aware, of course, that some researchers do their very best to obscure it. I have also tried to go one step further and shown that the direction of the conclusion correlates with the rigour of the study (btw: this was the paper that prompted Prof Hahn’s criticism and slander of my work and person).
So, problem sorted?
The trouble is that over-enthusiastic researchers may not always adhere to these guidelines, they may pretend to adhere but cut corners, or they may be dishonest and cheat. And what makes this even more tricky is the possibility that they do all this inadvertently; their enthusiasm could get the better of them, and they are doing research not to TEST WHETHER a treatment works but to PROVE THAT it works.
In the realm of SCAM we have a lot of this – trust me, I have seen it often with my own eyes, regrettably sometimes even within my own team of co-workers. The reason for this is that SCAM is loaded with emotion and quasi-religious beliefs; and these provide a much stronger conflict of interest than money could ever do, in my experience.
And how might we tackle this thorny issue?
After thinking long and hard about it, I came up in 2012 with my TRUSTWORTHYNESS INDEX:
If we calculated the percentage of a researcher’s papers arriving at positive conclusions and divided this by the percentage of his papers drawing negative conclusions, we might have a useful measure. A realistic example might be the case of a clinical researcher who has published a total of 100 original articles. If 50% had positive and 50% negative conclusions about the efficacy of the therapy tested, his TI would be 1.
Depending on what area of clinical medicine this person is working in, 1 might be a figure that is just about acceptable in terms of the trustworthiness of the author. If the TI goes beyond 1, we might get concerned; if it reaches 4 or more, we should get worried.
An example would be a researcher who has published 100 papers of which 80 are positive and 20 arrive at negative conclusions. His TI would consequently amount to 4. Most of us equipped with a healthy scepticism would consider this figure highly suspect.
Of course, this is all a bit simplistic, and, like all other citation metrics, my TI provides us not with any level of proof; it merely is a vague indicator that something might be amiss. And, as stressed already, the cut-off point for any scientist’s TI very much depends on the area of clinical research we are dealing with. The lower the plausibility and the higher the uncertainty associated with the efficacy of the experimental treatments, the lower the point where the TI might suggest something to be fishy.
Based on this concept, I later created the ALTERNATIVE MEDICINE HALL OF FAME. This is a list of researchers who manage to go through life researching their particular SCAM without ever publishing a negative conclusion about it. In terms of TI, these people have astronomically high values. The current list is not yet long, but it is growing:
John Weeks (editor of JCAM)
Deepak Chopra (US entrepreneur)
Cheryl Hawk (US chiropractor)
David Peters (osteopathy, homeopathy, UK)
Nicola Robinson (TCM, UK)
Peter Fisher (homeopathy, UK)
Simon Mills (herbal medicine, UK)
Gustav Dobos (various, Germany)
Claudia Witt (homeopathy, Germany and Switzerland)
George Lewith (acupuncture, UK)
John Licciardone (osteopathy, US)
The logical consequence of a high TI would be that researchers of that nature are banned from obtaining research funds and publishing papers, because their contribution is merely to confuse us and make science less reliable.
I am sure there are other ways of addressing the problem of being mislead by false-positive research. If you can think of one, I’d be pleased to hear about it.
To update my article of 2008, I have searched the Cochrane Library for all Cochrane reviews specifically targeted at acupuncture or related interventions such as acupressure, electro-acupuncture and moxibustion (on 1/6/2020). More general reviews which included some evidence on acupuncture but were not specifically on this topic (e.g. complementary therapies for enuresis) were omitted.
It turned out that almost all the 32 reviews available in 2008 had been updated (some several times), a few had been abandoned and many new reviews have been added. In fact, the 32 reviews of 2008 have today grown into 54.
Here are the conclusions of and links to these papers:
- BELL’S PALSY The quality of the included trials was inadequate to allow any conclusion about the efficacy of acupuncture. More research with high quality trials is needed.
- PRIMARY DYSMENORRHOEA There is insufficient evidence to demonstrate whether or not acupuncture or acupressure are effective in treating primary dysmenorrhoea, and for most comparisons no data were available on adverse events. The quality of the evidence was low or very low for all comparisons. The main limitations were risk of bias, poor reporting, inconsistency and risk of publication bias.
- PREVENTION OF EPISODIC MIGRAINE The available evidence suggests that adding acupuncture to symptomatic treatment of attacks reduces the frequency of headaches. Contrary to the previous findings, the updated evidence also suggests that there is an effect over sham, but this effect is small. The available trials also suggest that acupuncture may be at least similarly effective as treatment with prophylactic drugs. Acupuncture can be considered a treatment option for patients willing to undergo this treatment. As for other migraine treatments, long‐term studies, more than one year in duration, are lacking.
- RHEUMATOID ARTHRITIS Although the results of the study on electroacupuncture show that electroacupuncture may be beneficial to reduce symptomatic knee pain in patients with RA 24 hours and 4 months post treatment, the reviewers concluded that the poor quality of the trial, including the small sample size preclude its recommendation. The reviewers further conclude that acupuncture has no effect on ESR, CRP, pain, patient’s global assessment, number of swollen joints, number of tender joints, general health, disease activity and reduction of analgesics. These conclusions are limited by methodological considerations such as the type of acupuncture (acupuncture vs electroacupuncture), the site of intervention, the low number of clinical trials and the small sample size of the included studies.
- PREVENTION OF TENSION TYPE HEADACHE: The available results suggest that acupuncture is effective for treating frequent episodic or chronic tension‐type headaches, but further trials ‐ particularly comparing acupuncture with other treatment options ‐ are needed.
- SHOULDER PAIN Due to a small number of clinical and methodologically diverse trials, little can be concluded from this review. There is little evidence to support or refute the use of acupuncture for shoulder pain although there may be short‐term benefit with respect to pain and function. There is a need for further well designed clinical trials.
- EPILEPSY Available RCTs are small, heterogeneous and have high risk of bias. The current evidence does not support acupuncture for treating epilepsy.
- CHRONIC ASTHMA There is not enough evidence to make recommendations about the value of acupuncture in asthma treatment. Further research needs to consider the complexities and different types of acupuncture.
- POLYCYSTIC OVARIAN SYNDROME For true acupuncture versus sham acupuncture we cannot exclude clinically relevant differences in live birth rate, multiple pregnancy rate, ovulation rate, clinical pregnancy rate or miscarriage. Number of intermenstrual days may improve in participants receiving true acupuncture compared to sham acupuncture. True acupuncture probably worsens adverse events compared to sham acupuncture. No studies reported data on live birth rate and multiple pregnancy rate for the other comparisons: physical exercise or no intervention, relaxation and clomiphene. Studies including Diane‐35 did not measure fertility outcomes as the women in these trials did not seek fertility.We are uncertain whether acupuncture improves ovulation rate (measured by ultrasound three months post treatment) compared to relaxation or Diane‐35. The other comparisons did not report on this outcome.Adverse events were recorded in the acupuncture group for the comparisons physical exercise or no intervention, clomiphene and Diane‐35. These included dizziness, nausea and subcutaneous haematoma. Evidence was very low quality with very wide CIs and very low event rates.There are only a limited number of RCTs in this area, limiting our ability to determine effectiveness of acupuncture for PCOS.
- CHRONIC HEPATITIS B The clinical effects of acupuncture for chronic hepatitis B remain unknown. The included trials lacked data on all‐cause mortality, health‐related quality of life, serious adverse events, hepatitis‐B related mortality, and hepatitis‐B related morbidity. The vast number of excluded trials lacked clear descriptions of their design and conduct. Whether acupuncture influences adverse events considered not to be serious is uncertain. It remains unclear if acupuncture affects HBeAg, and if it is associated with reduction in detectable HBV DNA. Based on available data from only one or two small trials on adverse events considered not to be serious and on the surrogate outcomes HBeAg and HBV DNA, the certainty of evidence is very low. In view of the wide usage of acupuncture, any conclusion that one might try to draw in the future should be based on data on patient and clinically relevant outcomes, assessed in large, high‐quality randomised sham‐controlled trials with homogeneous groups of participants and transparent funding.
- ENDOMETRIOSIS The evidence to support the effectiveness of acupuncture for pain in endometriosis is limited, based on the results of only a single study that was included in this review. This review highlights the necessity for developing future studies that are well‐designed, double‐blinded, randomised controlled trials that assess various types of acupuncture in comparison to conventional therapies.
- VASCULAR DEMENTIA The effectiveness of acupuncture for vascular dementia is uncertain. More evidence is required to show that vascular dementia can be treated effectively by acupuncture. There are no RCTs and high quality trials are few. Randomized double‐blind placebo‐controlled trials are urgently needed.
- FUNCTIONAL DYSPEPSIA It remains unknown whether manual acupuncture or electroacupuncture is more effective or safer than other treatments for patients with FD.
- SMOKING CESSATION Although pooled estimates suggest possible short‐term effects there is no consistent, bias‐free evidence that acupuncture, acupressure, or laser therapy have a sustained benefit on smoking cessation for six months or more. However, lack of evidence and methodological problems mean that no firm conclusions can be drawn. Electrostimulation is not effective for smoking cessation. Well‐designed research into acupuncture, acupressure and laser stimulation is justified since these are popular interventions and safe when correctly applied, though these interventions alone are likely to be less effective than evidence‐based interventions.
- RESTLESS LEG SYNDROME There is insufficient evidence to determine whether acupuncture is an efficacious and safe treatment for RLS. Further well‐designed, large‐scale clinical trials are needed.
- COCAINE DEPENDENCE There is currently no evidence that auricular acupuncture is effective for the treatment of cocaine dependence. The evidence is not of high quality and is inconclusive. Further randomised trials of auricular acupuncture may be justified.
- LABOUR PAIN Acupuncture in comparison to sham acupuncture may increase satisfaction with pain management and reduce use of pharmacological analgesia. Acupressure in comparison to a combined control and usual care may reduce pain intensity. However, for other comparisons of acupuncture and acupressure, we are uncertain about the effects on pain intensity and satisfaction with pain relief due to very low‐certainty evidence. Acupuncture may have little to no effect on the rates of caesarean or assisted vaginal birth. Acupressure probably reduces the need for caesarean section in comparison to a sham control. There is a need for further high‐quality research that include sham controls and comparisons to usual care and report on the outcomes of sense of control in labour, satisfaction with the childbirth experience or satisfaction with pain relief.
- ISCAEMIC ENCEPHALOPATHY The rationale for acupuncture in neonates with HIE is unclear and the evidence from randomized controlled trial is lacking. Therefore, we do not recommend acupuncture for the treatment of HIE in neonates. High quality randomized controlled trials on acupuncture for HIE in neonates are needed.
- LOW BACK PAIN The data do not allow firm conclusions about the effectiveness of acupuncture for acute low‐back pain. For chronic low‐back pain, acupuncture is more effective for pain relief and functional improvement than no treatment or sham treatment immediately after treatment and in the short‐term only. Acupuncture is not more effective than other conventional and “alternative” treatments. The data suggest that acupuncture and dry‐needling may be useful adjuncts to other therapies for chronic low‐back pain. Because most of the studies were of lower methodological quality, there certainly is a further need for higher quality trials in this area.
- AUTISM Current evidence does not support the use of acupuncture for treatment of ASD. There is no conclusive evidence that acupuncture is effective for treatment of ASD in children and no RCTs have been carried out with adults. Further high quality trials of larger size and longer follow‐up are needed.
- CARPAL TUNNEL SYNDROME Acupuncture and laser acupuncture may have little or no effect in the short term on symptoms of CTS in comparison with placebo or sham acupuncture. It is uncertain whether acupuncture and related interventions are more or less effective in relieving symptoms of CTS than corticosteroid nerve blocks, oral corticosteroids, vitamin B12, ibuprofen, splints, or when added to NSAIDs plus vitamins, as the certainty of any conclusions from the evidence is low or very low and most evidence is short term. The included studies covered diverse interventions, had diverse designs, limited ethnic diversity, and clinical heterogeneity. High‐quality randomised controlled trials (RCTs) are necessary to rigorously assess the effects of acupuncture and related interventions upon symptoms of CTS. Based on moderate to very‐low certainty evidence, acupuncture was associated with no serious adverse events, or reported discomfort, pain, local paraesthesia and temporary skin bruises, but not all studies provided adverse event data.
- ADHA A comprehensive search showed that there is no evidence base of randomised or quasi‐randomised controlled trials to support the use of acupuncture as a treatment for ADHD in children and adolescents. Due to the lack of trials, we cannot reach any conclusions about the efficacy and safety of acupuncture for ADHD in children and adolescents. This review highlights the need for further research in this area in the form of high quality, large scale, randomised controlled trials.
- FIBROMYALGIA There is low to moderate‐level evidence that compared with no treatment and standard therapy, acupuncture improves pain and stiffness in people with fibromyalgia. There is moderate‐level evidence that the effect of acupuncture does not differ from sham acupuncture in reducing pain or fatigue, or improving sleep or global well‐being. EA is probably better than MA for pain and stiffness reduction and improvement of global well‐being, sleep and fatigue. The effect lasts up to one month, but is not maintained at six months follow‐up. MA probably does not improve pain or physical functioning. Acupuncture appears safe. People with fibromyalgia may consider using EA alone or with exercise and medication. The small sample size, scarcity of studies for each comparison, lack of an ideal sham acupuncture weaken the level of evidence and its clinical implications. Larger studies are warranted.
- GLAUCOMA At this time, it is impossible to draw reliable conclusions from available data to support the use of acupuncture for treatment of patients with glaucoma. Because of ethical considerations, RCTs comparing acupuncture alone with standard glaucoma treatment or placebo are unlikely to be justified in countries where the standard of care has already been established.
- UTERINE FIBROIDS The effectiveness of acupuncture for the management of uterine fibroids remains uncertain. More evidence is required to establish the efficacy and safety of acupuncture for uterine fibroids. There is a continued need for well designed RCTs with long term follow up.
- HIP OSTEOARTHRITIS Acupuncture probably has little or no effect in reducing pain or improving function relative to sham acupuncture in people with hip osteoarthritis. Due to the small sample size in the studies, the confidence intervals include both the possibility of moderate benefits and the possibility of no effect of acupuncture. One unblinded trial found that acupuncture as an addition to routine primary physician care was associated with benefits on pain and function. However, these reported benefits are likely due at least partially to RCT participants’ greater expectations of benefit from acupuncture. Possible side effects associated with acupuncture treatment were minor.
- HYPERTENSION At present, there is no evidence for the sustained BP lowering effect of acupuncture that is required for the management of chronically elevated BP. The short‐term effects of acupuncture are uncertain due to the very low quality of evidence. The larger effect shown in non‐sham acupuncture controlled trials most likely reflects bias and is not a true effect. Future RCTs must use sham acupuncture controls and assess whether there is a BP lowering effect of acupuncture that lasts at least seven days.
- GASTROPARESISThere is very low‐certainty evidence for a short‐term benefit with acupuncture alone or acupuncture combined with gastrokinetic drugs compared with the drug alone, in terms of the proportion of people who experienced improvement in diabetic gastroparesis. There is evidence of publication bias and a positive bias of small study effects. The reported benefits should be interpreted with great caution because of the unclear overall risk of bias, unvalidated measurements of change in subjective symptoms, publication bias and small study reporting bias, and lack of data on long‐term outcomes; the effects reported in this review may therefore differ significantly from the true effect. One sham‐controlled trial provided low‐certainty evidence of no difference between real and sham acupuncture in terms of short‐term symptom improvement in diabetic gastroparesis, when measured by a validated scale. No studies reported changes in quality of life or the use of medication.Due to the absence of data, no conclusion can be made regarding effects of acupuncture on gastroparesis of other aetiologies. Reports of harm have remained largely incomplete, precluding assessments of the safety of acupuncture in this population. Future research should focus on reducing the sources of bias in the trial design as well as transparent reporting. Harms of interventions should be explicitly reported.
- ACUTE STROKE This updated review indicates that apparently improved outcomes with acupuncture in acute stroke are confounded by the risk of bias related to use of open controls. Adverse events related to acupuncture were reported to be minor and usually did not result in stopping treatment. Future studies are needed to confirm or refute any effects of acupuncture in acute stroke. Trials should clearly report the method of randomization, concealment of allocation, and whether blinding of participants, personnel, and outcome assessors was achieved, while paying close attention to the effects of acupuncture on long‐term functional outcomes.
- INSOMNIA Due to poor methodological quality, high levels of heterogeneity and publication bias, the current evidence is not sufficiently rigorous to support or refute acupuncture for treating insomnia. Larger high‐quality clinical trials are required.
- SCHIZOPHRENIA Limited evidence suggests that acupuncture may have some antipsychotic effects as measured on global and mental state with few adverse effects. Better designed large studies are needed to fully and fairly test the effects of acupuncture for people with schizophrenia.
- ACUTE HORDEOLUM Low‐certainty evidence suggests that acupuncture with or without conventional treatments may provide short‐term benefits for treating acute hordeolum when compared with conventional treatments alone. The certainty of the evidence was low to very low mainly due to small sample sizes, inadequate allocation concealment, lack of masking of the outcome assessors, inadequate or unclear randomization method, and a high or unreported number of dropouts. All RCTs were conducted in China, which may limit their generalizability to non‐Chinese populations.Because no RCTs included a valid sham acupuncture control, we cannot rule out a potential expectation/placebo effect associated with acupuncture. As resolution is based on clinical observation, the outcome could be influenced by the observer’s knowledge of the assigned treatment. Adverse effects of acupuncture were reported sparsely in the included RCTs, and, when reported, were rare. RCTs with better methodology, longer follow‐up, and which are conducted among other populations are warranted to provide more general evidence regarding the benefit of acupuncture to treat acute hordeolum.
- STROKE REHABILITATION From the available evidence, acupuncture may have beneficial effects on improving dependency, global neurological deficiency, and some specific neurological impairments for people with stroke in the convalescent stage, with no obvious serious adverse events. However, most included trials were of inadequate quality and size. There is, therefore, inadequate evidence to draw any conclusions about its routine use. Rigorously designed, randomised, multi‐centre, large sample trials of acupuncture for stroke are needed to further assess its effects.
- DEPRESSION The reduction in severity of depression was less when acupuncture was compared with control acupuncture than when acupuncture was compared with no treatment control, although in both cases, results were rated as providing low‐quality evidence. The reduction in severity of depression with acupuncture given alone or in conjunction with medication versus medication alone is uncertain owing to the very low quality of evidence. The effect of acupuncture compared with psychological therapy is unclear. The risk of adverse events with acupuncture is also unclear, as most trials did not report adverse events adequately. Few studies included follow‐up periods or assessed important outcomes such as quality of life. High‐quality randomised controlled trials are urgently needed to examine the clinical efficacy and acceptability of acupuncture, as well as its effectiveness, compared with acupuncture controls, medication, or psychological therapies.
- GAG REFLEX We found very low‐certainty evidence from four trials that was insufficient to conclude if there is any benefit of acupuncture, acupressure or laser at P6 point in reducing gagging and allowing successful completion of dental procedures. We did not find any evidence on any other interventions for managing the gag reflex during dental treatment. More well‐designed and well‐reported trials evaluating different interventions are needed.
- PERIPHERAL JOINT OSTEOARTHRITIS Sham‐controlled trials show statistically significant benefits; however, these benefits are small, do not meet our pre‐defined thresholds for clinical relevance, and are probably due at least partially to placebo effects from incomplete blinding. Waiting list‐controlled trials of acupuncture for peripheral joint osteoarthritis suggest statistically significant and clinically relevant benefits, much of which may be due to expectation or placebo effects.
- ELBOW PAIN There is insufficient evidence to either support or refute the use of acupuncture (either needle or laser) in the treatment of lateral elbow pain. This review has demonstrated needle acupuncture to be of short term benefit with respect to pain, but this finding is based on the results of 2 small trials, the results of which were not able to be combined in meta‐analysis. No benefit lasting more than 24 hours following treatment has been demonstrated. No trial assessed or commented on potential adverse effect. Further trials, utilising appropriate methods and adequate sample sizes, are needed before conclusions can be drawn regarding the effect of acupuncture on tennis elbow.
- MUMPS There is no evidence to determine the efficacy and safety of acupuncture in the treatment of children with mumps, although the excluded studies suggest that acupuncture is effective in improving swelling and pain of the parotid gland and assisting the body temperature to return to normal. We cannot make any recommendations for practice and nor can the results be generalised to clinical practice.
- MENOPAUSAL HOT FLUSHES We found insufficient evidence to determine whether acupuncture is effective for controlling menopausal vasomotor symptoms. When we compared acupuncture with sham acupuncture, there was no evidence of a significant difference in their effect on menopausal vasomotor symptoms. When we compared acupuncture with no treatment there appeared to be a benefit from acupuncture, but acupuncture appeared to be less effective than HT. These findings should be treated with great caution as the evidence was low or very low quality and the studies comparing acupuncture versus no treatment or HT were not controlled with sham acupuncture or placebo HT. Data on adverse effects were lacking.
- ASSISTED CONCEPTION There is no evidence that acupuncture improves live birth or pregnancy rates in assisted conception.
- HICCUPS A total of four studies (305 participants) met the inclusion criteria. All of these studies sought to determine the effectiveness of different acupuncture techniques in the treatment of persistent and intractable hiccups. All four studies had a high risk of bias, did not compare the intervention with placebo, and failed to report side effects or adverse events for either the treatment or control groups. Due to methodological differences we were unable to perform a meta‐analysis of the results. No studies investigating pharmacological interventions for persistent and intractable hiccups met the inclusion criteria.
- DYSPHAGIA IN ACUTE STROKE There is not enough evidence to make any conclusion about the therapeutic effect of acupuncture for dysphagia after acute stroke. High quality and large scale randomised controlled trials are needed.
- MOXIBUSTION FOR BREECH PRESENTATION This review found limited evidence to support the use of moxibustion for correcting breech presentation. There is some evidence to suggest that the use of moxibustion may reduce the need for oxytocin. When combined with acupuncture, moxibustion may result in fewer births by caesarean section; and when combined with postural management techniques may reduce the number of non‐cephalic presentations at birth, however, there is a need for well‐designed randomised controlled trials to evaluate moxibustion for breech presentation which report on clinically relevant outcomes as well as the safety of the intervention.
- CANCER PAIN There is insufficient evidence to judge whether acupuncture is effective in treating cancer pain in adults.
- URINARY INCONTINENCE The effect of acupuncture for stress urinary incontinence for adults is uncertain. There is not enough evidence to determine whether acupuncture is more effective than drug treatment.
- INDUCTION OF LABOUR Overall, there was no clear benefit from acupuncture or acupressure in reducing caesarean section rate. The quality of the evidence varied between low to high. Few trials reported on neonatal morbidity or maternal mortality outcomes. Acupuncture showed some benefit in improving cervical maturity, however, more well‐designed trials are needed. Future trials could include clinically relevant safety outcomes.
- NEUROPATHIC PAIN Due to the limited data available, there is insufficient evidence to support or refute the use of acupuncture for neuropathic pain in general, or for any specific neuropathic pain condition when compared with sham acupuncture or other active therapies. Five studies are still ongoing and seven studies are awaiting classification due to the unclear treatment duration, and the results of these studies may influence the current findings.
- PREMENSTRUAL SYNDROME The limited evidence available suggests that acupuncture and acupressure may improve both physical and psychological symptoms of PMS when compared to a sham control. There was insufficient evidence to determine whether there was a difference between the groups in rates of adverse events. There is no evidence comparing acupuncture or acupressure versus current ISPMD recommended treatments for PMS such as selective serotonin reuptake inhibitors (SSRIs). Further research is required, using validated outcome measures for PMS, adequate blinding and suitable comparator groups reflecting current best practice.
- IRRITABLE BOWEL SYNDROME Sham‐controlled RCTs have found no benefits of acupuncture relative to a credible sham acupuncture control for IBS symptom severity or IBS‐related quality of life. In comparative effectiveness Chinese trials, patients reported greater benefits from acupuncture than from two antispasmodic drugs (pinaverium bromide and trimebutine maleate), both of which have been shown to provide a modest benefit for IBS. Future trials may help clarify whether or not these reportedly greater benefits of acupuncture relative to pharmacological therapies are due entirely to patients’ preferences for acupuncture or greater expectations of improvement on acupuncture relative to drug therapy.
- ANKLE SPRAIN The currently available evidence from a very heterogeneous group of randomised and quasi‐randomised controlled trials evaluating the effects of acupuncture for the treatment of acute ankle sprains does not provide reliable support for either the effectiveness or safety of acupuncture treatments, alone or in combination with other non‐surgical interventions; or in comparison with other non‐surgical interventions. Future rigorous randomised clinical trials with larger sample sizes will be necessary to establish robust clinical evidence concerning the effectiveness and safety of acupuncture treatment for acute ankle sprains.
- MYOPAIA Two trials are included in this review but no conclusions can be drawn for the benefit of co‐acupressure for slowing progress of myopia in children. Further evidence in the form of RCTs are needed before any recommendations can be made for the use of acupuncture treatment in clinical use. These trials should compare acupuncture to placebo and have large sample sizes. Other types of acupuncture (such as auricular acupuncture) should be explored further as well as compliance with treatment for at least six months or longer. Axial length elongation of the eye should be investigated for at least one year. The potential to reduce/eliminate pain from acupuncture experienced by children should also be reviewed.
- CHRONIC KIDNEY DISEASE There was very low quality of evidence of the short‐term effects of manual acupressure as an adjuvant intervention for fatigue, depression, sleep disturbance and uraemic pruritus in patients undergoing regular haemodialysis. The paucity of evidence indicates that there is little evidence of the effects of other types of acupuncture for other outcomes, including pain, in patients with other stages of CKD. Overall high or unclear risk of bias distorts the validity of the reported benefit of acupuncture and makes the estimated effects uncertain. The incomplete reporting of acupuncture‐related harm does not permit us to assess the safety of acupuncture and related interventions. Future studies should investigate the effects and safety of acupuncture for pain and other common symptoms in patients with CKD and those undergoing dialysis.
- REHABILITATION OF BRAIN INJURY The low methodological quality of the included studies does not allow us to make conclusive judgments on the efficacy and safety of acupuncture in either the acute treatment and/or rehabilitation of TBI. Its beneficial role for these indications remains uncertain. Further research with high quality trials is required.
- POST-OPERATIVE NAUSEA AND VOMITING There is low‐quality evidence supporting the use of PC6 acupoint stimulation over sham. Compared to the last update in 2009, no further sham comparison trials are needed. We found that there is moderate‐quality evidence showing no difference between PC6 acupoint stimulation and antiemetic drugs to prevent PONV. Further PC6 acupoint stimulation versus antiemetic trials are futile in showing a significant difference, which is a new finding in this update. There is inconclusive evidence supporting the use of a combined strategy of PC6 acupoint stimulation and antiemetic drug over drug prophylaxis, and further high‐quality trials are needed.
The next and last part of this series will provide a few short comments on the current evidence.
The aim of this systematic review was to assess the efficacy of homeopathic remedies (HRs) in the treatment of mental disorders.
Italian psychiatrists performed a Medline/Embase search for studies written in English and published from any date to October 23, 2018. All randomized controlled trials enrolling patients with any psychiatric disorder and comparing HR with placebo, no treatment, or other psychotropic drugs were included.
A total of 212 studies were screened, 9 met all selection criteria and reported data on major depressive disorder (MDD) (n = 4), generalized anxiety disorder (n = 1), attention-deficit/hyperactivity disorder (n = 2), and premenstrual syndrome/dysphoric disorder (n = 2). Eight of 9 randomized controlled trials showed high risk of bias. Homeopathy showed greater efficacy in MDD compared with fluoxetine, and in premenstrual syndrome/dysphoric disorder compared with placebo, whereas no difference emerged between homeopathy and placebo in MDD and attention deficit/hyperactivity disorder.
The authors concluded that the available data on homeopathy in psychiatric disorders are insufficient to support their use in clinical practice.
In their discussion section, they also add an interesting note of caution: Ethical considerations should therefore prevent clinicians from recommending HRs, which have a cost either for patients or for health care systems, until when a sufficient amount of solid evidence becomes available. In addition, systematic reviews of randomized trials, if unavailable, are advisable for all medical conditions for which homeopathy is currently prescribed.
This is a rigorous, transparent and clear review which generates no surprises. Few critical thinkers would have expected a positive result. It also teaches us, I think, a valuable lesson about the difference between a rigorous and a flimsy review, between independent and biased research. In 2011, evidently pro-homeopathy authors published a paper of the latter kind. Here is its abstract:
Objective: To systematically review placebo-controlled randomized trials of homeopathy for psychiatric conditions.
Data sources: Eligible studies were identified using the following databases from database inception to April 2010: PubMed, CINAHL, PsycINFO, Hom-Inform, Cochrane CENTRAL, National Center for Complementary and Alternative Medicine grantee publications database, and ClinicalTrials.gov. Gray literature was also searched using Google, Google Scholar, the European Committee for Homeopathy, inquiries with homeopathic experts and manufacturers, and the bibliographic lists of included published studies and reviews. Search terms were as follows: (homeopath* or homoeopath*) and (placebo or sham) and (anxiety or panic or phobia or post-traumatic stress or PTSD or obsessive-compulsive disorder or fear or depress* or dysthym* or attention deficit hyperactivity or premenstrual syndrome or premenstrual disorder or premenstrual dysphoric disorder or traumatic brain injury or fibromyalgia or chronic fatigue syndrome or myalgic encephalitis or insomnia or sleep disturbance). Searches included only English-language literature that reported randomized controlled trials in humans.
Study selection: Trials were included if they met 7 criteria and were assessed for possible bias using the Scottish Intercollegiate Guidelines Network (SIGN) 50 guidelines. Overall assessments were made using the Grading of Recommendations Assessment, Development and Evaluation procedure. Identified studies were grouped into anxiety or stress, sleep or circadian rhythm complaints, premenstrual problems, attention-deficit/hyperactivity disorder, mild traumatic brain injury, and functional somatic syndromes.
Results: Twenty-five eligible studies were identified from an initial pool of 1,431. Study quality according to SIGN 50 criteria varied, with 6 assessed as good, 9 as fair, and 10 as poor. Outcome was unrelated to SIGN quality. Effect size could be calculated in 16 studies, and number needed to treat, in 10 studies. Efficacy was found for the functional somatic syndromes group (fibromyalgia and chronic fatigue syndrome), but not for anxiety or stress. For other disorders, homeopathy produced mixed effects. No placebo-controlled studies of depression were identified. Meaningful safety data were lacking in the reports, but the superficial findings suggested good tolerability of homeopathy. A funnel plot in 13 studies did not support publication bias (χ(2)(1) = 1.923, P = .166).
Conclusions: The database on studies of homeopathy and placebo in psychiatry is very limited, but results do not preclude the possibility of some benefit.
The two conclusions speak for themselves, I think. They should remind us that, although systematic reviews are in principle the most reliable source of evidence, it is still necessary to check the quality of the work and the independence of the worker.
This ‘Manifesto of the European Committee for Homeopathy (ECH) and the European Federation of Homeopathic Patients Associations (EFHPA)‘ has just been published. It is worth considering in more detail, I think. So, I will first reproduce the document in its entirety and subsequently provide some critical assessment of it.
Homeopathy: a solution for major healthcare problems in the EU
- Helps to reduce the need of antibiotics in human and veterinary health care, thus reducing the problem of antimicrobial resistance [i],[ii]
- Increases quality of life and reduces severity of complaints in patients with chronic disease, when integrated in health care [iii],[iv],[v],[vi],[vii],[viii]
- Can reduce the use of long-term conventional prescription drugs, when integrated in health care [ix]
Homeopathy: safe and cost-effective with a high patient satisfaction
- Can lead to lower health care costs, when integrated in health care, [x],[xi],[xii],
- Is safe, with high patient satisfaction [xiii],[xiv],[xv],[xvi]
- Patients using homeopathy have better outcomes than users of conventional treatment, with similar costs [xvii]
- Quality, safety and correct labelling of homeopathic products is guaranteed by Directive 2001/83 EC
EU consumers expect and demand homeopathy as part of their health care
- Reported as the most used medical complementary medicine in Europe [xviii]
- Three out of four European citizens know about homeopathy and out of them 29% use it for their day-to day health care [xix]
Scientific evidence of the highest calibre confirms the clinical efficacy of homeopathic medicine
- Clinical effects of homeopathic medicines have been confirmed by systematic reviews and meta- analyses [xx],[xxi],[xxii],[xxiii],[xxiv],[xxv],[xxvi]
There is convincing evidence for biological efficacy of homeopathic medicine
- Irrefutable scientific evidence has been published on the positive effects of homeopathic products in laboratory settings [xxvii],[xxviii]
[i] Grimaldi-Bensouda L, Bégaud B, Rossignol M, et al. Management of upper respiratory tract infections by different medical practices, including homeopathy, and consumption of antibiotics in primary care: the EPI3 cohort study in France 2007-2008. PLoS One. 2014 Mar 19;9(3):e89990
[ii] Camerlink I, Ellinger L, Bakker EJ, Lantinga EA. Homeopathy as replacement to antibiotics in the case of Escherichia coli diarrhoea in neonatal piglets. Homeopathy. 2010 Jan;99(1):57-62
[iii] Witt CM, Lüdtke R, Baur R, Willich SN. Homeopathic medical practice: long-term results of a cohort study with 3981 patients. BMC Public Health 2005; 5:115
[iv] Spence DS, Thompson EA, Barron SJ. Homeopathic treatment for chronic disease: a 6-year, university-hospital outpatient observational study. J Altern Complement Med 2005; 11:793–798
[v] Mathie RT, Robinson TW. Outcomes from homeopathic prescribing in medical practice: a prospective, research-targeted, pilot study. Homeopathy 2006; 95:199–205
[vi] Thompson EA, Mathie RT, Baitson ES, et al. Towards standard setting for patient-reported outcomes in the NHS homeopathic hospitals. Homeopathy 2008; 97:114–121
[vii] Witt CM, Lüdtke R, Mengler N, Willich SN. How healthy are chronically ill patients after eight years of homeopathic treatment?–Results from a long term observational study BMC Public Health 2008;8:413
[viii] Rossi E, Endrizzi C, Panozzo MA, Bianchi A, Da Frè M. Homeopathy in the public health system: a seven-year observational study at Lucca Hospital (Italy). Homeopathy 2009; 98:142–148
[ix] Grimaldi-Bensouda L, Abenhaim L, Massol J, et al. EPI3-LA-SER group. Homeopathic medical practice for anxiety and depression in primary care: the EPI3 cohort study. BMC Complement Altern Med. 2016 May 4; 16:125
[x] Kooreman P, Baars EW. Patients whose GP knows complementary medicine tend to have lower costs and live longer. Eur J Health Econ. 2012 Dec;13(6):769-76
[xi] Baars EW, Kooreman P. A 6-year comparative economic evaluation of healthcare costs and mortality rates of Dutch patients from conventional and CAM GPs. BMJ Open. 2014 Aug 27;4(8):e005332
[xii] Colas A, Danno K, Tabar C, Ehreth J, Duru G. Economic impact of homeopathic practice in general medicine in France. Health Econ Rev. 2015;5(1):55
[xiii] Van Wassenhoven M, Galen Y. An observational study of patients receiving homeopathic treatment. Homeopathy 2004 Jan;93(1):3-11
[xiv] Marian F, Joost K, Saini KD, von Ammon K, Thurneysen A, Busato A. Patient satisfaction and side effects in primary care: An observational study comparing homeopathy and conventional medicine. BMC Complement Altern Med. 2008 Sep 18; 8:52
[xv] Witt C, Keil T, Selim D, et al. Outcome and costs of homoeopathic and conventional treatment strategies: a comparative cohort study in patients with chronic disorders. Complement Ther Med. 2005;13(2):79-86
[xvi] Marian F, Joost K, Saini KD, von Ammon K, Thurneysen A, Busato A. Patient satisfaction and side effects in primary care: An observational study comparing homeopathy and conventional medicine. BMC Complement Altern Med. 2008 Sep 18; 8:52
[xvii] Bornhöft G, Wolf U, von Ammon K, Righetti M, Maxion-Bergemann S, Baumgartner S, Thurneysen AE, Matthiessen PF. Effectiveness, safety and cost-effectiveness of homeopathy in general practice – summarized health technology assessment.Forsch Komplementmed. 2006;13 Suppl 2:19-29. Epub 2006 Jun 26. Review
[xviii] Eardley S, Bishop FL, Prescott P, Cardini F, Brinkhaus B, Santos K Ͳ Rey, Vas J, von Ammon K, Hegyi G, Dragan S, Uehleke B, Fønnebø V, Lewith G. CAM use in Europe. The patients’ perspective.Part I: A systematic literature review of CAM prevalence in the EU. 2012. Online retrieved 19-11-2019. https://cam-europe.eu/wp-content/uploads/2018/09/CAMbrella-WP4-part_1final.pdf
[xix] Report of the European Commission, 1997. Online retrieved 15-12-2019 via https://www.hri-research.org/resources/essentialevidence/use-of-homeopathy-across-the-world/
[xx] Linde K, Clausius N, Ramirez G, Melchart D, Eitel F, Hedges LV, Jonas WB. Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials. Lancet. 1997 Sep 20;350(9081):834-4.
[xxi] Cucherat M, Haugh MC, Gooch M, Boissel JP.Evidence of clinical efficacy of homeopathy. A meta-analysis of clinical trials. HMRAG. Homeopathic Medicines Research Advisory Group. Eur J Clin Pharmacol. 2000 Apr;56(1):27-33
[xxii] Hahn RG. Homeopathy: meta-analyses of pooled clinical data. Forsch Komplementmed. 2013;20(5):376-81
[xxiii] Mathie RT, Van Wassenhoven M, Jacobs J et al. Model validity and risk of bias in randomised placebo-controlled trials of individualised homeopathic treatment. Complement Ther Med. 2016 Apr; 25:120-5
[xxiv] Mathie RT, Lloyd, SM, Legg, LA, Clausen J, Moss S, Davidson JR, Ford: Randomised placebo-controlled trials of individualised homeopathic treatment: systematic review and meta-analysis. Syst Rev 2014 Dec 6; 3:142
[xxv] Mathie RT, Clausen J. Veterinary homeopathy: systematic review of medical conditions studied by randomised placebo-controlled trials. Vet Rec. 2014 Oct 18;175(15):373-81.
[xxvi] Mathie RT, Clausen J. Veterinary homeopathy: meta-analysis of randomised placebo-controlled trials. Homeopathy. 2015 Jan;104(1):3-8.
[xxvii] Tournier A, Klein SD, Würtenberger S, Wolf U, Baumgartner S. Physicochemical Investigations of Homeopathic Preparations: A Systematic Review and Bibliometric Analysis-Part 2. J Altern Complement Med. 2019 Jul 10
[xxviii] Witt CM, Bluth M, Albrecht H, Weisshuhn TE, Baumgartner S, Willich SN. The in vitro evidence for an effect of high homeopathic potencies–a systematic review of the literature. Complement. Ther Med. 2007 Jun;15(2):128-38
Did I state above that the manifesto is worth considering in more detail? I need to retract or modify this statement.
Here are the considerations that are relevant, in my view:
- The statements in the manifesto are based on wishful thinking and do not reflect the reality based on the best evidence available today.
- The manifesto is the result of a mixture of cherry-picking and/or misinterpreting the evidence.
- Most of the cited studies have been discussed on this blog in previous posts which disclose their flaws and/or erroneous conclusions.
So, instead of discussing all the tedious details yet again, I will present here a corrected version of the manifesto:
Homeopathy: no solution for major healthcare problems in the EU
- Does not help to reduce the need of antibiotics in human and veterinary health care, thus reducing the problem of antimicrobial resistance
- does not increases quality of life and reduces severity of complaints in patients with chronic disease, when integrated in health care
- Cannot reduce the use of long-term conventional prescription drugs, when integrated in health care
Homeopathy: neither safe nor cost-effective with a high patient satisfaction
- Cannot lead to lower health care costs, when integrated in health care
- Is unsafe
- Patients using homeopathy have no better outcomes than users of conventional treatment, but cause higher costs
- Quality and correct labelling of homeopathic products is guaranteed by Directive 2001/83 EC
Some EU consumers expect and demand homeopathy as part of their health care
- Reported as a much-used complementary medicine in Europe
- Three out of four European citizens know about homeopathy and out of them many use it for their day-to day health care
Scientific evidence of the highest calibre fails to confirm the clinical efficacy of homeopathic medicine
- Clinical effects of homeopathic medicines have been confirmed by systematic reviews and meta- analyses to be no better than placebo
There is no convincing evidence for biological efficacy of homeopathic medicine
- No irrefutable scientific evidence has been published on the positive effects of homeopathic products in laboratory settings
He finds himself in the company of giants:
John Weeks (editor of JCAM)
Deepak Chopra (US entrepreneur)
Cheryl Hawk (US chiropractor)
David Peters (osteopathy, homeopathy, UK)
Nicola Robinson (TCM, UK)
Peter Fisher (homeopathy, UK)
Simon Mills (herbal medicine, UK)
Gustav Dobos (various, Germany)
Claudia Witt (homeopathy, Germany and Switzerland)
George Lewith (acupuncture, UK)
John Licciardone (osteopathy, US)
Why does Behnke deserve this honour?
No, there are better reasons.
On Twitter, Behnke describes himself as a research consultant for homeopathy at the Karl and Veronica Carstens-Foundation: Evidence based medicine, CAM, clinical and basic research, health. The Carstens Stiftung say he is ‘programme director integrative medicine’. On facebook, he is merely ‘ ‘Referent of ‘Redaktion Natur und Medizin’. And on ‘Research Gate’ he lists 12 areas of skills and expertise:
Evidence Based Medicine
Medical & Health Profession Education
Randomized Control Trials
Philosophy Of Science
Complementary & Alternative Medicine
If this is not impressive, I don’t know what is! Particularly, if one knows that he is not a medical doctor at all!!!
So, let’s look at the list to decide whether he deserves the honour of becoming a member of my ‘HALL OF FAME’. Specifically, let’s check how many Medline-listed articles he has to his name in each of the above areas:
Evidence Based Medicine = 0
Medical & Health Profession Education = 0
Meta-Analysis = 0
Observational Studies = 0
Science Communication = 0
Social Media = 0
Randomized Control Trials = 0
Clinical Research = 0
Philosophy Of Science = 0
Complementary & Alternative Medicine = 0e
Integrative Medicine = 0
Homeopathy = 0
(No, you don’t need to praise me for my detailed, time-consuming research. It was not difficult and very quick: Jens Behnke, the ‘research consultant, has precisely zero Medline-listed publications).
So has Behnke ever conducted:
- a meta-analysis? No
- an observational study? I don’t think so
- a randomised trial? No
- any other clinical research? No
In the past, I tended to admit to my HALL OF FAME mainly those SCAM researchers who had published plenty of papers but had no study to their name that drew a negative conclusion. Behnke is not in that league. He is nevertheless worthy for his highly elaborate concept. Remember, he is a ‘research consultant in homeopathy’, and homeopathy obeys different rules than any other form of quackery. One of its axioms holds that LESS IS MORE. And considering this principle, Behnke surely must be THE expert! No publication, in homeopathic logic, evidently means that he is better than anyone else.
So, a warm welcome to our new member Jens Behnke: MAY YOUR UNPRODUCTIVITY AS A EXPERT IN 12 DIFFERENT FIELDS OF INQUIRY LAST FOR MANY MORE YEARS!
And congratulations also to the Carstens Stiftung who have so far spent 36 000 000 Euro on SCAM-research and pay Behnke’s salary as ‘research consultant’: I am sure you guys deserve him!
In case Dr Behnke reads this: it is an internationally accepted standard of honesty and transparency that someone who has a doctor title and works in or comments on medical matters makes it clear that he/she is not medically trained or experienced, that in fact he/she is not a medical doctor. If not, one might think that this person is deliberately trying to mislead the public.
Yesterday’s blog disclosed the fact that the German ‘Natur und Medizin’, an organisation of the ‘Carstens Stiftung’, had published slanderous lies about me. Consequently, I published an ‘open letter’ urging them to correct their mistake so that they would spare us the agony and cost of using legal action.
I never doubted for a minute that they would do this (I do not assume they are stupid, just a tiny bit dishonest) – and, as it turned out, I was correct. Here is a reminder of what they had originally published:
… er ist dafür bekannt, dass er kein gutes Haar an komplementären Therapieverfahren lässt. Notfalls greift er auch zu absichtlichen Falschdarstellungen, erfindet Daten oder behauptet einfach, klinische Studien, die nicht die Negativ-Ergebnisse erbringen, die er erwartet, seien schlicht und ergreifend Betrug.…
My rough translation:
… he [Edzard Ernst] is known for not finding anything positive in SCAM. If all else fails, he uses deliberate misrepresentation , invents data , or simply claims that clinical trials which did not generate the negative findings he expected are simply falsifications …
The corrected new text passage is a little longer and now reads as follows (my rough translation):
… he [Edzard Ernst] is known for not finding anything positive in SCAM. Analyses of his publications by independent scientists draw the conclusion that he represents case-reports demonstrably wrongly  and that he arbitrarily alters or omits data . He claims occasionally that high-quality studies of SCAM which do not generate the negative findings he expected appeared to be scientifically sound, but are nevertheless not believable …
… er ist dafür bekannt, dass er kein gutes Haar an komplementären Therapieverfahren lässt. Analysen seiner Publikationen durch unabhängige Wissenschaftler gelangen zu der Schlussfolgerung, dass er Fallberichte nachweislich falsch darstelle und Daten willkürlich verändere oder auslasse. Er selbst behauptet mitunter über methodisch hochwertige Studien zur Komplementärmedizin, die nicht die Negativ-Ergebnisse erbringen, die er erwartet, sie sähen zwar nach wissenschaftlichen Maßstäben überzeugend aus, seien aber dennoch ‚unglaubwürdig‘.…
I would like to take this occasion to sincerely thank the ‘Natur und Medizin’ and the ‘Carstens Stiftung’ for this – much obliged guys, you made my day!
- They have shown wisdom in not wasting money on expensive lawyers (even though my brother, who is a lawyer, might have enjoyed the windfall).
- They have shown courage to hide behind papers like the one by Robert Hahn which have been discussed on this blog and elsewhere and found to be deluded.
- They have shown strength by not meekly apologising to me about their attempt to slander me and my work.
- They show leadership and innovative spirit by employing Jens Behnke, the author of the above lines, who does not seem to let the truth get in the way of a good story.
Last not least, my personal thanks to dear Jens (after your generosity, I am thinking about dedicating an entire blog post to you; your employer needs to know what a genius they have in you – watch this space) for yet again having demonstrated that the phenomenon known as ERNST’ S LAW is 100% correct.
An enthusiast of homeopathy recently posted an overview of systematic reviews of homeopathy concluding that the data we do have point towards homeopathy as having an effect greater than that of placebo:
In recent decades, homeopathy has been examined via a number of clinical trials, the number of which now allow meta-analysis. As we can see from the study findings, the type of homeopathy research (ie, individualized vs non-individualized, placebo-controlled vs non-placebo-controlled) can have a strong influence on the results, although trial quality also has a strong effect.
All meta-analyses performed in at least a somewhat open and rigorous manner have found statistically significant effects. This suggests that homeopathy has a greater-than-placebo effect, or at least a strong trend in that direction, when using data from the totality of homeopathy research, or from individualized, placebo-controlled trials. The meta-analyses with questionable methodology, one of which is undergoing government investigation for academic irregularities, have produced negative results, which have been demonstrated to be a direct result of their exclusion of vast swathes of the homeopathic clinical trial literature (based on arbitrary and unexplained criteria), as well as of their failure to differentiate – as Mathie has done – different types of homeopathic research.
The clinical data are flawed. Issues with methodology used in homeopathy RCTs, combined with a lack of research funding, have produced a lack of high-quality trials and data. However, the data we do have point towards homeopathy as having an effect greater than that of placebo.
There can be no argument with this conclusion, aside from possible new data emerging. Anyone who disputes this is going against the existing set of the highest-quality evidence on homeopathy.
His overview is based on the following publications:
|Kleijnen, 19911||All types of homeopathy (eg, single remedy vs combination). Methodological quality assessed; 105 trials. Results: Positive trend, regardless of type of homeopathy; 81 trials were positive, 24 showed no effect.|
|Linde, 19972||All types of homeopathy. Out of 185 trials, 119 met inclusion criteria; 89 of these had extractable data. Results: OR = 2.45 (95% CI 2.05-2.93).|
|Ernst, 19983||Individualized homeopathy; 5 trials determined to be high-quality. Results: OR = 0.|
|Linde, 19985||Individualized homeopathy; 32 trials, 19 of which had extractable data. Results: OR = 1.62 for all trials (95% CI 1.17-2.23). Only high-quality trials produced no significant trend.|
|Cucherat, 20009||All types of homeopathy; 118 trials, 16 of which met inclusion criteria. Used unusual method of combining p values. Results: All trials = p< 0.000036. Less than 10% dropouts: p<0.084; less than 5% dropouts (higher standards than most trials considered reliable): p<0.08 (non-significant).|
|Shang, 200511||All types of homeopathy; only 8 trials selected from 21 high-quality trials of 110 selected with unusual criteria. Results: OR = 0.88 (0.65-1.19). Result strongly disputed by statisticians.|
|Mathie, 201413||Individualized homeopathy; of the analysis pooled data from 22 higher-quality, individualized, double-blind RCTs. Results: OR = 1.53 (1.22-1.91) for all trials pooled; OR = 1.93 (1.16-3.38) for the 3 reliable trials.|
|NHMRC, 201516||Out of 176 studies, 171 were excluded, leaving only 5 for the study. Investigators used unprecedented methods, did not combine data, and are currently under investigation for outcome shopping. Results: Negative results.|
|Mathie, 201720||Non-individualized homeopathy; very few higher-quality trials. Results: For 54 trials with extractable data, SMD = -0.33 (-0.44, -0.21). When these were adjusted for publication bias, SMD = -0.16 (-0.46,-0.09). The 3 high-quality trials had non-significant results: SMD = -0.18 (-0.46, +0.09).|
|Mathie, 201821||Individualized, other-than-placebo-controlled trials; 11 trials found, 8 with extractable data. Results: 4 heterogeneous comparative trials showed a non-significant difference. One trial in this group was positive. Three heterogeneous trials with additive homeopathy showed a statistically significant SMD. No definitive conclusion possible due to trial heterogeneity, poor quality, and low number of trials.|
|Mathie, 201922||Non-individualized, other-than-placebo-controlled trials; 17 RCTs found, 14 with high risk of bias. Results: Significant heterogeneity prevented much comparison; 3 comparable trials showed a non-significant SMD.|
- Many older systematic reviews were omitted (including about 10 of my own papers). This is relevant because the author of the above review went beck until 1991 to find the reviews he included.
- Several new papers were missing as well. This is relevant because the author evidently included reviews up to 2019. Here are the key passages from the conclusions of some of them:
We tested whether p-curve accurately rejects the evidential value of significant results obtained in placebo-controlled clinical trials of homeopathic ultramolecular dilutions. Our results suggest that p-curve can accurately detect when sets of statistically significant results lack evidential value.
I am, of course, not saying that this overview amounts to anything like a systematic review. It merely gives you a flavour how trustworthy proponents of homeopathy are when they pretend to provide us with an objective evaluation of the best available evidence.
The systematic review assessed the evidence of Craniosacral Therapy (CST) for the treatment of chronic pain. Randomized clinical trials (RCTs) assessing the effects of CST in chronic pain patients were eligible. Pain intensity and functional disability were the primary outcomes. Risk of bias was assessed using the Cochrane tool.
Ten RCTs with a total of 681 patients suffering from neck and back pain, migraine, headache, fibromyalgia, epicondylitis, and pelvic girdle pain were included.
Compared to treatment as usual, CST showed greater post intervention effects on:
- pain intensity (SMD=-0.32, 95%CI=[−0.61,-0.02])
- disability (SMD=-0.58, 95%CI=[−0.92,-0.24]).
Compared to manual/non-manual sham, CST showed greater post intervention effects on:
- pain intensity (SMD=-0.63, 95%CI=[−0.90,-0.37])
- disability (SMD=-0.54, 95%CI=[−0.81,-0.28]) ;
Compared to active manual treatments, CST showed greater post intervention effects on:
- pain intensity (SMD=-0.53, 95%CI=[−0.89,-0.16])
- disability (SMD=-0.58, 95%CI=[−0.95,-0.21]) .
At six months, CST showed greater effects on pain intensity (SMD=-0.59, 95%CI=[−0.99,-0.19]) and disability (SMD=-0.53, 95%CI=[−0.87,-0.19]) versus sham. Secondary outcomes were all significantly more improved in CST patients than in other groups, except for six-month mental quality of life versus sham. Sensitivity analyses revealed robust effects of CST against most risk of bias domains. Five of the 10 RCTs reported safety data. No serious adverse events occurred. Minor adverse events were equally distributed between the groups.
The authors concluded that, in patients with chronic pain, this meta-analysis suggests significant and robust effects of CST on pain and function lasting up to six months. More RCTs strictly following CONSORT are needed to further corroborate the effects and safety of CST on chronic pain.
Robust effects! This looks almost convincing, particularly to an uncritical proponent of so-called alternative medicine (SCAM). However, a bit of critical thinking quickly discloses numerous problems, not with this (technically well-made) review, but with the interpretation of its results and the conclusions. Let me mention a few that spring into my mind:
- The literature searches were concluded in August 2018; why publish the paper only in 2020? Meanwhile, there might have been further studies which would render the review outdated even on the day it was published. (I know that there are many reasons for such a delay, but a responsible journal editor must insist on an update of the searches before publication.)
- Comparisons to ‘treatment as usual’ do not control for the potentially important placebo effects of CST and thus tell us nothing about the effectiveness of CST per se.
- The same applies to comparisons to ‘active’ manual treatments and ‘non-manual’ sham (the purpose of a sham is to blind patients; a non-manual sham defies this purpose).
- This leaves us with exactly two trials employing a sham that might have been sufficiently credible to be able to fool patients into believing that they were receiving the verum.
- One of these trials (ref 44) is far too flimsy to be taken seriously: it was tiny (n=23), did not adequately blind patients, and failed to mention adverse effects (thus violating research ethics [I cannot take such trials seriously]).
- The other trial (ref 41) is by the same research group as the review, and the authors award themselves a higher quality score than any other of the primary studies (perhaps even correctly, because the other trials are even worse). Yet, their study has considerable weaknesses which they fail to discuss: it was small (n=54), there was no check to see whether patient-blinding was successful, and – as with all the CST studies – the therapist was, of course, no blind. The latter point is crucial, I think, because patients can easily be influenced by the therapists via verbal or non-verbal communication to report the findings favoured by the therapist. This means that the small effects seen in such studies are likely to be due to this residual bias and thus have nothing to do with the intervention per se.
- Despite the fact that the review findings depend critically on their own primary study, the authors of the review declared that they have no conflict of interest.
Considering all this plus the rather important fact that CST completely lacks biological plausibility, I do not think that the conclusions of the review are warranted. I much prefer the ones from my own systematic review of 2012. It included 6 RCTs (all of which were burdened with a high risk of bias) and concluded that the notion that CST is associated with more than non‐specific effects is not based on evidence from rigorous RCTs.
So, why do the review authors first go to the trouble of conducting a technically sound systematic review and meta-analysis and then fail utterly to interpret its findings critically? I might have an answer to this question. Back in 2016, I included the head of this research group, Gustav Dobos, into my ‘hall of fame’ because he is one of the many SCAM researchers who never seem to publish a negative result. This is what I then wrote about him:
Dobos seems to be an ‘all-rounder’ whose research tackles a wide range of alternative treatments. That is perhaps unremarkable – but what I do find remarkable is the impression that, whatever he researches, the results turn out to be pretty positive. This might imply one of two things, in my view:
- all alternative therapies are effective,
- the ‘Trustworthiness Index’ of Prof Dobos is unusual.
I let my readers chose which possibility they deem to be more likely.
Acupuncture is often recommended for relieving symptoms of fibromyalgia syndrome (FMS). The aim of this systematic review was to ascertain whether verum acupuncture is more effective than sham acupuncture in FMS.
Ten RCTs with a total of 690 participants were eligible, and 8 RCTs were eventually included in the meta-analysis. Its results showed a sizable effect of verum acupuncture compared with sham acupuncture on pain relief, improving sleep quality and reforming general status. Its effect on fatigue was insignificant. When compared with a combination of simulation and improper location of needling, the effect of verum acupuncture for pain relief was the most obvious.
The authors concluded that verum acupuncture is more effective than sham acupuncture for pain relief, improving sleep quality, and reforming general status in FMS posttreatment. However, evidence that it reduces fatigue was not found.
I have a much more plausible conclusion for these findings: in (de-randomised) trials comparing real and sham acupuncture, patients are regularly de-blinded and therapists are invariably not blind. The resulting bias and not the alleged effectiveness of acupuncture explains the outcome.
And why do I think that this conclusion is much more plausible?
Firstly, because of Occam’s Razor.
Secondly, because this is roughly what my own systematic review of the subject found (The notion that acupuncture is an effective symptomatic treatment for fibromyaligia is not supported by the results from rigorous clinical trials. On the basis of this evidence, acupuncture cannot be recommended for fibromyalgia). This view is also shared by other critical reviews of the evidence (Current literature does not support the routine use of acupuncture for improving pain or quality of life in FM). Perhaps more crucially, the current Cochrane review seems to concur: There is low to moderate-level evidence that compared with no treatment and standard therapy, acupuncture improves pain and stiffness in people with fibromyalgia. There is moderate-level evidence that the effect of acupuncture does not differ from sham acupuncture in reducing pain or fatigue, or improving sleep or global well-being. EA is probably better than MA for pain and stiffness reduction and improvement of global well-being, sleep and fatigue. The effect lasts up to one month, but is not maintained at six months follow-up. MA probably does not improve pain or physical functioning. Acupuncture appears safe. People with fibromyalgia may consider using EA alone or with exercise and medication. The small sample size, scarcity of studies for each comparison, lack of an ideal sham acupuncture weaken the level of evidence and its clinical implications. Larger studies are warranted.
The journal NATURE has just published an excellent article by Andrew D. Oxman and an alliance of 24 leading scientists outlining the importance and key concepts of critical thinking in healthcare and beyond. The authors state that the Key Concepts for Informed Choices is not a checklist. It is a starting point. Although we have organized the ideas into three groups (claims, comparisons and choices), they can be used to develop learning resources that include any combination of these, presented in any order. We hope that the concepts will prove useful to people who help others to think critically about what evidence to trust and what to do, including those who teach critical thinking and those responsible for communicating research findings.
Here I take the liberty of citing a short excerpt from this paper:
Claims about effects should be supported by evidence from fair comparisons. Other claims are not necessarily wrong, but there is an insufficient basis for believing them.
Claims should not assume that interventions are safe, effective or certain.
- Interventions can cause harm as well as benefits.
- Large, dramatic effects are rare.
- We can rarely, if ever, be certain about the effects of interventions.
Seemingly logical assumptions are not a sufficient basis for claims.
- Beliefs alone about how interventions work are not reliable predictors of the presence or size of effects.
- An outcome may be associated with an intervention but not caused by it.
- More data are not necessarily better data.
- The results of one study considered in isolation can be misleading.
- Widely used interventions or those that have been used for decades are not necessarily beneficial or safe.
- Interventions that are new or technologically impressive might not be better than available alternatives.
- Increasing the amount of an intervention does not necessarily increase its benefits and might cause harm.
Trust in a source alone is not a sufficient basis for believing a claim.
- Competing interests can result in misleading claims.
- Personal experiences or anecdotes alone are an unreliable basis for most claims.
- Opinions of experts, authorities, celebrities or other respected individuals are not solely a reliable basis for claims.
- Peer review and publication by a journal do not guarantee that comparisons have been fair.
Studies should make fair comparisons, designed to minimize the risk of systematic errors (biases) and random errors (the play of chance).
Comparisons of interventions should be fair.
- Comparison groups and conditions should be as similar as possible.
- Indirect comparisons of interventions across different studies can be misleading.
- The people, groups or conditions being compared should be treated similarly, apart from the interventions being studied.
- Outcomes should be assessed in the same way in the groups or conditions being compared.
- Outcomes should be assessed using methods that have been shown to be reliable.
- It is important to assess outcomes in all (or nearly all) the people or subjects in a study.
- When random allocation is used, people’s or subjects’ outcomes should be counted in the group to which they were allocated.
Syntheses of studies should be reliable.
- Reviews of studies comparing interventions should use systematic methods.
- Failure to consider unpublished results of fair comparisons can bias estimates of effects.
- Comparisons of interventions might be sensitive to underlying assumptions.
Descriptions should reflect the size of effects and the risk of being misled by chance.
- Verbal descriptions of the size of effects alone can be misleading.
- Small studies might be misleading.
- Confidence intervals should be reported for estimates of effects.
- Deeming results to be ‘statistically significant’ or ‘non-significant’ can be misleading.
- Lack of evidence for a difference is not the same as evidence of no difference.
What to do depends on judgements about the problem, the relevance (applicability or transferability) of evidence available and the balance of expected benefits, harm and costs.
Problems, goals and options should be defined.
- The problem should be diagnosed or described correctly.
- The goals and options should be acceptable and feasible.
Available evidence should be relevant.
- Attention should focus on important, not surrogate, outcomes of interventions.
- There should not be important differences between the people in studies and those to whom the study results will be applied.
- The interventions compared should be similar to those of interest.
- The circumstances in which the interventions were compared should be similar to those of interest.
Expected pros should outweigh cons.
- Weigh the benefits and savings against the harm and costs of acting or not.
- Consider how these are valued, their certainty and how they are distributed.
- Important uncertainties about the effects of interventions should be reduced by further fair comparisons.
END OF QUOTE
I have nothing to add to this, except perhaps to point out how very relevant all of this, of course, is for SCAM and to warmly recommend you study the full text of this brilliant paper.