MD, PhD, FMedSci, FRSB, FRCP, FRCPEd.

meta-analysis

1 2 3 6

George Vithoulkas, has been mentioned on this blog repeatedly. He is a lay homeopath – one that has no medical background – and has, over the years, become an undisputed hero within the world of homeopathy. Yet, Vithoulkas’ contribution to homeopathy research is perilously close to zero. Judging from a recent article in which he outlines the rules of rigorous research, his understanding of research methodology is even closer to zero. Here is a crucial excerpt from this paper intercepted by a few comment from me in brackets and bold print.

Which are [the] homoeopathic principles to be respected [in clinical trials and meta-analyses]?

1. Homoeopathy does not treat diseases, but only diseased individuals. Therefore, every case may need a different remedy although the individuals may be suffering from the same pathology. This rule was violated by almost all the trials in most meta-analyses. (This statement is demonstrably false; there even has been a meta-analysis of 32 trials that respect this demand)

2. In the homoeopathic treatment of serious chronic pathology, if the remedy is correct usually a strong initial aggravation takes place []. Such an aggravation may last from a few hours to a few weeks and even then we may have a syndrome-shift and not the therapeutic results expected. If the measurements take place in the aggravation period, the outcome will be classified negative. (Homeopathic aggravations exist only in the mind of homeopaths; our systematic review failed to find proof for their existence.)

This factor was also ignored in most trials []. At least sufficient time should be given in the design of the trial, in order to account for the aggravation period. The contrary happened in a recent study [], where the aggravation period was evaluated as a negative sign and the homoeopathic group was pronounced worse than the placebo []. (There are plenty of trials where the follow-up period is long enough to account for this [non-existing] phenomenon.)

3. In severe chronic conditions, the homoeopath may need to correctly prescribe a series of remedies before the improvement is apparent. Such a second or third prescription should take place only after evaluating the effects of the previous remedies []. Again, this rule has also been ignored in most studies. (Again, this is demonstrably wrong; there are many trials where the homeopath was able to adjust his/her prescription according to the clinical response of the patient.)

4. As the prognosis of a chronic condition and the length of time after which any amelioration set in may differ from one to another case [], the treatment and the study-design respectively should take into consideration the length of time the disease was active and also the severity of the case. (This would mean that conditions that have a short history, like post-operative ileus, bruising after injury, common cold, etc. should respond well after merely a short treatment with homeopathics. As this is not so, Vithoulkas’ argument seems to be invalid.)

5. In our experience, Homeopathy has its best results in the beginning stages of chronic diseases, where it might be possible to prevent the further development of the chronic state and this is its most important contribution. Examples of pathologies to be included in such RCTs trials are ulcerative colitis, sinusitis, asthma, allergic conditions, eczema, gangrene rheumatoid arthritis as long as they are within the first six months of their appearance. (Why then is there a lack of evidence that any of the named conditions respond to homeopathy?)

In conclusion, three points should be taken into consideration relating to trials that attempt to evaluate the effectiveness of homoeopathy.

First, it is imperative that from the point of view of homoeopathy, the above-mentioned principles should be discussed with expert homoeopaths before researchers undertake the design of any homoeopathic protocol. (I am not aware of any trial where this was NOT done!)

Second, it would be helpful if medical journals invited more knowledgeable peer-reviewers who understand the principles of homoeopathy. (I am not aware of any trial where this was NOT done!)

Third, there is a need for at least one standardized protocol for clinical trials that will respect not only the state-of-the-art parameters from conventional medicine but also the homoeopathic principles []. (Any standardised protocol would be severely criticised; a good study protocol must always take account of the specific research question and therefore cannot be standardised.)

Fourth, experience so far has shown that the therapeutic results in homeopathy vary according to the expertise of the practitioner. Therefore, if the objective is to validate the homeopathic therapeutic modality, the organizers of the trial have to pick the best possible prescribers existing in the field. (I am not aware of any trial where this was NOT done!)

Only when these points are transposed and put into practice, the trials will be respected and accepted by both homoeopathic practitioners and conventional medicine and can be eligible for meta-analysis.

___________________________________________________________________

I suspect what the ‘GREAT VITHOULKAS’ really wanted to express are ‘THE TWO ESSENTIAL PRINCIPLES OF HOMEOPATHY RESEARCH’:

  1. A well-designed study of homeopathy can always be recognised by its positive result.
  2. Any trial that fails to yield a positive finding is, by definition, wrongly designed.

“Eating elderberries can help minimise influenza symptoms.” This statement comes from a press release by the University of Sydney. As it turned out, the announcement was not just erroneous but it also had concealed that the in-vitro study that formed the basis for the press-release was part-funded by the very company, Pharmacare, which sells elderberry-based flu remedies.

“This is an appalling misrepresentation of this Pharmacare-funded in-vitro study,” said associate professor Ken Harvey, president of Friends of Science in Medicine. “It was inappropriate and misleading to imply from this study that an extract was ‘proven to fight flu’.” A University of Sydney spokeswoman confirmed Pharmacare was shown a copy of the press release before it was published.

This is an embarrassing turn of events, no doubt. But what about elderberry (Sambucus nigra) and the flu? Is there any evidence?

A systematic review quantified the effects of elderberry supplementation. Supplementation with elderberry was found to substantially reduce upper respiratory symptoms. The quantitative synthesis of the effects yielded a large mean effect size. The authors concluded that these findings present an alternative to antibiotic misuse for upper respiratory symptoms due to viral infections, and a potentially safer alternative to prescription drugs for routine cases of the common cold and influenza.

WHAT?!?!

The alternative to antibiotic misuse can only be the correct use of antibiotics. And, in the case of viral infections such as the flu, this can only be the non-use of antibiotics. My trust in this review, published in a SCAM journal of dubious repute, has instantly dropped to zero.

Perhaps a recent overview recently published in THE MEDICAL LETTER provides a more trustworthy picture:

No large randomized, controlled trials evaluating the effectiveness of elderberry for prevention or treatment of influenza have been conducted to date. Elderberry appears to have some activity against influenza virus strains in vitro. In two small studies (conducted outside the US), adults with influenza A or B virus infection taking elderberry extract reported a shorter duration of symptoms compared to those taking placebo. Consuming uncooked blue or black elderberries can cause nausea and vomiting. The rest of the plant (bark, stems, leaves, and root) contains sambunigrin, which can release cyanide. No data are available on the safety of elderberry use during pregnancy or while breastfeeding. CONCLUSION — Prompt treatment with an antiviral drug such as oseltamivir (Tamiflu, and generics) has been shown to be effective in large randomized, controlled trials in reducing the duration of influenza symptoms, and it may reduce the risk of influenza-related complications. There is no acceptable evidence to date that elderberry is effective for prevention or treatment of influenza and its safety is unclear.

Any take-home messages?

Yes:

  1. Elderberry supplements are not of proven effectiveness against the flu.
  2. The press officers at universities should be more cautious when writing press-releases.
  3. They should involve the scientists and avoid the sponsors of the research.
  4. In-vitro studies can never tell us anything about clinical effectiveness.
  5. SCAM-journals’ articles must be taken with a pinch of salt.
  6. Consumers are being misled left, right and centre.

Radix Salviae Miltiorrhizae (Danshen) is a herbal remedy that is part of many TCM herbal mixtures. Allegedly, Danshen has been used in clinical practice for over 2000 years.

But is it effective?

The aim of this systematic review was to evaluate the current available evidence of Danshen for the treatment of cancer. English and Chinese electronic databases were searched from PubMed, the Cochrane Library, EMBASE, and the China National Knowledge Infrastructure (CNKI), VIP database, Wanfang database until September 2018. The methodological quality of the included studies was evaluated by using the method of Cochrane system.

Thirteen RCTs with 1045 participants were identified. The studies investigated the lung cancer (n = 5), leukemia (n = 3), liver cancer (n = 3), breast or colon cancer (n = 1), and gastric cancer (n = 1). A total of 83 traditional Chinese medicines were used in all prescriptions and there were three different dosage forms. The meta-analysis suggested that Danshen formulae had a significant effect on RR (response rate) (OR 2.38, 95% CI 1.66-3.42), 1-year survival (OR 1.70 95% CI 1.22-2.36), 3-year survival (OR 2.78, 95% CI 1.62-4.78), and 5-year survival (OR 8.45, 95% CI 2.53-28.27).

The authors concluded that the current research results showed that Danshen formulae combined with chemotherapy for cancer treatment was better than conventional drug treatment plan alone.

I am getting a little tired of discussing systematic reviews of so-called alternative medicine (SCAM) that are little more than promotion, free of good science. But, because such articles do seriously endanger the life of many patients, I do nevertheless succumb occasionally. So here are a few points to explain why the conclusions of the Chinese authors are nonsense:

  • Even though the authors claim the trials included in their review were of high quality, most were, in fact, flimsy.
  • The trials used no less than 83 different herbal mixtures of dubious quality containing Danshen. It is therefore not possible to define which mixture worked and which did not.
  • There is no detailed discussion of the adverse effects and no mention of possible herb-drug interactions.
  • There seemed to be a sizable publication bias hidden in the data.
  • All the eligible studies were conducted in China, and we know that such trials are unreliable to say the least.
  • Only four articles were published in English which means those of us who cannot read Chinese are unable to check the correctness of the data extraction of the review authors.

I know it sounds terribly chauvinistic, but I do truly believe that we should simply ignore Chinese articles, if they have defects that set our alarm bells ringing – if not, we are likely to do a significant disservice to healthcare and progress.

Did we not have a flurry of systematic reviews of homeopathy in recent months?

And were they not a great disappointment to homeopaths and their patients?

Just as we thought that this is more than enough evidence to show that homeopathy is not effective, here comes another one.

This new review evaluated RCTs of non-individualised homeopathic treatment (NIHT) in which the control group received treatments other than placebo (OTP). Specifically, its aim was to determine the comparative effectiveness of NIHT on health-related outcomes for any given condition.

For each eligible trial, published in the peer-reviewed literature up to the end of 2016, the authors assessed its risk of bias (internal validity) using the seven-domain Cochrane tool, and its relative pragmatic or explanatory attitude (external validity) using the 10-domain PRECIS tool. The researchers grouped RCTs by whether these examined homeopathy as an alternative treatment (study design 1a), adjunctively with another intervention (design 1b), or compared with no intervention (design 2). RCTs were sub-categorised as superiority trials or equivalence/non-inferiority trials. For each RCT, a single ‘main outcome measure’ was selected to use in meta-analysis.

Seventeen RCTs, representing 15 different medical conditions, were eligible for inclusion. Three of the trials were more pragmatic than explanatory, two were more explanatory than pragmatic, and 12 were equally pragmatic and explanatory. Fourteen trials were rated ‘high risk of bias’ overall; the other three trials were rated ‘uncertain risk of bias’ overall. Ten trials had data that were extractable for meta-analysis. Significant heterogeneity undermined the planned meta-analyses or their meaningful interpretation. For the three equivalence or non-inferiority trials with extractable data, the small, non-significant, pooled effect size was consistent with a conclusion that NIHT did not differ from treatment by a comparator (Ginkgo biloba or betahistine) for vertigo or (cromolyn sodium) for seasonal allergic rhinitis.

The authors concluded that the current data preclude a decisive conclusion about the comparative effectiveness of NIHT. Generalisability of findings is restricted by the limited external validity identified overall. The highest intrinsic quality was observed in the equivalence and non-inferiority trials of NIHT.

I do admire the authors’ tenacity in meta-analysing homeopathy trials and empathise with their sadness of the multitude of negative results they thus have to publish. However, I do disagree with their conclusions. In my view, at least two firm conclusions ARE possible:

  1. This dataset confirms yet again that the methodological quality of homeopathy trials is lousy.
  2. The totality of the trial evidence analysed here fails to show that non-individualised homeopathy is effective.

In case you wonder why the authors are not more outspoken about their own findings, perhaps you need to read their statement of conflicts of interest:

Authors RTM, YYYF, PV and AKLT are (or were) associated with a homeopathy organisation whose significant aim is to clarify and extend an evidence base in homeopathy. RTM holds an independent research consultancy contract with the Deutsche Homöopathie-Union, Karlsruhe, Germany. YYYF and AKLT belong to Living Homeopathy Ltd., which has contributed funding to some (but not this current) HRI project work. RTM and PV have no other relationships or activities that could appear to have influenced the submitted work. JRTD had no support from any organisation for the submitted work; in the last 3 years, and for activities outside the submitted study, he received personal fees, royalties or out-of-pocket expenses for advisory work, invitational lectures, use of rating scales, published book chapters or committee membership; he receives royalties from Springer Publishing Company for his book, A Century of Homeopaths: Their Influence on Medicine and Health. JTRD has no other relationships or activities that could appear to have influenced the submitted study.

If one had wanted to add insult to injury, one could have added that, if, despite such conflicts of interest, the overall result of this new review turned out to be not positive, the evidence must be truly negative.

The Spanish Ministries of Health and Sciences have announced their ‘Health Protection Plan against Pseudotherapies’. Very wisely, they have included chiropractic under this umbrella. To a large degree, this is the result of Spanish sceptics pointing out that alternative therapies are a danger to public health, helped perhaps a tiny bit also by the publication of two of my books (see here and here) in Spanish. Unsurprisingly, such delelopments alarm Spanish chiropractors who fear for their livelihoods. A quickly-written statement of the AEQ (Spanish Chiropractic Association) is aimed at averting the blow. It makes the following 11 points (my comments are below):

1. The World Health Organization (WHO) defines chiropractic as a healthcare profession. It is independent of any other health profession and it is neither a therapy nor a pseudotherapy.

2. Chiropractic is statutorily recognised as a healthcare profession in many European countries including Portugal, France, Italy, Switzerland, Belgium, Denmark, Sweden, Norway and the United Kingdom10, as well as in the USA, Canada and Australia, to name a few.

3. Chiropractic members of the AEQ undergo university-level training of at least 5 years full-time (300 ECTS points). Chiropractic training is offered within prestigious institutions such as the Medical Colleges of the University of Zurich and the University of Southern Denmark.

4. Chiropractors are spinal health care experts. Chiropractors practice evidence-based, patient-centred conservative interventions, which include spinal manipulation, exercise prescription, patient education and lifestyle advice.

5. The use of these interventions for the treatment of spine-related disorders is consistent with guidelines and is supported by high quality scientific evidence, including multiple systematic reviews undertaken by the prestigious Cochrane collaboration15, 16, 17.

6. The Global Burden of Disease study shows that spinal disorders are the leading cause of years lived with disability worldwide, exceeding depression, breast cancer and diabetes.

7. Interventions used by chiropractors are recommended in the 2018 Low Back Pain series of articles published in The Lancet and clinical practice guidelines from Denmark, Canada, the European Spine Journal, American College of Physicians and the Global Spine Care Initiative.

8. The AEQ supports and promotes scientific research, providing funding and resources for the development of high quality research in collaboration with institutions of high repute, such as Fundación Jiménez Díaz and the University of Alcalá de Henares.

9. The AEQ strenuously promotes among its members the practice of evidence-based, patient-centred care, consistent with a biopsychosocial model of health.

10. The AEQ demands the highest standards of practice and professional ethics, by implementing among its members the Quality Standard UNE-EN 16224 “Healthcare provision by chiropractors”, issued by the European Committee of Normalisation and ratified by AENOR.

11. The AEQ urges the Spanish Government to regulate chiropractic as a healthcare profession. Without such legislation, citizens of Spain cannot be assured that they are protected from unqualified practitioners and will continue to face legal uncertainties and barriers to access an essential, high-quality, evidence-based healthcare service.

END OF QUOTE

I think that some comments might be in order (they follow the numbering of the AEQ):

  1. The WHO is the last organisation I would consult for information on alternative medicine; during recent years, they have published mainly nonsense on this subject. How about asking the inventor of chiropractic? D.D. Palmer defined it as “a science of healing without drugs.” Chiropractors nowadays prefer to be defined as a profession which has the advantage that one cannot easily pin them down for doing mainly spinal manipulation; if one does, they indignantly respond “but we also use many other interventions, like life-style advice, for instance, and nobody can claim this to be nonsense” (see also point 4 below).
  2. Perfect use of a classical fallacy: appeal to authority.
  3. Appeal to authority, plus ignorance of the fact that teaching nonsense even at the highest level must result in nonsense.
  4. This is an ingenious mix of misleading arguments and lies: most chiros pride themselves of treating also non-spinal conditions. Very few interventions used by chiros are evidence-based. Exercise prescription, patient education and lifestyle advice are hardy typical for chiros and can all be obtained more authoratively from other healthcare professionals.
  5. Plenty of porkies here too. For instance, the AEQ cite three Cochrane reviews. The first concluded that high-quality evidence suggests that there is no clinically relevant difference between SMT and other interventions for reducing pain and improving function in patients with chronic low-back pain. The second stated that combined chiropractic interventions slightly improved pain and disability in the short term and pain in the medium term for acute/subacute LBP. However, there is currently no evidence that supports or refutes that these interventions provide a clinically meaningful difference for pain or disability in people with LBP when compared to other interventions. And the third concluded that, although support can be found for use of thoracic manipulation versus control for neck pain, function and QoL, results for cervical manipulation and mobilisation versus control are few and diverse. Publication bias cannot be ruled out. Research designed to protect against various biases is needed. Findings suggest that manipulation and mobilisation present similar results for every outcome at immediate/short/intermediate-term follow-up. Multiple cervical manipulation sessions may provide better pain relief and functional improvement than certain medications at immediate/intermediate/long-term follow-up. Since the risk of rare but serious adverse events for manipulation exists, further high-quality research focusing on mobilisation and comparing mobilisation or manipulation versus other treatment options is needed to guide clinicians in their optimal treatment choices. Hardly the positive endorsement implied by the AEQ!
  6. Yes, but that is not an argument for chiropractic; in fact, it’s another fallacy.
  7. Did they forget the many guidelines, institutions and articles that do NOT recommend chiropractic?
  8. I believe the cigarette industry also sponsors research; should we therefore all start smoking?
  9. I truly doubt that the AEQ strenuously promotes among its members the practice of evidence-based healthcare; if they did, they would have to discourage spinal manipulation!
  10. The ‘highest standards of practice and professional ethics’ are clearly not compatible with chiropractors’ use of spinal manipulation. In our recent book, we explained in full detail why this is so.
  11. An essential, high-quality, evidence-based healthcare service? Chiropractic is certainly not essential, rarely high-quality, and clearly not evidence-based.

Nice try AEQ.

But not good enough, I am afraid.

The primary objective of this paper was to assess the efficacy of homeopathy by systematically reviewing existing systematic reviews and meta-analyses and to systematically review trials on open-label placebo (OLP) treatments. A secondary objective was to understand whether homoeopathy as a whole may be considered as a placebo treatment. Electronic databases and previously published papers were systematically searched for systematic reviews and meta-analyses on homoeopathy efficacy. In total, 61 systematic reviews of homeopathy were included.

The same databases plus the Journal of Interdisciplinary Placebo Studies (JIPS) were also systematically searched for randomised controlled trials (RCTs) on OLP treatments, and 10 studies were included.

Qualitative syntheses showed that homoeopathy efficacy can be considered comparable to placebo. Twenty‐five reviews demonstrated that homoeopathy efficacy is comparable to placebo, 20 reviews did not come to a definite conclusion, and 16 reviews concluded that homoeopathy has some effect beyond placebo (in some cases of the latter category, authors  drew cautious conclusions, due to low methodological quality of included trials, high risk of bias and sparse data).

Qualitative syntheses also showed that OLP treatments may be effective in some health conditions.

The authors concluded that, if homoeopathy efficacy is comparable to placebo, and if placebo treatments can be effective in some conditions, then homoeopathy as a whole may be considered as a placebo treatment. Reinterpreting homoeopathy as a placebo treatment would define limits and possibilities of this practice. This perspective shift suggests a strategy to manage patients who seek homoeopathic care and to reconcile them with mainstream medicine in a sustainable way.

The authors also mention in their discussion section that  one of the most important work which concluded that homoeopathy has some effect beyond placebo is the meta‐analysis performed by Linde et al. (1997), which included 119 trials with 2,588 participants and aimed to assess the efficacy of homoeopathy for many conditions. Among these ones, there were conditions with various degrees of placebo responsiveness. This work was thoroughly re‐analysed by Linde himself and other authors (Ernst, 1998; Ernst & Pittler, 2000; Linde et al., 1999; Morrison et al., 2000; Sterne et al., 2001), who, selecting high‐quality extractable data and taking into consideration some methodological issues and biases of included trials (like publication bias and biases within studies), underscored that it cannot be demonstrated that homoeopathy has effects beyond placebo.

I agree with much of what the authors state. However, I fail to see that homeopathy should be used as an OLP treatment. I have several reasons for this, for instance:

  1. Placebo effects are unreliable and do occur only in some but not all patients.
  2. Placebo effects are usually of short duration.
  3. Placebo effects are rarely of a clinically relevant magnitude.
  4. The use of placebo, even when given as OLP, usually involves deception which is unethical.
  5. Placebos might replace effective treatments which would amount to neglect.
  6. One does not need a placebo for generating a placebo effect.

The idea that homeopathic remedies could be used in clinical practice as placebos to generate positive health outcomes is by no means new. I know that many doctors have used it that way. The idea that homeopathy could be employed as OLP, might be new, but it is neither practical, nor ethical, nor progressive.

Regardless of this particular debate, this new review confirms yet again:

HOMEOPATHY = PLACEBO THERAPY

This systematic review was aimed at evaluating the effects of acupuncture on the quality of life of migraineurs.  Only randomized controlled trials that were published in Chinese and English were included. In total, 62 trials were included for the final analysis; 50 trials were from China, 3 from Brazil, 3 from Germany, 2 from Italy and the rest came from Iran, Israel, Australia and Sweden.

Acupuncture resulted in lower Visual Analog Scale scores than medication at 1 month after treatment and 1-3 months after treatment. Compared with sham acupuncture, acupuncture resulted in lower Visual Analog Scale scores at 1 month after treatment.

The authors concluded that acupuncture exhibits certain efficacy both in the treatment and prevention of migraines, which is superior to no treatment, sham acupuncture and medication. Further, acupuncture enhanced the quality of life more than did medication.

The authors comment in the discussion section that the overall quality of the evidence for most outcomes was of low to moderate quality. Reasons for diminished quality consist of the following: no mentioned or inadequate allocation concealment, great probability of reporting bias, study heterogeneity, sub-standard sample size, and dropout without analysis.

Further worrisome deficits are that only 14 of the 62 studies reported adverse effects (this means that 48 RCTs violated research ethics!) and that there was a high level of publication bias indicating that negative studies had remained unpublished. However, the most serious concern is the fact that 50 of the 62 trials originated from China, in my view. As I have often pointed out, such studies have to be categorised as highly unreliable.

In view of this multitude of serious problems, I feel that the conclusions of this review must be re-formulated:

Despite the fact that many RCTs have been published, the effect of acupuncture on the quality of life of migraineurs remains unproven.

 

Often referred to as “Psychological acupressure”, the emotional freedom technique (EFT) works by releasing blockages within the energy system which are the source of emotional intensity and discomfort. These blockages in our energy system, in addition to challenging us emotionally, often lead to limiting beliefs and behaviours and an inability to live life harmoniously. Resulting symptoms are either emotional and/ or physical and include lack of confidence and self esteem, feeling stuck anxious or depressed, or the emergence of compulsive and addictive behaviours. It is also now finally widely accepted that emotional disharmony is a key factor in physical symptoms and dis-ease and for this reason these techniques are being extensively used on physical issues, including chronic illness with often astounding results. As such these techniques are being accepted more and more in medical and psychiatric circles as well as in the range of psychotherapies and healing disciplines.

An EFT treatment involves the use of fingertips rather than needles to tap on the end points of energy meridians that are situated just beneath the surface of the skin. The treatment is non-invasive and works on the ethos of making change as simple and as pain free as possible.

EFT is a common sense approach that draws its power from Eastern discoveries that have been around for over 5,000 years. In fact Albert Einstein also told us back in the 1920’s that everything (including our bodies) is composed of energy. These ideas have been largely ignored by Western Healing Practices and as they are unveiled in our current times, human process is reopening itself to the forgotten truth that everything is Energy and the potential that this offers us.

END OF QUOTE

If you ask me, this sounds as though EFT combines pseudo-psychological with acupuncture-BS.

But I may be wrong.

What does the evidence tell us?

A systematic review included 14 RCTs of EFT with a total of 658 patients.  The pre-post effect size for the EFT treatment group was 1.23 (95% confidence interval, 0.82-1.64; p < 0.001), whereas the effect size for combined controls was 0.41 (95% confidence interval, 0.17-0.67; p = 0.001). Emotional freedom technique treatment demonstrated a significant decrease in anxiety scores, even when accounting for the effect size of control treatment. However, there were too few data available comparing EFT to standard-of-care treatments such as cognitive behavioural therapy, and further research is needed to establish the relative efficacy of EFT to established protocols.  Meta-analyses indicate large effect sizes for posttraumatic stress disorder, depression, and anxiety; however, treatment effects may be due to components EFT shares with other therapies.

Another, more recent analysis reviewed whether EFTs acupressure component was an active ingredient. Six studies of adults with diagnosed or self-identified psychological or physical symptoms were compared (n = 403), and three (n = 102) were identified. Pretest vs. posttest EFT treatment showed a large effect size, Cohen’s d = 1.28 (95% confidence interval [CI], 0.56 to 2.00) and Hedges’ g = 1.25 (95% CI, 0.54 to 1.96). Acupressure groups demonstrated moderately stronger outcomes than controls, with weighted posttreatment effect sizes of d = -0.47 (95% CI, -0.94 to 0.0) and g = -0.45 (95% CI, -0.91 to 0.0). Meta-analysis indicated that the acupressure component was an active ingredient and outcomes were not due solely to placebo, nonspecific effects of any therapy, or non-acupressure components.

From these and other reviews, one could easily get the impression that my above-mentioned suspicion is erroneous and EFT is an effective therapy. But I still do have my doubts.

Why?

These reviews conveniently forget to mention that the primary studies tend to be of poor or even very poor quality. The most common flaws include tiny sample sizes, wrong statistical approach, lack of blinding, lack of control of placebo and other nonspecific effects. Reviews of such studies thus turn out to be a confirmation of the ‘rubbish in, rubbish out’ principle: any summary of flawed studies are likely to produce a flawed result.

Until I have good quality trials to convince me otherwise, EFT is in my view:

  1. implausible and
  2. not of proven effectiveness for any condition.

Aromatherapy usually involves the application of diluted essential (volatile) oils via a gentle massage of the body surface. The chemist Rene-Maurice Gattefosse (1881-1950) coined the term ‘aromatherapy’ after experiencing that lavender oil helped to cure a severe burn of his hand. In 1937, he published a book on the subject: Aromathérapie: Les Huiles Essentielles, Hormones Végétales. Later, the French surgeon Jean Valnet used essential oils to help heal soldiers’ wounds in World War II.

Aromatherapy is currently one of the most popular of all alternative therapies. The reason for its popularity seems simple: it is an agreeable, luxurious form of pampering. Whether it truly merits to be called a therapy is debatable.

The authors of this systematic review stated that they wanted to critically assess the effect of aromatherapy on the psychological symptoms as noted in the postmenopausal and elderly women. They conducted electronic literature searches and fount 4 trials that met their inclusion criteria. The findings demonstrated that aromatherapy massage significantly improves psychological symptoms in menopausal, elderly women as compared to controls. In one trial, aromatherapy massage was no more effective than the untreated group regarding their experience of symptoms such as nervousness.

The authors concluded that aromatherapy may be beneficial in attenuating the psychological symptoms that these women may experience, such as anxiety and depression, but it is not considered as an effective treatment to manage nervousness symptom among menopausal women. This finding should be observed in light of study limitations.

In the discussion section, the authors state that to the best of our knowledge, this is the first meta-analysis evaluating the effect of aromatherapy on the psychological symptoms. I believe that they might be mistaken. Here are two of my own papers (other researchers have published further reviews) on the subject:

  1. Aromatherapy is the therapeutic use of essential oil from herbs, flowers, and other plants. The aim of this overview was to provide an overview of systematic reviews evaluating the effectiveness of aromatherapy. We searched 12 electronic databases and our departmental files without restrictions of time or language. The methodological quality of all systematic reviews was evaluated independently by two authors. Of 201 potentially relevant publications, 10 met our inclusion criteria. Most of the systematic reviews were of poor methodological quality. The clinical subject areas were hypertension, depression, anxiety, pain relief, and dementia. For none of the conditions was the evidence convincing. Several SRs of aromatherapy have recently been published. Due to a number of caveats, the evidence is not sufficiently convincing that aromatherapy is an effective therapy for any condition.
  2. Aromatherapy is becoming increasingly popular; however there are few clear indications for its use. To systematically review the literature on aromatherapy in order to discover whether any clinical indication may be recommended for its use, computerised literature searches were performed to retrieve all randomised controlled trials of aromatherapy from the following databases: MEDLINE, EMBASE, British Nursing Index, CISCOM, and AMED. The methodological quality of the trials was assessed using the Jadad score. All trials were evaluated independently by both authors and data were extracted in a pre-defined, standardised fashion. Twelve trials were located: six of them had no independent replication; six related to the relaxing effects of aromatherapy combined with massage. These studies suggest that aromatherapy massage has a mild, transient anxiolytic effect. Based on a critical assessment of the six studies relating to relaxation, the effects of aromatherapy are probably not strong enough for it to be considered for the treatment of anxiety. The hypothesis that it is effective for any other indication is not supported by the findings of rigorous clinical trials.

Omitting previous research may be odd, but it is not a fatal flaw. What makes this review truly dismal is the fact that the authors fail to discuss the poor quality of the primary studies. They are of such deplorable rigor that one can really not draw any conclusion at all from them. I therefore find the conclusions of this new paper unacceptable and think that our statement (even though a few years old) is much more accurate: the evidence is not sufficiently convincing that aromatherapy is an effective therapy for any condition.

I remember reading this paper entitled ‘Comparison of acupuncture and other drugs for chronic constipation: A network meta-analysis’ when it first came out. I considered discussing it on my blog, but then decided against it for a range of reasons which I shall explain below. The abstract of the original meta-analysis is copied below:

The objective of this study was to compare the efficacy and side effects of acupuncture, sham acupuncture and drugs in the treatment of chronic constipation. Randomized controlled trials (RCTs) assessing the effects of acupuncture and drugs for chronic constipation were comprehensively retrieved from electronic databases (such as PubMed, Cochrane Library, Embase, CNKI, Wanfang Database, VIP Database and CBM) up to December 2017. Additional references were obtained from review articles. With quality evaluations and data extraction, a network meta-analysis (NMA) was performed using a random-effects model under a frequentist framework. A total of 40 studies (n = 11032) were included: 39 were high-quality studies and 1 was a low-quality study. NMA showed that (1) acupuncture improved the symptoms of chronic constipation more effectively than drugs; (2) the ranking of treatments in terms of efficacy in diarrhoea-predominant irritable bowel syndrome was acupuncture, polyethylene glycol, lactulose, linaclotide, lubiprostone, bisacodyl, prucalopride, sham acupuncture, tegaserod, and placebo; (3) the ranking of side effects were as follows: lactulose, lubiprostone, bisacodyl, polyethylene glycol, prucalopride, linaclotide, placebo and tegaserod; and (4) the most commonly used acupuncture point for chronic constipation was ST25. Acupuncture is more effective than drugs in improving chronic constipation and has the least side effects. In the future, large-scale randomized controlled trials are needed to prove this. Sham acupuncture may have curative effects that are greater than the placebo effect. In the future, it is necessary to perform high-quality studies to support this finding. Polyethylene glycol also has acceptable curative effects with fewer side effects than other drugs.

END OF 1st QUOTE

This meta-analysis has now been retracted. Here is what the journal editors have to say about the retraction:

After publication of this article [1], concerns were raised about the scientific validity of the meta-analysis and whether it provided a rigorous and accurate assessment of published clinical studies on the efficacy of acupuncture or drug-based interventions for improving chronic constipation. The PLOS ONE Editors re-assessed the article in collaboration with a member of our Editorial Board and noted several concerns including the following:

  • Acupuncture and related terms are not mentioned in the literature search terms, there are no listed inclusion or exclusion criteria related to acupuncture, and the outcome measures were not clearly defined in terms of reproducible clinical measures.
  • The study included acupuncture and electroacupuncture studies, though this was not clearly discussed or reported in the Title, Methods, or Results.
  • In the “Routine paired meta-analysis” section, both acupuncture and sham acupuncture groups were reported as showing improvement in symptoms compared with placebo. This finding and its implications for the conclusions of the article were not discussed clearly.
  • Several included studies did not meet the reported inclusion criteria requiring that studies use adult participants and assess treatments of >2 weeks in duration.
  • Data extraction errors were identified by comparing the dataset used in the meta-analysis (S1 Table) with details reported in the original research articles. Errors included aspects of the study design such as the experimental groups included in the study, the number of study arms in the trial, number of participants, and treatment duration. There are also several errors in the Reference list.
  • With regard to side effects, 22 out of 40 studies were noted as having reported side effects. It was not made clear whether side effects were assessed as outcome measures for the other 18 studies, i.e. did the authors collect data clarifying that there were no side effects or was this outcome measure not assessed or reported in the original article. Without this clarification the conclusion comparing side effect frequencies is not well supported.
  • The network geometry presented in Fig 5 is not correct and misrepresents some of the study designs, for example showing two-arm studies as three-arm studies.
  • The overall results of the meta-analysis are strongly reliant on the evidence comparing acupuncture versus lactulose treatment. Several of the trials that assessed this comparison were poorly reported, and the meta-analysis dataset pertaining to these trials contained data extraction errors. Furthermore, potential bias in studies assessing lactulose efficacy in acupuncture trials versus lactulose efficacy in other trials was not sufficiently addressed.

While some of the above issues could be addressed with additional clarifications and corrections to the text, the concerns about study inclusion, the accuracy with which the primary studies’ research designs and data were represented in the meta-analysis, and the reporting quality of included studies directly impact the validity and accuracy of the dataset underlying the meta-analysis. As a consequence, we consider that the overall conclusions of the study are not reliable. In light of these issues, the PLOS ONE Editors retract the article. We apologize that these issues were not adequately addressed during pre-publication peer review.

LZ disagreed with the retraction. YM and XD did not respond.

END OF 2nd QUOTE

Let me start by explaining why I initially decided not to discuss this paper on my blog. Already the first sentence of the abstract put me off, and an entire chorus of alarm-bells started ringing once I read further.

  • A meta-analysis is not a ‘study’ in my book, and I am somewhat weary of researchers who employ odd or unprecise language.
  • We all know (and I have discussed it repeatedly) that studies of acupuncture frequently fail to report adverse effects (in doing this, their authors violate research ethics!). So, how can it be a credible aim of a meta-analysis to compare side-effects in the absence of adequate reporting?
  • The methodology of a network meta-analysis is complex and I know not a lot about it.
  • Several things seemed ‘too good to be true’, for instance, the funnel-plot and the overall finding that acupuncture is the best of all therapeutic options.
  • Looking at the references, I quickly confirmed my suspicion that most of the primary studies were in Chinese.

In retrospect, I am glad I did not tackle the task of criticising this paper; I would probably have made not nearly such a good job of it as PLOS ONE eventually did. But it was only after someone raised concerns that the paper was re-reviewed and all the defects outlined above came to light.

While some of my concerns listed above may have been trivial, my last point is the one that troubles me a lot. As it also related to dozens of Cochrane reviews which currently come out of China, it is worth our attention, I think. The problem, as I see it, is as follows:

  • Chinese (acupuncture, TCM and perhaps also other) trials are almost invariably reporting positive findings, as we have discussed ad nauseam on this blog.
  • Data fabrication seems to be rife in China.
  • This means that there is good reason to be suspicious of such trials.
  • Many of the reviews that currently flood the literature are based predominantly on primary studies published in Chinese.
  • Unless one is able to read Chinese, there is no way of evaluating these papers.
  • Therefore reviewers of journal submissions tend to rely on what the Chinese review authors write about the primary studies.
  • As data fabrication seems to be rife in China, this trust might often not be justified.
  • At the same time, Chinese researchers are VERY keen to publish in top Western journals (this is considered a great boost to their career).
  • The consequence of all this is that reviews of this nature might be misleading, even if they are published in top journals.

I have been struggling with this problem for many years and have tried my best to alert people to it. However, it does not seem that my efforts had even the slightest success. The stream of such reviews has only increased and is now a true worry (at least for me). My suspicion – and I stress that it is merely that – is that, if one would rigorously re-evaluate these reviews, their majority would need to be retracted just as the above paper. That would mean that hundreds of papers would disappear because they are misleading, a thought that should give everyone interested in reliable evidence sleepless nights!

So, what can be done?

Personally, I now distrust all of these papers, but I admit, that is not a good, constructive solution. It would be better if Journal editors (including, of course, those at the Cochrane Collaboration) would allocate such submissions to reviewers who:

  • are demonstrably able to conduct a CRITICAL analysis of the paper in question,
  • can read Chinese,
  • have no conflicts of interest.

In the case of an acupuncture review, this would narrow it down to perhaps just a handful of experts worldwide. This probably means that my suggestion is simply not feasible.

But what other choice do we have?

One could oblige the authors of all submissions to include full and authorised English translations of non-English articles. I think this might work, but it is, of course, tedious and expensive. In view of the size of the problem (I estimate that there must be around 1 000 reviews out there to which the problem applies), I do not see a better solution.

(I would truly be thankful, if someone had a better one and would tell us)

1 2 3 6
Recent Comments

Note that comments can be edited for up to five minutes after they are first submitted but you must tick the box: “Save my name, email, and website in this browser for the next time I comment.”

The most recent comments from all posts can be seen here.

Archives
Categories