systematic review

1 2 3 12

Don’t get me wrong, I have nothing against systematic reviews. Quite to the contrary, I am sure they are an important source of information for patients, doctors, scientists, policy makers and others – after all, I have published more than 300 of such papers!

Having said that, I do dislike a certain type of systematic review, namely systematic reviews by Chinese authors evaluating TCM therapies and arriving at misleading conclusions. Such papers are currently swamping the marked.

At first glance, they look fine. On closer scrutiny, however, most turn out to be stereotypically useless, boring and promotional. The type of article I mean starts by stating its objective which usually is to evaluate the evidence for a traditional Chinese therapy as a treatment of a condition which few people in their right mind would treat with any form of TCM. It continues with details about the methodologies employed and then, in the results section, informs the reader that x studies were included in the review which mostly reported encouraging results but were wide open to bias. And then comes the crucial bit: THE CONCLUSIONS.

They are as predictable as they are misleading. let me give you two examples only published in the last few days.

The first review drew the following conclusions: This systematic review suggests that Chinese Herbal Medicine as an adjunctive therapy can improve cognitive impairment and enhance immediate response and quality of life in Senile Vascular Dementia patients. However, because of limitations of methodological quality in the included studies, further research of rigorous design is needed.

The second review concluded that the evidence that external application of traditional Chinese medicine is an effective treatment for venous ulcers is encouraging, but not conclusive due to the low methodological quality of the RCTs. Therefore, more high-quality RCTs with larger sample sizes are required.

Why does that sort of thing frustrate me so much? Because it is utterly meaningless and potentially harmful:

  • I don’t know what treatments the authors are talking about.
  • Even if I managed to dig deeper, I cannot get the information because practically all the primary studies are published in obscure journals in Chinese language.
  • Even if I  did read Chinese, I do not feel motivated to assess the primary studies because we know they are all of very poor quality – too flimsy to bother.
  • Even if they were formally of good quality, I would have my doubts about their reliability; remember: 100% of these trials report positive findings!
  • Most crucially, I am frustrated because conclusions of this nature are deeply misleading and potentially harmful. They give the impression that there might be ‘something in it’, and that it (whatever ‘it’ might be) could be well worth trying. This may give false hope to patients and can send the rest of us on a wild goose chase.

So, to ease the task of future authors of such papers, I decided give them a text for a proper EVIDENCE-BASED conclusion which they can adapt to fit every review. This will save them time and, more importantly perhaps, it will save everyone who might be tempted to read such futile articles the effort to study them in detail. Here is my suggestion for a conclusion soundly based on the evidence, not matter what TCM subject the review is about:


The search for an effective treatment of obesity is understandably intense. Many scientists are looking in the plant kingdom for a solution, but so far none has been forthcoming – as we have already discussed on this blog before (e. g. here, and here). One herbal slimming aid is currently becoming popular: Yerba Mate also called Ilex paraguariensis, a plant many of us know from teas and other beverages. Our review concluded that the evidence for it was unconvincing but that it merited further study. This was 10 years ago, and meanwhile the evidence has moved on.

The aim of a recent study was to investigate the efficacy of Yerba Mate supplementation in subjects with obesity. For this purpose, a randomized, double-blind, placebo-controlled trial was conducted. Korean subjects with obesity (body mass index (BMI) ≥ 25 but < 35 kg/m(2) and waist-hip ratio (WHR) ≥ 0.90 for men and ≥ 0.85 for women) were given oral supplements of Yerba Mate capsules (n = 15) or placebos (n = 15) for 12 weeks. They took three capsules per each meal, total three times in a day (3 g/day). Outcome measures were efficacy (abdominal fat distribution, anthropometric parameters and blood lipid profiles) and safety (adverse events, laboratory test results and vital signs).

During 12 weeks of Yerba Mate supplementation, statistically significant decreases in body fat mass and percent body fat compared to the placebo group were noted significant. The WHR was significantly also decreased in the Yerba Mate group compared to the placebo group. No clinically significant changes in any safety parameters were observed.

The authors concluded that Yerba Mate supplementation decreased body fat mass, percent body fat and WHR. Yerba Mate was a potent anti-obesity reagent that did not produce significant adverse effects. These results suggested that Yerba Mate supplementation may be effective for treating obese individuals.

These are encouraging results, but the conclusions go way too far, for my taste. The study was tiny and does therefore not lend itself to far-reaching generalisations. What would be helpful, is a review of other evidence. As it happens, such a paper has just become available. Its authors evaluated the impact of yerba maté on obesity and obesity-related inflammation and demonstrate that yerba maté suppresses adipocyte differentiation as well as triglyceride accumulation and reduces inflammation. Animal studies show that yerba maté modulates signaling pathways that regulate adipogenesis, antioxidant, anti-inflammatory and insulin signaling responses.

The review authors concluded that the use of yerba maté might be useful against obesity, improving the lipid parameters in humans and animal models. In addition, yerba maté modulates the expression of genes that are changed in the obese state and restores them to more normal levels of expression. In doing so, it addresses several of the abnormal and disease-causing factors associated with obesity. Protective and ameliorative effects on insulin resistance were also observed… it seems that yerba maté beverages and supplements might be helpful in the battle against obesity.

I am still not fully convinced that this dietary supplement is the solution to the current obesity epidemic. But the evidence is encouraging – more so than for most of the many other ‘natural’ slimming aids that are presently being promoted for this condition by gurus like Dr Oz.

What we needed now is not the ill-informed, self-interested voice of charlatans; what we need is well-designed research to define efficacy, effect size and risks.

Recently an interesting article caught my eye. It was published in the official journal of the ‘Deutscher Zentralverein Homoeopathischer Aerzte’ (the professional body of German doctor homeopath which mostly acts as a lobby group). Unfortunately it is in German – but I will try to take you through what I believe to be the most important issue.

The article seems to have the aim to defame Natalie Grams, the homeopath who had the courage to change her mind about homeopathy and to even write a book about her transformation. This book impressed me so much that I wrote a post about it when it was first published. The book did, however, not impress her ex-colleagues. Consequently the book review by the German lobbyists is full of personal attacks and almost devoid of credible facts.

A central claim of the defamatory piece is that, contrary to what she claims in her book, homeopathy is supported by sound evidence. Here is the crucial quote: Meta-Analysen von Kleijnen (1991), Linde (1997), Cucherat (2000) und Mathie (2014) [liefern] allesamt positive Ergebnisse zur Wirksamkeit der Homöopathie… This translates as follows: meta-analyses of Kleijnen, Linde, Cucherat and Matie all provide positive results regarding the effectiveness of homeopathy. As this is a claim, we hear ad nauseam whenever we discuss the issue with homeopathy (in the UK, most homeopathic bodies and even the Queen’s homeopath, P Fisher, have issued very similar statements), it may be worth addressing it once and for all.


This paper was the result of an EU-funded project in which I was involved as well; I therefore know about it first hand. The meta-analysis itself is quite odd in that it simply averages the p-values of all the included studies and thus provides a new overall p-value across all trials. As far as I know, this is not an accepted meta-analytic method and seems rather a lazy way of doing the job. The man on our EU committee was its senior author, professor Boissel, who did certainly not present it to us as a positive result for homeopathy (even Peter Fisher who also was a panel member should be able to confirm this). What is more, the published conclusions are not nearly as positive as out lobbyists seem to think: ‘There is some evidence that homeopathic treatments are more effective than placebo; however, the strength of this evidence is low because of the low methodological quality of the trials. Studies of high methodological quality were more likely to be negative than the lower quality studies. Further high quality studies are needed to confirm these results.’

Anybody who claims this is a proof for homeopathy’s efficacy should be sent back to school to learn how to read and understand English, in my view.

The meta-analysis by Linde et al seems to be the flag-ship in the homeopathic fleet. For those who don’t know it, here is its abstract in full:

BACKGROUND: Homeopathy seems scientifically implausible, but has widespread use. We aimed to assess whether the clinical effect reported in randomised controlled trials of homeopathic remedies is equivalent to that reported for placebo.

METHODS: We sought studies from computerised bibliographies and contracts with researchers, institutions, manufacturers, individual collectors, homeopathic conference proceedings, and books. We included all languages. Double-blind and/or randomised placebo-controlled trials of clinical conditions were considered. Our review of 185 trials identified 119 that met the inclusion criteria. 89 had adequate data for meta-analysis, and two sets of trial were used to assess reproducibility. Two reviewers assessed study quality with two scales and extracted data for information on clinical condition, homeopathy type, dilution, “remedy”, population, and outcomes.

FINDINGS: The combined odds ratio for the 89 studies entered into the main meta-analysis was 2.45 (95% CI 2.05, 2.93) in favour of homeopathy. The odds ratio for the 26 good-quality studies was 1.66 (1.33, 2.08), and that corrected for publication bias was 1.78 (1.03, 3.10). Four studies on the effects of a single remedy on seasonal allergies had a pooled odds ratio for ocular symptoms at 4 weeks of 2.03 (1.51, 2.74). Five studies on postoperative ileus had a pooled mean effect-size-difference of -0.22 standard deviations (95% CI -0.36, -0.09) for flatus, and -0.18 SDs (-0.33, -0.03) for stool (both p < 0.05).

INTERPRETATION: The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo. However, we found insufficient evidence from these studies that homeopathy is clearly efficacious for any single clinical condition. Further research on homeopathy is warranted provided it is rigorous and systematic.

Again, the conclusions are not nearly as strongly in favour of homeopaths as the German lobby group assumes. Moreover, this paper has been extensively criticised for a wide range of reasons which I shall not have to repeat here. However, one point is often over-looked: this is not an assessment of RCTs, it is an analysis of studies which were double-blind and/or randomised and placebo-controlled. This means that it includes trials that were not randomised and studies that were not double-blind.

But this is just by the way. What seems much more important is the fact that, in response to the plethora of criticism to their article, the same authors published a re-analysis of exactly the same data-set two years later. Having considered the caveats and limitations more carefully, they now concluded that ‘in the study set investigated, there was clear evidence that studies with better methodological quality tended to yield less positive results.’

It is most intriguing to see how homeopaths cite their ‘flagship’ on virtually every possible occasion, while forgetting that a quasi correction has been published which puts the prior conclusions in a very different light !


The much-cited article by Kleijnen is now far too old to be truly relevant. It includes not even half of the trials available today. But, for what it’s worth, here are Kleijnen’s conclusions: At the moment the evidence of clinical trials is positive but not sufficient to draw definitive conclusions because most trials are of low methodological quality and because of the unknown role of publication bias. This indicates that there is a legitimate case for further evaluation of homoeopathy, but only by means of well performed trials.

If the homeopathy lobby today proclaims that this paper constitutes proof of efficacy, they are in my view deliberately misleading the public.


The Mathie meta-analysis has been extensively discussed on this blog (see here and here). It is not an overall meta-analysis but merely evaluates the subset of those trials that employed individualised homeopathy. Crucially, it omits the two most rigorous studies which happen to be negative. Its conclusions are as follows: ‘Medicines prescribed in individualised homeopathy may have small, specific treatment effects. Findings are consistent with sub-group data available in a previous ‘global’ systematic review. The low or unclear overall quality of the evidence prompts caution in interpreting the findings. New high-quality RCT research is necessary to enable more decisive interpretation.’

Again, I would suggest that anyone who interprets this as stating that this provides ‘positive results regarding the effectiveness of homeopathy’ is not telling the truth.


  1.  Some systematic reviews and meta-analyses do indeed suggest that the trial data are positive. However, they all caution that such a result might be false-positive.
  2. None of these papers provide anything near a proof for the effectiveness of homeopathy.
  3. Homeopathy has not been shown to be more than a placebo therapy.
  4. To issue statements to the contrary is dishonest.

If you start reading the literature on chiropractic, you are bound to have surprises. The paucity of rigorous and meaningful research is one of them. I am constantly on the look-out for such papers but am regularly frustrated. Over the years, I got the impression that chiropractors tend to view research as an exercise in promotion – that is promotion of their very own trade.

Take this article, for instance. It seems to be a systematic review of chiropractic for breastfeeding. This is an interesting indication; remember: in 1998, Simon Singh wrote in the Guardian this comment “The British Chiropractic Association claims that their members can help treat children with colic, sleeping and feeding problems, frequent ear infections, asthma and prolonged crying, even though there is not a jot of evidence. This organisation is the respectable face of the chiropractic profession and yet it happily promotes bogus treatments.” As a consequence, he got sued for libel; he won, of course, but ever since, chiropractors across the world are trying to pretend that there is some evidence for their treatments after all.

The authors of the new review searched Pubmed [1966-2013], Manual, Alternative and Natural Therapy Index System (MANTIS) [1964-2013] and Index to Chiropractic Literature [1984-2013] for the relevant literature. The search terms utilized “breastfeeding”, “breast feeding”, “breastfeeding difficulties”, “breastfeeding difficulty”, “TMJ dysfunction”, “temporomandibular joint”, “birth trauma” and “infants”, in the appropriate Boolean combinations. They also examined non-peer-reviewed articles as revealed by Index to Chiropractic Literature and conducted a secondary analysis of references. Inclusion criteria for their review included all papers on breastfeeding difficulties regardless of peer-review. Articles were excluded if they were not written in the English language.

The following articles met the inclusion criteria: 8 case reports, 2 case series, 3 cohort studies and 6 manuscripts (5 case reports and a case series) that involved breastfeeding difficulties as a secondary complaint. The findings revealed a “theoretical and clinical framework based on the detection of spinal and extraspinal subluxations involving the cervico-cranio-mandibular complex and assessment of the infant while breastfeeding.”

Based on these results, the authors concluded that chiropractors care of infants with breastfeeding difficulties by addressing spinal and extraspinal subluxations involving the cervico-cranio-mandibular complex.

Have I promised too much?

I had thought that chiropractors had abandoned the subluxation nonsense! Not really, it seems.

I had thought that systematic reviews are about evidence of therapeutic effectiveness! Not in the weird world of chiropractic.

I would have thought that we all knew that ‘chiropractors care of infants with breastfeeding difficulties’ and do not need a review to confirm it! Yes, but what is good for business deserves another meaningless paper.

I would have thought that the conclusions of scientific articles need to be appropriate and based on the data provided! It seems that, in the realm of chiropractic, these rules do not apply.

An appropriate conclusion should have stated something like THERE IS NO GOOD EVIDENCE THAT CHIROPRACTIC CARE AIDS BREASTFEEDING. But that would have been entirely inappropriate from the chiropractic point of view because it is not a conclusion that promotes the sort of quackery most chiropractors rely upon for a living. And the concern over income is surely more important than telling the truth!

Much has been written on this blog and elsewhere about the risks of spinal manipulation. It relates almost exclusively to the risks of manipulating patients’ necks. There is far less on the safety of thrust joint manipulation (TJM) when applied to the thoracic spine. A new paper focusses on this specific topic.

The purpose of this review was to retrospectively analyse documented case reports in the literature describing patients who had experienced severe adverse events (AE) after receiving TJM to their thoracic spine.

Case reports published in peer reviewed journals were searched in Medline (using Ovid Technologies, Inc.), Science Direct, Web of Science, PEDro (Physiotherapy Evidence Database), Index of Chiropractic literature, AMED (Allied and Alternative Medicine Database), PubMed and the Cumulative Index to Nursing and Allied Health (CINHAL) from January 1950 to February 2015.

Case reports were included if they: (1) were peer-reviewed; (2) were published between 1950 and 2015; (3) provided case reports or case series; and (4) had TJM as an intervention. The authors only looked at serious complications, not at the much more frequent transient AEs after spinal manipulations. Articles were excluded if: (1) the AE occurred without TJM (e.g. spontaneous); (2) the article was a systematic or literature review; or (3) it was written in a language other than English or Spanish. Data extracted from each case report included: gender; age; who performed the TJM and why; presence of contraindications; the number of manipulation interventions performed; initial symptoms experienced after the TJM; as well as type of severe AE that resulted.

Ten cases, reported in 7 articles, were reviewed. Cases involved females (8) more than males (2), with mean age being 43.5 years. The most frequent AE reported was injury (mechanical or vascular) to the spinal cord (7/10); pneumothorax and hematothorax (2/10) and CSF leak secondary to dural sleeve injury (1/10) were also reported.

The authors point out that there were only a small number of case reports published in the literature and there may have been discrepancies between what was reported and what actually occurred, since physicians dealing with the effects of the AE, rather than the clinician performing the TJM, published the cases.

The authors concluded that serious AE do occur in the thoracic spine, most commonly, trauma to the spinal cord, followed by pneumothorax. This suggests that excessive peak forces may have been applied to thoracic spine, and it should serve as a cautionary note for clinicians to decrease these peak forces.

These are odd conclusions, in my view, and I think I ought to add a few points:

  • As I stated above, the actual rate of experiencing AEs after having chiropractic spinal manipulations is much larger; it is around 50%.
  • Most complications on record occur with chiropractors, while other professions are far less frequently implicated.
  • The authors’ statement about ‘excessive peak force’ is purely speculative and is therefore not a legitimate conclusion.
  • As the authors mention, it is  hardly ever the chiropractor who reports a serious complication when it occurs.
  • In fact, there is no functioning reporting scheme where the public might inform themselves about such complications.
  • Therefore their true rate is anyone’s guess.
  • As there is no good evidence that thoracic spinal manipulations are effective for any condition, the risk/benefit balance for this intervention fails to be positive.
  • Many consumers believe that a chiropractor will only manipulate in the region where they feel pain; this is not necessarily true – they will manipulate where they believe to diagnose ‘SUBLUXATIONS’, and that can be anywhere.
  • Finally, I would not call a review that excludes all languages other than English and Spanish ‘systematic’.


Being constantly on the look-out for new, good quality articles on alternative therapy which suggest that a treatment might actually work, I was excited to find not just one or two but four recent publications on an old favourite of mine: massage therapy.

The first paper described a study aimed to investigate the effect of whole body massage on the vital signs, Glasgow Coma Scale (GCS) scores and arterial blood gases (ABG) in trauma ICU patients.

In a randomized, double-blind trial, 108 trauma ICU patients received whole body massage or routine care only. The patients vital signs; systolic blood pressure (SBP), diastolic blood pressure (DBP), respiratory rate (RR), pulse rate (PR), Temperature (T), GCS score and ABG parameters were measured in both groups before the intervention and 1 hour and 3 hours after the intervention. The patient in experimental group received full body massage in 45 minute by a family member.

Significant differences were observed between experimental and control groups in SBP 1 hour and 3 hours after intervention, DBP, RR and PR 1 hour after intervention, and GCS 1 hour and 3 hours after intervention. Significant differences were also observed between experimental and control groups in O2 saturation, PH and pO2. No significant differences between experimental and control groups were noted in Temperature, pCO2 and HCO3.

The authors concluded that massage therapy is a safe and effective treatment in intensive care units to reduce patient’s physical and psychological problems. Therefore the use of massage therapy is recommended to clinical practice as a routine method.

The second paper reported a clinical trial on 66 male and female nurses working in intensive care units of Isfahan University of Medical Sciences, Iran.

Patients were randomly divided into experimental and control groups. The Occupational Stress Inventory (OSI) (Osipow and Spokane, 1987) was completed by participants of the two groups before, immediately after, and 2 weeks after the intervention. Swedish massage was performed on participants of the experimental group for 25 min in each session, twice a week for 4 weeks.

Results showed a significant difference in favour of the massage therapy in overall mean occupation stress scores between experimental and control groups two weeks after the intervention.

The authors concluded that it is recommended that massage, as a valuable noninvasive method, be used for nurses in intensive care units to reduce their stress, promote mental health, and prevent the decrease in quality of nursing work life.

The third paper described a randomized controlled trial evaluating the effects of post-operative massage in patients undergoing abdominal colorectal surgery.

One hundred twenty-seven patients were randomized to receive a 20-min massage or social visit and relaxation session on postoperative days 2 and 3. Vital signs and psychological well-being (pain, tension, anxiety, satisfaction with care, relaxation) were assessed before and after each intervention.

Post-operative massage significantly improved the patients’ perception of pain, tension, and anxiety, but overall satisfaction was unchanged.

The authors concluded that massage may be beneficial during postoperative recovery for patients undergoing abdominal colorectal surgery. Further studies are warranted to optimize timing and duration and to determine other benefits in this clinical setting.

The fourth paper reported a systematic review was to evaluate the effectiveness of massage on the short- and long-term outcomes of pre-term infants.

Literature searches were conducted using the PRISMA framework. Validity of included studies was assessed using criteria defined by the Cochrane Collaboration. Assessments were carried out independently by two reviewers with a third reviewer to resolve differences.

Thirty-four studies met the inclusion criteria, 3 were quasi-experimental, 1 was a pilot study, and the remaining 30 were RCTs. The outcomes that could be used in the meta-analysis and found in more than three studies suggested that massage improved daily weight gain by 0.53 g, and resulted in a significant improvement in mental scores by 7.89 points. There were no significant effects on length of hospital stay, caloric intake, or weight at discharge. Other outcomes were not analyzed either because the units of measurement varied between studies, or because means and standard deviations were not provided by the authors. The quality of the studies was variable with methods of randomization and blinding of assessment unclear in 18 of the 34 trials.

The authors concluded that massage therapy could be a comforting measure for infants in the NICU to improve weight gain and enhance mental development. However, the high heterogeneity, the weak quality in some studies, and the lack of a scientific association between massage and developmental outcomes preclude making definite recommendations and highlight the need for further RCTs to contribute to the existing body of knowledge.

I am not saying that these articles are flawless, nor that I agree with all of their conclusion. What I am trying to indicate is that we finally have here an alternative therapy that is promising.


When I worked in Germany and later in Austria, massage was considered to be entirely mainstream. It was only after I had moved to the UK when I realised that, in English-speaking countries, it is mostly considered to be alternative. Perhaps this classification is wrong?

Perhaps we should differentiate according to what type of massage we are talking about. In the realm of alternative medicine – and not just there, I suppose – this seems good advice indeed.

The above papers are about classical massage therapy, but there are some types pf massage which are less than conventional: aura-massage, Marma massage, Indian head massage, shiatsu etc. etc. the list seems endless. These are alternative in more than one sense, and they have one thing in common: there is, as far as I can see, no good evidence to show that they do anything to human health.

My conclusion therefore is that, even with something as common as massage therapy, we need to be careful not to be roped in by the charlatans.

Regular readers of this blog will be aware of the many bogus claims made by chiropractors. One claim, however, namely the one postulating chiropractors can effectively treat low back pain with spinal manipulation, is rarely viewed as being bogus. Chiropractors are usually able to produce evidence that does suggest the claim to be true, and therefore even most critics of chiropractic back off on this particular issue.

But is the claim really true?

A recent trial might provide the answer.

The purpose of this study was to compare the effectiveness of chiropractic spinal manipulative therapy (cSMT) to a sham intervention on pain (Visual Analogue Scale, SF-36 pain subscale), disability (Oswestry Disability Index), and physical function (SF-36 subscale, Timed Up and Go) by performing a randomized placebo-controlled trial at 2 Veteran Affairs Clinics.

Older veterans (≥ 65 years of age) who were naive to chiropractic were recruited. A total of 136 who suffered from chronic low back pain (LBP) were included in the study – with 69 being randomly assigned to cSMT and 67 to the sham intervention. Patients were treated twice per week for 4 weeks. The outcomes were assessed at baseline, 5, and 12 weeks post baseline.

Both groups demonstrated significant decrease in pain and disability at 5 and 12 weeks. At 12 weeks, there was no significant difference in pain and a statistically significant decline in disability scores in the cSMT group when compared to the control group. There were no significant differences in adverse events between the groups.

The authors concluded that cSMT did not result in greater improvement in pain when compared to our sham intervention; however, cSMT did demonstrate a slightly greater improvement in disability at 12 weeks. The fact that patients in both groups showed improvements suggests the presence of a nonspecific therapeutic effect.

Hold on, I hear you say, this does not mean that cSMT is a placebo in the treatment of LBP! There are other studies that yield positive results. Let’s not cherry-pick our evidence!

Absolutely correct! To avoid cherry-picking, lets see what the current Cochrane review tells us about cSMT and chronic LBP. Here is the conclusion of this review based on 26 RCTs: High quality evidence suggests that there is no clinically relevant difference between SMT and other interventions for reducing pain and improving function in patients with chronic low-back pain.


While my last post was about the risk following some naturopaths’ advice, this one is about the effectiveness of naturopathic treatments. This is a complex subject, not least because naturopaths use a wide range of therapies (as the name implies, they pride themselves of employing all therapeutic means supplied by nature). Some of these interventions are clearly supported by good evidence; for instance, nobody would doubt the effectiveness of a healthy diet or the benefits of regular exercise. But what about all the other treatments naturopaths use? The best approach to find an answer might be to assess not each single therapy but to evaluate the entire package of the naturopathic approach, and not a single study but all such trials.

This is precisely what US researchers have recently done. The purpose of this interesting, new systematic review was to compile and consolidate research that has investigated the whole practice of naturopathic medicine as it is practiced in community settings in order to better assess the quantity and quality of the research, and clinical effect, if any.

In order to get included into the review, studies had to report results from multi-modal treatment delivered by North American naturopathic doctors. The effect size for each study was calculated; no meta-analysis was undertaken.

Fifteen studies met the authors’ inclusion criteria. They covered a wide range of chronic diseases. Most studies had low to medium risks of bias including acknowledged limitations of pragmatic trials. Effect sizes for the primary medical outcomes varied and were statistically significant in 10 out of 13 studies. A quality of life metric was included in all of the RCTs with medium effect size and statistical significance in some subscales.

The authors concluded that previous reports about the lack of evidence or benefit of naturopathic medicine (NM) are inaccurate; a small but compelling body of research exists. Further investigation is warranted into the effectiveness of whole practice NM across a range of health conditions.

This sounds like good news for naturopathy! However, there are several important caveats:

  • the authors seem to have only looked at US studies (naturopathy is a European tradition!),
  • the searches were done three years ago, and more recent data were thus omitted,
  • the authors included all sorts of investigations, even uncontrolled studies; only 6 were RCTs,
  • rigorous trials were very scarce; and for each condition, they were even more so,
  • the authors mention the PRISMA guidelines for systematic reviews implying that they followed them but, in fact, they did not.

My biggest concern, however, is something else. It relates to the interventions tested in these studies. The authors claim that their results table provides full details on this issue but this is unfortunately not true. All we have by way of an explanation is the authors’ remark that the interventions tested in the studies of their review included diet counseling and nutritional recommendations, specific home exercises and physical activity recommendations, deep breathing techniques or other stress reduction strategies, dietary supplements including vitamins, hydrotherapy, soft-tissue manual techniques, electrical muscle stimulation, and botanical medicines.

Survey data from two US states tell us that the most commonly prescribed naturopathic therapeutics are botanical medicines (51% of visits in Connecticut, 43% in Washington), vitamins (41% and 43%), minerals (35% and 39%), homeopathy (29% and 19%) and allergy treatments (11% and 13%). They also inform us that the mean length of a consultation with an US naturopath is about 40 minutes.

I think, this puts things into perspective. If I advise a patient with diabetes or hypertension or coronary heat disease to follow an appropriate diet, exercise and to adhere to some stress reduction program, if in addition I show empathy and compassion during a 40 minute consultation and make sure that my advise is taken seriously and subsequently adhered to, the outcome is likely to be positive. Naturopaths may elect to call this package of intervention ‘naturopathy’, however, I would call it good conventional medicine.

The problem, I think is clear: good therapeutic advice is effective but it is not naturopathy, and it cannot be used to justify the use of doubtful interventions like homeopathy or all sorts of dodgy supplements. Testing whole treatment packages of this nature can therefore lead to highly misleading results, particularly if the researchers draw unwarranted conclusions about specific schools of health care.

Of all alternative treatments, aromatherapy (i.e. the application of essential oils to the body, usually by gentle massage or simply inhalation) seems to be the most popular. This is perhaps understandable because it certainly is an agreeable form of ‘pampering’ for someone in need of come TLC. But is aromatherapy more than that? Is it truly a ‘THERAPY’?

A recent systematic review was aimed at evaluating the existing data on aromatherapy interventions as a means of improving the quality of sleep. Electronic literature searches were performed to identify relevant studies published between 2000 and August 2013. Randomized controlled and quasi-experimental trials that included aromatherapy for the improvement of sleep quality were considered for inclusion. Of the 245 publications identified, 13 studies met the inclusion criteria, and 12 studies could be used for a meta-analysis.

The meta-analysis of the 12 studies revealed that the use of aromatherapy was effective in improving sleep quality. Subgroup analysis showed that inhalation aromatherapy was more effective than aromatherapy applied via massage.

The authors concluded that readily available aromatherapy treatments appear to be effective and promote sleep. Thus, it is essential to develop specific guidelines for the efficient use of aromatherapy.

Perfect! Let’s all rush out and get some essential oils for inhalation to improve our sleep (remarkably, the results imply that aroma therapists are redundant!).

Not so fast! As I see it, there are several important caveats we might want to consider before spending our money this way:

  1. Why did this review focus on such a small time-frame? (Systematic reviews should include all the available evidence of a pre-defined quality.)
  2. The quality of the included studies was often very poor, and therefore the overall conclusion cannot be definitive.
  3. The effect size of armoatherapy is small. In 2000, we published a similar review and concluded that aromatherapy has a mild, transient anxiolytic effect. Based on a critical assessment of the six studies relating to relaxation, the effects of aromatherapy are probably not strong enough for it to be considered for the treatment of anxiety. The hypothesis that it is effective for any other indication is not supported by the findings of rigorous clinical trials.
  4. It seems uncertain which essential oil is best suited for this indication.
  5. Aromatherapy is not always entirely free of risks. Another of our reviews showed that aromatherapy has the potential to cause adverse effects some of which are serious. Their frequency remains unknown. Lack of sufficiently convincing evidence regarding the effectiveness of aromatherapy combined with its potential to cause adverse effects questions the usefulness of this modality in any condition.
  6. There are several effective ways for improving sleep when needed; we need to know how aromatherapy compares to established treatments for that indication.

All in all, I think stronger evidence is required that aromatherapy is more that pampering.


The press officers of journals like to send out press-releases of articles which are deemed to be particularly good and important. Sadly, it is not often that articles on alternative medicine fulfil these criteria. I was therefore excited to receive this press-release which seemed encouraging, to say the least:

Medical evidence supports the potential for acupuncture to be significantly more effective in the treatment of dermatologic conditions such as dermatitis, pruritus, and urticaria than alternative treatment options, “placebo acupuncture,” or no treatment, according to a review of the medical literature published in The Journal of Alternative and Complementary Medicine, a peer-reviewed publication from Mary Ann Liebert, Inc., publishers

The abstract was equally promising:

Objectives: Acupuncture is a form of Traditional Chinese Medicine that has been used to treat a broad range of medical conditions, including dermatologic disorders. This systematic review aims to synthesize the evidence on the use of acupuncture as a primary treatment modality for dermatologic conditions.

Methods: A systematic search of MEDLINE, EMBASE, and the Cochrane Central Register was performed. Studies were limited to clinical trials, controlled studies, case reports, comparative studies, and systematic reviews published in the English language. Studies involving moxibustion, electroacupuncture, or blood-letting were excluded.

Results: Twenty-four studies met inclusion criteria. Among these, 16 were randomized controlled trials, 6 were prospective observational studies, and 2 were case reports. Acupuncture was used to treat atopic dermatitis, urticaria, pruritus, acne, chloasma, neurodermatitis, dermatitis herpetiformis, hyperhidrosis, human papillomavirus wart, breast inflammation, and facial elasticity. In 17 of 24 studies, acupuncture showed statistically significant improvements in outcome measurements compared with placebo acupuncture, alternative treatment options, and no intervention.

Conclusions: Acupuncture improves outcome measures in the treatment of dermatitis, chloasma, pruritus, urticaria, hyperhidrosis, and facial elasticity. Future studies should ideally be double-blinded and standardize the control intervention.

One has to read the actual full text article to understand that the evidence presented here is dodgy to the extreme. In fact, one has to go into the tedious details of the methods section to find the reasons why:  All searches were limited to clinical trials, controlled studies, case reports, comparative studies, and systematic reviews published in the English language.

There are many more weaknesses of this review, but the inclusion of uncontrolled studies and even anecdotes is, in my view, a virtual death sentence to its credibility. It means that no general conclusions about the effectiveness of acupuncture, such as the authors have decided to make, are possible.

Such overt exaggerations are sadly no rarities in the realm of alternative medicine.  I think, this begs a number of serious questions:

  1. Does this cross the line between flawed research and scientific misconduct?
  2. Why did the reviewers not pick up these flaws?
  3. Why did the editor pass this article for publication?
  4. How can the publisher tolerate such dubious behaviour?
  5. Should this journal (which I have commented on before here and which is one with the highest impact factor of all the alt med journals) be de-listed from Medline?

I don’t think that we will get answers from the people responsible for this disgrace, but I would like to learn my readers’ opinions.

1 2 3 12
Recent Comments

Note that comments can now be edited for up to five minutes after they are first submitted.

Click here for a comprehensive list of recent comments.