MD, PhD, FMedSci, FRSB, FRCP, FRCPEd.

study design

Medline is the biggest electronic databank for articles published in medicine and related fields. It is therefore the most important source of information in this area. I use it regularly to monitor what new papers have been published in the various fields of alternative medicine.

As the number of Medline-listed papers dated 2018 on homeopathy has just reached 100, I thought it might be the moment to run a quick analysis on this material. The first thing to note is that it took until August for 100 articles dated 2018 to emerge. To explain how embarrassing this is, we need a few comparative figures. At the same moment (6/9/18), we have, for instance:

  • 126576 articles for surgery
  • 5001 articles or physiotherapy
  • 30215 articles for psychiatry
  • 60161 articles for pharmacology

Even compared to other types of alternative medicine, homeopathy is being dwarfed. Currently the figures are, for instance:

  • 2232 for herbal medicine
  • 1949 for dietary supplements
  • 1222 for acupuncture

This does not look as though homeopathy is a frightfully active area of research, if I may say so. Looking at the type of articles (yes, I did look at all the 100 papers and categorised them the best I could) published in homeopathy, things get even worse:

  • 29 were comments, letters, editorials, etc.
  • 16 were basic and pre-clinical papers,
  • 12 were non-systematic reviews,
  • 10 were surveys,
  • 7 were case-reports,
  • 5 were pilot or feasibility studies,
  • 5 were systematic reviews,
  • 5 were controlled clinical trials,
  • 2 were case series,
  • the rest of the articles was not on homeopathy at all.

I find this pretty depressing. Most of the 100 papers turn out to be no real research at all. Crucial topics are not being covered. There was, for example, not a single paper on the risks of homeopathy (no, don’t tell me it is harmless; it can and does regularly cost the lives of patients who trust the bogus claims of homeopaths). There was no article investigating the important question whether the practice of homeopathy does not violate the rules of medical ethics (think of informed consent or the imperative to do more harm than good). And a mere 5 clinical trials is just a dismal amount, in my view.

In a previous post, I have already shown that, in 2015, homeopathy research was deplorable. My new analysis suggests that the situation has become much worse. One might even go as far as asking whether 2018 might turn out to be the year when homeopathy research finally died a natural death.

PROGRESS AT LAST!!!

This systematic review included 18 studies assessing homeopathy in depression. Two double-blind placebo-controlled trials of homeopathic medicinal products (HMPs) for depression were assessed. The first trial (N = 91) with high risk of bias found HMPs were non-inferior to fluoxetine at 4 (p = 0.654) and 8 weeks (p = 0.965); whereas the second trial (N = 133), with low risk of bias, found HMPs was comparable to fluoxetine (p = 0.082) and superior to placebo (p < 0.005) at 6 weeks.

The remaining research had unclear/high risk of bias. A non-placebo-controlled RCT found standardised treatment by homeopaths comparable to fluvoxamine; a cohort study of patients receiving treatment provided by GPs practising homeopathy reported significantly lower consumption of psychotropic drugs and improved depression; and patient-reported outcomes showed at least moderate improvement in 10 of 12 uncontrolled studies. Fourteen trials provided safety data. All adverse events were mild or moderate, and transient. No evidence suggested treatment was unsafe.

The authors concluded that limited evidence from two placebo-controlled double-blinded trials suggests HMPs might be comparable to antidepressants and superior to placebo in depression, and patients treated by homeopaths report improvement in depression. Overall, the evidence gives a potentially promising risk benefit ratio. There is a need for additional high quality studies.

It is worth having a look at these two studies, I think.

The 1st (2011) study is from Brazil

Here is its abstract:

Homeopathy is a complementary and integrative medicine used in depression, The aim of this study is to investigate the non-inferiority and tolerability of individualized homeopathic medicines [Quinquagintamillesmial (Q-potencies)] in acute depression, using fluoxetine as active control. Ninety-one outpatients with moderate to severe depression were assigned to receive an individualized homeopathic medicine or fluoxetine 20 mg day−1 (up to 40 mg day−1) in a prospective, randomized, double-blind double-dummy 8-week, single-center trial. Primary efficacy measure was the analysis of the mean change in the Montgomery & Åsberg Depression Rating Scale (MADRS) depression scores, using a non-inferiority test with margin of 1.45. Secondary efficacy outcomes were response and remission rates. Tolerability was assessed with the side effect rating scale of the Scandinavian Society of Psychopharmacology. Mean MADRS scores differences were not significant at the 4th (P = .654) and 8th weeks (P = .965) of treatment. Non-inferiority of homeopathy was indicated because the upper limit of the confidence interval (CI) for mean difference in MADRS change was less than the non-inferiority margin: mean differences (homeopathy-fluoxetine) were −3.04 (95% CI −6.95, 0.86) and −2.4 (95% CI −6.05, 0.77) at 4th and 8th week, respectively. There were no significant differences between the percentages of response or remission rates in both groups. Tolerability: there were no significant differences between the side effects rates, although a higher percentage of patients treated with fluoxetine reported troublesome side effects and there was a trend toward greater treatment interruption for adverse effects in the fluoxetine group. This study illustrates the feasibility of randomized controlled double-blind trials of homeopathy in depression and indicates the non-inferiority of individualized homeopathic Q-potencies as compared to fluoxetine in acute treatment of outpatients with moderate to severe depression.

There are many important points to make about this trial:

  1. Contrary to what the reviewers claim, the trial had no placebo group.
  2. It was a double-dummy equivalence study comparing individualised homeopathy with the antidepressant fluoxetine.
  3. Fluoxetine might have been under-dosed (see below).
  4. Equivalence studies require large sample sizes, and with just 91 patients (only 55 of whom finished the study), this trial was underpowered which means the finding of equivalence is false positive.
  5. The authors noted that a higher percentage of troublesome adverse effects reported by patients receiving fluoxetine. This means that the trial was not double-blind; patients were able to tell by their side-effects which group they were in.
  6. The authors also state that more patients randomized to homeopathy than to fluoxetine were excluded due to worsening of their depressive symptoms. I think this confirms that homeopathy was ineffective.

The 2nd (2015) study is from Mexico

Here is its abstract:

Background: Perimenopausal period refers to the interval when women’s menstrual cycles become irregular and is characterized by an increased risk of depression. Use of homeopathy to treat depression is widespread but there is a lack of clinical trials about its efficacy in depression in peri- and postmenopausal women. The aim of this study was to assess efficacy and safety of individualized homeopathic treatment versus placebo and fluoxetine versus placebo in peri- and postmenopausal women with moderate to severe depression.

Methods/Design: A randomized, placebo-controlled, double-blind, double-dummy, superiority, three-arm trial with a 6 week follow-up study was conducted. The study was performed in a public research hospital in Mexico City in the outpatient service of homeopathy. One hundred thirty-three peri- and postmenopausal women diagnosed with major depression according to DSM-IV (moderate to severe intensity) were included. The outcomes were: change in the mean total score among groups on the 17-item Hamilton Rating Scale for Depression, Beck Depression Inventory and Greene Scale, after 6 weeks of treatment, response and remission rates, and safety. Efficacy data were analyzed in the intention-to-treat population (ANOVA with Bonferroni post-hoc test).

Results: After a 6-week treatment, homeopathic group was more effective than placebo by 5 points in Hamilton Scale. Response rate was 54.5% and remission rate, 15.9%. There was a significant difference among groups in response rate definition only, but not in remission rate. Fluoxetine-placebo difference was 3.2 points. No differences were observed among groups in the Beck Depression Inventory. Homeopathic group was superior to placebo in Greene Climacteric Scale (8.6 points). Fluoxetine was not different from placebo in Greene Climacteric Scale.

Conclusion: Homeopathy and fluoxetine are effective and safe antidepressants for climacteric women. Homeopathy and fluoxetine were significantly different from placebo in response definition only. Homeopathy, but not fluoxetine, improves menopausal symptoms scored by Greene Climacteric Scale.

And here are my critical remarks about this trial:

  1. The aim of a small study like this cannot be to assess or draw conclusions about the safety of the interventions used; for this purpose, we need sample sizes that are at least one dimension bigger.
  2. Fluoxetine might have been under-dosed (see below).
  3. The blinding of patients might have been jeopardized by patients experiencing the specific side-effects of fluoxetine. The authors reported adverse effects in all three groups. However, the characteristic and most common side-effects of fluoxetine (such as hives, itching, skin rash, restlessness, inability to sit still) were not included.

________________________________________________

Usual Adult Dose for Depression

Immediate-release oral formulations:
Initial dose: 20 mg orally once a day in the morning, increased after several weeks if sufficient clinical improvement is not observed
Maintenance dose: 20 to 60 mg orally per day
Maximum dose: 80 mg orally per day

Delayed release oral capsules:
Initial dose: 90 mg orally once a week, commenced 7 days after the last daily dose of immediate-release fluoxetine 20 mg formulations.

_________________________________________________

Considering all this, I feel that the conclusions of the above review are far too optimistic and not justified. In fact, I find them misleading, dangerous, unethical and depressing.

Bacterial vaginosis is a common condition which is more than a triviality. It can have serious consequences including pelvic inflammatory disease, endometritis, postoperative vaginal cuff infections, preterm labor, premature rupture of membranes, and chorioamnionitis. Therefore, it is important to treat it adequately with antibiotics. But are there other options as well?

There are plenty of alternative or ‘natural’ treatments on offer. But do they work?

This trial was conducted on 127 women with bacterial vaginosis to compare a vaginal suppository of metronidazole with Forzejeh, a vaginal suppository of herbal Persian medicine combination of

  • Tribulus terrestris,
  • Myrtus commuis,
  • Foeniculum vulgare,
  • Tamarindus indica.

The patients (63 in metronidazole group and 64 in Forzejeh group) received the medications for 1 week. Their symptoms including the amount and odour of discharge and cervical pain were assessed using a questionnaire. Cervical inflammation and Amsel criteria (pH of vaginal discharge, whiff test, presence of clue cells and Gram staining) were investigated at the beginning of the study and 14 days after treatment.

The amount and odour of discharge, Amsel criteria, pelvic pain and cervical inflammation significantly decreased in Forzejeh and metronidazole groups (p = <.001). There was no statistically significant difference between the metronidazole and Fozejeh groups with respect to any of the clinical symptoms or the laboratory assessments.

The authors concluded that Forzejeh … has a therapeutic effect the same as metronidazole in bacterial vaginosis.

The plants in Fozejeh are assumed to have antimicrobial activities. Forzejeh has been used in folk medicine for many years but was only recently standardised. According to the authors, this study shows that the therapeutic effects of Forzejeh on bacterial vaginosis is similar to metronidazole.

Yet, I am far less convinced than these Iranian researchers. As this trial compared two active treatments, it was an equivalence study. As such, it requires a different statistical approach and a much larger sample size. The absence of a difference between the two groups is most likely due to the fact that the study was too small to show it.

If I am correct, the present investigation only demonstrates yet again that, with flawed study-designs, it is easy to produce false-positive results.

In one of his many comments, our friend Iqbal just linked to an article that unquestionably is interesting. Here is its abstract (the link also provides the full paper):

Objective: The objective was to assess the usefulness of homoeopathic genus epidemicus (Bryonia alba 30C) for the prevention of chikungunya during its epidemic outbreak in the state of Kerala, India.

Materials and Methods: A cluster- randomised, double- blind, placebo -controlled trial was conducted in Kerala for prevention of chikungunya during the epidemic outbreak in August-September 2007 in three panchayats of two districts. Bryonia alba 30C/placebo was randomly administered to 167 clusters (Bryonia alba 30C = 84 clusters; placebo = 83 clusters) out of which data of 158 clusters was analyzed (Bryonia alba 30C = 82 clusters; placebo = 76 clusters) . Healthy participants (absence of fever and arthralgia) were eligible for the study (Bryonia alba 30 C n = 19750; placebo n = 18479). Weekly follow-up was done for 35 days. Infection rate in the study groups was analysed and compared by use of cluster analysis.

Results: The findings showed that 2525 out of 19750 persons of Bryonia alba 30 C group suffered from chikungunya, compared to 2919 out of 18479 in placebo group. Cluster analysis showed significant difference between the two groups [rate ratio = 0.76 (95% CI 0.14 – 5.57), P value = 0.03]. The result reflects a 19.76% relative risk reduction by Bryonia alba 30C as compared to placebo.

Conclusion: Bryonia alba 30C as genus epidemicus was better than placebo in decreasing the incidence of chikungunya in Kerala. The efficacy of genus epidemicus needs to be replicated in different epidemic settings.

________________________________________________________________________________

I have often said the notion that homeopathy might prevent epidemics is purely based on observational data. Here I stand corrected. This is an RCT! What is more, it suggests that homeopathy might be effective. As this is an important claim, let me quickly post just 10 comments on this study. I will try to make this short (I only looked at it briefly), hoping that others complete my criticism where I missed important issues:

  1. The paper was published in THE INDIAN JOURNAL OF RESEARCH IN HOMEOPATHY. This is not a publication that could be called a top journal. If this study really shows something as revolutionarily new as its conclusions imply, one must wonder why it was published in an obscure and inaccessible journal.
  2. Several of its authors are homeopaths who unquestionably have an axe to grind, yet they do not declare any conflicts of interest.
  3. The abstract states that the trial was aimed at assessing the usefulness of Bryonia C30, while the paper itself states that it assessed its efficacy. The two are not the same, I think.
  4. The trial was conducted in 2007 and published only 7 years later; why the delay?
  5. The criteria for the main outcome measure were less than clear and had plenty of room for interpretation (“Any participant who suffered from fever and arthralgia (characteristic symptoms of chikungunya) during the follow-up period was considered as a case of chikungunya”).
  6. I fail to follow the logic of the sample size calculation provided by the authors and therefore believe that the trial was woefully underpowered.
  7. As a cluster RCT, its unit of assessment is the cluster. Yet the significant results seem to have been obtained by using single patients as the unit of assessment (“At the end of follow-ups it was observed that 12.78% (2525 out of 19750) healthy individuals, administered with Bryonia alba 30 C, were presented diagnosed as probable case of chikungunya, whereas it was 15.79% (2919 out of 18749) in the placebo group”).
  8. The p-value was set at 0.05. As we have often explained, this is far too low considering that the verum was a C30 dilution with zero prior probability.
  9. Nine clusters were not included in the analysis because of ‘non-compliance’. I doubt whether this was the correct way of dealing with this issue and think that an intention to treat analysis would have been better.
  10. This RCT was published 4 years ago. If true, its findings are nothing short of a sensation. Therefore, one would have expected that, by now, we would see several independent replications. The fact that this is not the case might mean that such RCTs were done but failed to confirm the findings above.

As I said, I would welcome others to have a look and tell us what they think about this potentially important study.

Music therapy is the use of music for therapeutic purposes. Several forms of music therapy exist. They can consist of a patient listening to live or recorded music, or of patients participating in performing music. Music therapy is usually employed to complement other treatments; it is never a curative or causal approach and mostly aimed at inducing relaxation and enhancing physical and emotional well-being, or at promoting motor and communication skills.

There is a paucity of rigorous studies assessing the effectiveness of music therapy for specific condition, not least due to methodological obstacles and funding issues. Several systematic reviews of clinical studies have nevertheless emerged and results are generally encouraging. As for hypertension, the evidence is contradictory whether passive listening to music works. One review concluded that Music may improve systolic blood pressure and should be considered as a component of care of hypertensive patients. And another review revealed a trend towards a decrease in blood pressure in hypertensive patients who received music interventions, but failed to establish a cause-effect relationship between music interventions and blood pressure reduction.

A new study might bring more clarity:

Its authors evaluated the effect of musical auditory stimulus associated with anti-hypertensive medication on heart rate (HR) autonomic control in hypertensive subjects. They included in this trial 37 well-controlled hypertensive patients designated for anti-hypertensive medication. Heart rate variability (HRV) was calculated from the HR monitor recordings of two different, randomly sorted protocols (control and music) on two separate days. Patients were examined in a resting condition 10 minutes before medication and 20 minutes, 40 minutes and 60 minutes after oral anti-hypertensive medications. Music was played throughout the 60 minutes after medication with the same intensity for all subjects in the music protocol.

The results showed analogous response of systolic and diastolic arterial pressure in both protocols. HR decreased 60 minutes after medication in the music protocol, while it remained unchanged in the control protocol. The effects of anti-hypertensive medication on SDNN (Standard deviation of all normal RR intervals), LF (low frequency, nu), HF (high frequency, nu) and alpha-1 scale were more intense in the music protocol. Blood pressure readings showed no significant differences between the two groups.

The authors concluded that musical auditory stimulus increased HR autonomic responses to anti-hypertensive medication in well-controlled hypertensive subjects.

So, there were some acute effects on HRV. But what is the clinical relevance of this effect? I am not sure, and the authors tell us little about this.

Crucially, there was no effect on blood pressure. But the study design might have been ill-suited for detecting one. I think that a much simpler trial with two parallel groups of untreated hypertensives would have been more efficient for this purpose.

As a music-lover, I would like to believe that music can be used therapeutically. Yet, for hypertensives, I find it difficult to see how this could work. Even if passive listening to music had an anti-hypertensive effect, could it be employed in clinical routine? I somehow doubt it; we can hear music for a while, but our daily activities would largely prohibit doing it for prolonged periods (and most likely it would become a nuisance after a while and would put our pressure up rather than down – think of the background music that bothers us in some shops, for instance). And how would it work when we sleep, a time during which blood pressure control can be vital?

As a music-lover, I would also argue that listening to music can be pleasantly relaxing – presumably, the anti-hypertensive effect observed in some trials relies on this effect. But surely, it can also have the opposite effect. If I strongly dislike a piece of music, I might increase my blood pressure. If a piece moves me deeply, it could easily do the same. It is probably only a certain type of music that induces relaxation; and, to make it even more complex, this type might differ from person to person.

So, is music therapy potentially a usable anti-hypertensive?

Somehow, I don’t think so!

Can I tempt you to run a little (hopefully instructive) thought-experiment with you? It is quite simple: I will tell you about the design of a clinical trial, and you will tell me what the likely outcome of this study would be.

Are you game?

Here we go:

_____________________________________________________________________________

Imagine we conduct a trial of acupuncture for persistent pain (any type of pain really). We want to find out whether acupuncture is more than a placebo when it comes to pain-control. Of course, we want our trial to look as rigorous as possible. So, we design it as a randomised, sham-controlled, partially-blinded study. To be really ‘cutting edge’, our study will not have two but three parallel groups:

1. Standard needle acupuncture administered according to a protocol recommended by a team of expert acupuncturists.

2. Minimally invasive sham-acupuncture employing shallow needle insertion using short needles at non-acupuncture points. Patients in groups 1 and 2 are blinded, i. e. they are not supposed to know whether they receive the sham or real acupuncture.

3. No treatment at all.

We apply the treatments for a sufficiently long time, say 12 weeks. Before we start, after 6 and 12 weeks, we measure our patients’ pain with a validated method. We use sound statistical methods to compare the outcomes between the three groups.

WHAT DO YOU THINK THE RESULT WOULD BE?

You are not sure?

Well, let me give you some hints:

Group 3 is not going to do very well; not only do they receive no therapy at all, but they are also disappointed to have ended up in this group as they joined the study in the hope to get acupuncture. Therefore, they will (claim to) feel a lot of pain.

Group 2 will be pleased to receive some treatment. However, during the course of the 6 weeks, they will get more and more suspicious. As they were told during the process of obtaining informed consent that the trial entails treating some patients with a sham/placebo, they are bound to ask themselves whether they ended up in this group. They will see the short needles and the shallow needling, and a percentage of patients from this group will doubtlessly suspect that they are getting the sham treatment. The doubters will not show a powerful placebo response. Therefore, the average pain scores in this group will decrease – but only a little.

Group 1 will also be pleased to receive some treatment. As the therapists cannot be blinded, they will do their best to meet the high expectations of their patients. Consequently, they will benefit fully from the placebo effect of the intervention and the pain score of this group will decrease significantly.

So, now we can surely predict the most likely result of this trial without even conducting it. Assuming that acupuncture is a placebo-therapy, as many people do, we now see that group 3 will suffer the most pain. In comparison, groups 1 and 2 will show better outcomes.

Of course, the main question is, how do groups 1 and 2 compare to each other? After all, we designed our sham-controlled trial in order to answer exactly this issue: is acupuncture more than a placebo? As pointed out above, some patients in group 2 would have become suspicious and therefore would not have experienced the full placebo-response. This means that, provided the sample sizes are sufficiently large, there should be a significant difference between these two groups favouring real acupuncture over sham. In other words, our trial will conclude that acupuncture is better than placebo, even if acupuncture is a placebo.

THANK YOU FOR DOING THIS THOUGHT EXPERIMENT WITH ME.

Now I can tell you that it has a very real basis. The leading medical journal, JAMA, just published such a study and, to make matters worse, the trial was even sponsored by one of the most prestigious funding agencies: the NIH.

Here is the abstract:

___________________________________________________________________________

Musculoskeletal symptoms are the most common adverse effects of aromatase inhibitors and often result in therapy discontinuation. Small studies suggest that acupuncture may decrease aromatase inhibitor-related joint symptoms.

Objective:

To determine the effect of acupuncture in reducing aromatase inhibitor-related joint pain.

Design, Setting, and Patients:

Randomized clinical trial conducted at 11 academic centers and clinical sites in the United States from March 2012 to February 2017 (final date of follow-up, September 5, 2017). Eligible patients were postmenopausal women with early-stage breast cancer who were taking an aromatase inhibitor and scored at least 3 on the Brief Pain Inventory Worst Pain (BPI-WP) item (score range, 0-10; higher scores indicate greater pain).

Interventions:

Patients were randomized 2:1:1 to the true acupuncture (n = 110), sham acupuncture (n = 59), or waitlist control (n = 57) group. True acupuncture and sham acupuncture protocols consisted of 12 acupuncture sessions over 6 weeks (2 sessions per week), followed by 1 session per week for 6 weeks. The waitlist control group did not receive any intervention. All participants were offered 10 acupuncture sessions to be used between weeks 24 and 52.

Main Outcomes and Measures:

The primary end point was the 6-week BPI-WP score. Mean 6-week BPI-WP scores were compared by study group using linear regression, adjusted for baseline pain and stratification factors (clinically meaningful difference specified as 2 points).

Results:

Among 226 randomized patients (mean [SD] age, 60.7 [8.6] years; 88% white; mean [SD] baseline BPI-WP score, 6.6 [1.5]), 206 (91.1%) completed the trial. From baseline to 6 weeks, the mean observed BPI-WP score decreased by 2.05 points (reduced pain) in the true acupuncture group, by 1.07 points in the sham acupuncture group, and by 0.99 points in the waitlist control group. The adjusted difference for true acupuncture vs sham acupuncture was 0.92 points (95% CI, 0.20-1.65; P = .01) and for true acupuncture vs waitlist control was 0.96 points (95% CI, 0.24-1.67; P = .01). Patients in the true acupuncture group experienced more grade 1 bruising compared with patients in the sham acupuncture group (47% vs 25%; P = .01).

Conclusions and Relevance:

Among postmenopausal women with early-stage breast cancer and aromatase inhibitor-related arthralgias, true acupuncture compared with sham acupuncture or with waitlist control resulted in a statistically significant reduction in joint pain at 6 weeks, although the observed improvement was of uncertain clinical importance.

__________________________________________________________________________

Do you see how easy it is to deceive (almost) everyone with a trial that looks rigorous to (almost) everyone?

My lesson from all this is as follows: whether consciously or unconsciously, SCAM-researchers often build into their trials more or less well-hidden little loopholes that ensure they generate a positive outcome. Thus even a placebo can appear to be effective. They are true masters of producing false-positive findings which later become part of a meta-analysis which is, of course, equally false-positive. It is a great shame, in my view, that even top journals (in the above case JAMA) and prestigious funders (in the above case the NIH) cannot (or want not to?) see behind this type of trickery.

“Non-reproducible single occurrences are of no significance to science”, this quote by Karl Popper often seems to get forgotten in medicine, particularly in alternative medicine. It indicates that findings have to be reproducible to be meaningful – if not, we cannot be sure that the outcome in question was caused by the treatment we applied.

This is thus a question of cause and effect.

The statistician Sir Austin Bradford Hill proposed in 1965 a set of 9 criteria to provide evidence of a relationship between a presumed cause and an observed effect while demonstrating the connection between cigarette smoking and lung cancer. One of his criteria is consistency or reproducibility: Consistent findings observed by different persons in different places with different samples strengthens the likelihood of an effect.

By mentioning ‘different persons’, Hill seems to also establish the concept of INDEPENDENT replication.

Let me try to explain this with an example from the world of SCAM.

  1. A homeopath feels that childhood diarrhoea could perhaps be treated with individualised homeopathic remedies.  She conducts a trial, finds a positive result and concludes that the statistically significant decrease in the duration of diarrhea in the treatment group suggests that homeopathic treatment might be useful in acute childhood diarrhea. Further study of this treatment deserves consideration.
  2. Unsurprisingly, this study is met with disbelieve by many experts. Some go as far as doubting its validity, and several letters to the editor appear expressing criticism. The homeopath is thus motivated to run another trial to prove her point. Its results are consistent with the finding from the previous study that individualized homeopathic treatment decreases the duration of diarrhea and number of stools in children with acute childhood diarrhea.
  3. We now have a replication of the original finding. Yet, for a range of reasons, sceptics are far from satisfied. The homeopath thus runs a further trial and publishes a meta-analysis of all there studies. The combined analysis shows a duration of diarrhoea of 3.3 days in the homeopathy group compared with 4.1 in the placebo group (P = 0.008). She thus concludes that the results from these studies confirm that individualized homeopathic treatment decreases the duration of acute childhood diarrhea and suggest that larger sample sizes be used in future homeopathic research to ensure adequate statistical power. Homeopathy should be considered for use as an adjunct to oral rehydration for this illness.

To most homeopaths it seems that this body of evidence from three replication is sound and solid. Consequently, they frequently cite these publications as a cast-iron proof of their assumption that individualised homeopathy is effective. Sceptics, however, are still not convinced.

Why?

The studies have been replicated alright, but what is missing is an INDEPENDENT replication.

To me this word implies two things:

  1. The results have to be reproduced by another research group that is unconnected to the one that conducted the three previous studies.
  2. That group needs to be independent from any bias that might get in the way of conducting a rigorous trial.

And why do I think this latter point is important?

Simply because I know from many years of experience that a researcher, who strongly believes in homeopathy or any other subject in question, will inadvertently introduce all sorts of biases into a study, even if its design is seemingly rigorous. In the end, these flaws will not necessarily show in the published article which means that the public will be mislead. In other words, the paper will report a false-positive finding.

It is possible, even likely, that this has happened with the three trials mentioned above. The fact is that, as far as I know, there is no independent replication of these studies.

In the light of all this, Popper’s axiom as applied to medicine should perhaps be modified: findings without independent replication are of no significance. Or, to put it even more bluntly: independent replication is an essential self-cleansing process of science by which it rids itself from errors, fraud and misunderstandings.

On this blog, we constantly discuss the shortcomings of clinical trials of (and other research into) alternative medicine. Yet, there can be no question that research into conventional medicine is often unreliable as well.

What might be the main reasons for this lamentable fact?

A recent BMJ article discussed 5 prominent reasons:

Firstly, much research fails to address questions that matter. For example, new drugs are tested against placebo rather than against usual treatments. Or the question may already have been answered, but the researchers haven’t undertaken a systematic review that would have told them the research was not needed. Or the research may use outcomes, perhaps surrogate measures, that are not useful.

Secondly, the methods of the studies may be inadequate. Many studies are too small, and more than half fail to deal adequately with bias. Studies are not replicated, and when people have tried to replicate studies they find that most do not have reproducible results.

Thirdly, research is not efficiently regulated and managed. Quality assurance systems fail to pick up the flaws in the research proposals. Or the bureaucracy involved in having research funded and approved may encourage researchers to conduct studies that are too small or too short term.

Fourthly, the research that is completed is not made fully accessible. Half of studies are never published at all, and there is a bias in what is published, meaning that treatments may seem to be more effective and safer than they actually are. Then not all outcome measures are reported, again with a bias towards those are positive.

Fifthly, published reports of research are often biased and unusable. In trials about a third of interventions are inadequately described meaning they cannot be implemented. Half of study outcomes are not reported.

END OF QUOTE

Apparently, these 5 issues are the reason why 85% of biomedical research is being wasted.

That is in CONVENTIONAL medicine, of course.

What about alternative medicine?

There is no question in my mind that the percentage figure must be even higher here. But do the same reasons apply? Let’s go through them again:

  1. Much research fails to address questions that matter. That is certainly true for alternative medicine – just think of the plethora of utterly useless surveys that are being published.
  2. The methods of the studies may be inadequate. Also true, as we have seen hundreds of time on this blog. Some of the most prevalent flaws include in my experience small sample sizes, lack of adequate controls (e.g. A+B vs B design) and misleading conclusions.
  3. Research is not efficiently regulated and managed. True, but probably not a specific feature of alternative medicine research.
  4. Research that is completed is not made fully accessible. most likely true but, due to lack of information and transparency, impossible to judge.
  5. Published reports of research are often biased and unusable. This is unquestionably a prominent feature of alternative medicine research.

All of this seems to indicate that the problems are very similar – similar but much more profound in the realm of alternative medicine, I’d say based on many years of experience (yes, what follows is opinion and not evidence because the latter is hardly available).

The thing is that, like almost any other job, research needs knowledge, skills, training, experience, integrity and impartiality to do it properly. It simply cannot be done well without such qualities. In alternative medicine, we do not have many individuals who have all or even most of these qualities. Instead, we have people who often are evangelic believers in alternative medicine, want to further their field by doing some research and therefore acquire a thin veneer of scientific expertise.

In my 25 years of experience in this area, I have not often seen researchers who knew that research is for testing hypotheses and not for trying to prove one’s hunches to be correct. In my own team, those who were the most enthusiastic about a particular therapy (and were thus seen as experts in its clinical application), were often the lousiest researchers who had the most difficulties coping with the scientific approach.

For me, this continues to be THE problem in alternative medicine research. The investigators – and some of them are now sufficiently skilled to bluff us to believe they are serious scientists – essentially start on the wrong foot. Because they never were properly trained and educated, they fail to appreciate how research proceeds. They hardly know how to properly establish a hypothesis, and – most crucially – they don’t know that, once that is done, you ought to conduct investigation after investigation to show that your hypothesis is incorrect. Only once all reasonable attempts to disprove it have failed, can your hypothesis be considered correct. These multiple attempts of disproving go entirely against the grain of an enthusiast who has plenty of emotional baggage and therefore cannot bring him/herself to honestly attempt to disprove his/her beloved hypothesis.

The plainly visible result of this situation is the fact that we have dozens of alternative medicine researchers who never publish a negative finding related to their pet therapy (some of them were admitted to what I call my HALL OF FAME on this blog, in case you want to verify this statement). And the lamentable consequence of all this is the fast-growing mountain of dangerously misleading (but often seemingly robust) articles about alternative treatments polluting Medline and other databases.

Is homeopathy effective for specific conditions? The FACULTY OF HOMEOPATHY (FoH, the professional organisation of UK doctor homeopaths) say YES. In support of this bold statement, they cite a total of 35 systematic reviews of homeopathy with a focus on specific clinical areas. “Nine of these 35 reviews presented conclusions that were positive for homeopathy”, they claim. Here they are:

Allergies and upper respiratory tract infections 8,9
Childhood diarrhoea 10
Post-operative ileus 11
Rheumatic diseases 12
Seasonal allergic rhinitis (hay fever) 13–15
Vertigo 16

And here are the references (I took the liberty of adding my comments in blod):

8. Bornhöft G, Wolf U, Ammon K, et al. Effectiveness, safety and cost-effectiveness of homeopathy in general practice – summarized health technology assessment. Forschende Komplementärmedizin, 2006; 13 Suppl 2: 19–29.

This is the infamous ‘Swiss report‘ which, nowadays, only homeopaths take seriously.

9. Bellavite P, Ortolani R, Pontarollo F, et al. Immunology and homeopathy. 4. Clinical studies – Part 1. Evidence-based Complementary and Alternative Medicine: eCAM, 2006; 3: 293–301.

This is not a systematic review as it lacks any critical assessment of the primary data and includes observational studies and even case series.

10. Jacobs J, Jonas WB, Jimenez-Perez M, Crothers D. Homeopathy for childhood diarrhea: combined results and metaanalysis from three randomized, controlled clinical trials. Pediatric Infectious Disease Journal, 2003; 22: 229–234.

This is a meta-analysis by Jennifer Jacobs (who recently featured on this blog) of 3 studies by Jennifer Jacobs; hardly convincing I’d say.

11. Barnes J, Resch K-L, Ernst E. Homeopathy for postoperative ileus? A meta-analysis. Journal of Clinical Gastroenterology, 1997; 25: 628–633.

This is my own paper! It concluded that “several caveats preclude a definitive judgment.”

12. Jonas WB, Linde K, Ramirez G. Homeopathy and rheumatic disease. Rheumatic Disease Clinics of North America, 2000; 26: 117–123.

This is not a systematic review; here is the (unabridged) abstract:

Despite a growing interest in uncovering the basic mechanisms of arthritis, medical treatment remains symptomatic. Current medical treatments do not consistently halt the long-term progression of these diseases, and surgery may still be needed to restore mechanical function in large joints. Patients with rheumatic syndromes often seek alternative therapies, with homeopathy being one of the most frequent. Homeopathy is one of the most frequently used complementary therapies worldwide.

Proper systematic reviews fail to show that homeopathy is an effective treatment for rheumatic conditions (see for instance here and here).

13. Wiesenauer M, Lüdtke R. A meta-analysis of the homeopathic treatment of pollinosis with Galphimia glauca. Forschende Komplementärmedizin und Klassische Naturheilkunde, 1996; 3: 230–236.

This is a meta-analysis by Wiesenauer of trials conducted by Wiesenauer.

My own, more recent analysis of these data arrived at a considerably less favourable conclusion: “… three of the four currently available placebo-controlled RCTs of homeopathic Galphimia glauca (GG) suggest this therapy is an effective symptomatic treatment for hay fever. There are, however, important caveats. Most essentially, independent replication would be required before GG can be considered for the routine treatment of hay fever. (Focus on Alternative and Complementary Therapies September 2011 16(3))

14. Taylor MA, Reilly D, Llewellyn-Jones RH, et al. Randomised controlled trials of homoeopathy versus placebo in perennial allergic rhinitis with overview of four trial series. British Medical Journal, 2000; 321: 471–476.

This is a meta-analysis by David Reilly of 4 RCTs which were all conducted by David Reilly. This attracted heavy criticism; see here and here, for instance.

15. Bellavite P, Ortolani R, Pontarollo F, et al. Immunology and homeopathy. 4. Clinical studies – Part 2. Evidence-based Complementary and Alternative Medicine: eCAM, 2006; 3: 397–409.

This is not a systematic review as it lacks any critical assessment of the primary data and includes observational studies and even case series.

16. Schneider B, Klein P, Weiser M. Treatment of vertigo with a homeopathic complex remedy compared with usual treatments: a meta-analysis of clinical trials. Arzneimittelforschung, 2005; 55: 23–29.

This is a meta-analysis of 2 (!) RCTs and 2 observational studies of ‘Vertigoheel’, a preparation which is not a homeopathic but a homotoxicologic remedy (it does not follow the ‘like cures like’ assumption of homeopathy) . Moreover, this product contains pharmacologically active substances (and nobody doubts that active substances can have effects).

________________________________________________________________________________

So, positive evidence from 9 systematic reviews in 6 specific clinical areas?

I let you answer this question.

Shiatsu is an alternative therapy that is popular, but has so far attracted almost no research. Therefore, I was excited when I saw a new paper on the subject. Sadly, my excitement waned quickly when I stared reading the abstract.

This single-blind randomized controlled study was aimed to evaluate shiatsu on mood, cognition, and functional independence in patients undergoing physical activity. Alzheimer disease (AD) patients with depression were randomly assigned to the “active group” (Shiatsu + physical activity) or the “control group” (physical activity alone).

Shiatsu was performed by the same therapist once a week for ten months. Global cognitive functioning (Mini Mental State Examination – MMSE), depressive symptoms (Geriatric Depression Scale – GDS), and functional status (Activity of Daily Living – ADL, Instrumental ADL – IADL) were assessed before and after the intervention.

The researchers found a within-group improvement of MMSE, ADL, and GDS in the Shiatsu group. However, the analysis of differences before and after the interventions showed a statistically significant decrease of GDS score only in the Shiatsu group.

The authors concluded that the combination of Shiatsu and physical activity improved depression in AD patients compared to physical activity alone. The pathomechanism might involve neuroendocrine-mediated effects of Shiatsu on neural circuits implicated in mood and affect regulation.

The Journal Complementary Therapies in Medicine also published three ‘Highlights’ of this study:

  • We first evaluated the effect of Shiatsu in depressed patients with Alzheimer’s disease (AD).
  • Shiatsu significantly reduced depression in a sample of mild-to-moderate AD patients.
  • Neuroendocrine-mediated effect of Shiatsu may modulate mood and affect neural circuits.

Where to begin?

1 The study is called a ‘pilot’. As such it should not draw conclusions about the effectiveness of Shiatsu.

2 The design of the study was such that there was no accounting for the placebo effect (the often-discussed ‘A+B vs B’ design); therefore, it is impossible to attribute the observed outcome to Shiatsu. The ‘highlight’ – Shiatsu significantly reduced depression in a sample of mild-to-moderate AD patients – therefore turns out to be a low-light.

3 As this was a study with a control group, within-group changes are irrelevant and do not even deserve a mention.

4 The last point about the mode of action is pure speculation, and not borne out of the data presented.

5 Accumulating so much nonsense in one research paper is, in my view, unethical.

Research into alternative medicine does not have a good reputation – studies like this one are not inclined to improve it.

Recent Comments

Note that comments can be edited for up to five minutes after they are first submitted but you must tick the box: “Save my name, email, and website in this browser for the next time I comment.”

The most recent comments from all posts can be seen here.

Categories