As promised, I would like to correct the errors in my previous assessment of this paper. To remind everyone:
This systematic review evaluated individualized homeopathy as a treatment for children with attention deficit and hyperactivity disorder (ADHD) when compared to placebo or usual care alone.
Thirty-seven online sources were searched up to March 2021. Studies investigating the effects of individualized homeopathy against any control in ADHD were eligible. Data were extracted to a predefined excel sheet independently by two reviewers.
Six studies were analyzed:
- 5 were RCTs
- 2 were controlled against standard treatments;
- 4 were placebo-controlled and double-blinded.
The meta-analysis revealed a significant effect size across studies of Hedges’ g = 0.542 (95% CI 0.311-0.772; z = 4,61; p < 0.001) against any control and of g = 0.605 (95% CI 0.05-1.16; z = 2.16, p = 0.03) against placebo. The effect estimations are based on studies with an average sample size of 52 participants.
The authors concluded that individualized homeopathy showed a clinically relevant and statistically robust effect in the treatment of ADHD.
_______________________________
Now that I was able to access the full papers, I would like to offer a thorough analysis.
To get included in the review, primary studies had to be:
- Published after 1980,
- Investigating an individualized homeopathic intervention in childhood ADHD,
- Comparing the intervention to a control condition (placebo, standard care or treatment as usual, both of which are referred to as “active control”) in a randomized or non-randomized parallel-group study
design with one or more arms.
Six studies were included:
- Fibert, P., Peasgood, T. & Relton, C. Rethinking ADHD intervention trials: feasibility testing of two treatments and a methodology. Eur. J. Pediatr. 178, 983–993 (2019). – DOI
- Fibert, P., Relton, C., Heirs, M. & Bowden, D. A comparative consecutive case series of 20 children with a diagnosis of ADHD receiving homeopathic treatment, compared with 10 children receiving usual care. Homeopathy 105, 194–201 (2016). – DOI
- Jacobs, J., Williams, A. L., Girard, C., Njike, V. Y. & Katz, D. Homeopathy for attention-deficit/hyperactivity disorder: a pilot randomized-controlled trial. J. Altern. Complement. Med. 11, 799–806 (2005). – DOI
- Jones, M. The efficacy of homoeopathic simillimum in the treatment of attention-deficit/hyperactivity disorder (AD/HD) in schoolgoing children aged 6-11 years. https://openscholar.dut.ac.za/bitstream/10321/534/1/Jones_2009.pdf (2009).
- Frei, H. et al. Homeopathic treatment of children with attention deficit hyperactivity disorder: a randomised, double blind, placebo controlled crossover trial. Eur. J. Pediatr. 164, 758–767 (2005). – DOI
- Oberai, P. et al. Homoeopathic management of attention deficit hyperactivity disorder: a randomised placebo-controlled pilot trial. Indian J. Res. Homoeopathy 7, 158–167 (2013).
Exclusion criteria were:
- Homeopathic intervention not individualized,
- Serious methodological flaws, such as incidental unblinding, failure to report important data, or insufficient data for meta-analysis.
One study was excluded:
- Lamont, J. Homoeopathic treatment of attention deficit hyperactivity disorder. Br. Homeopathic J. 86, 196–200 (1997). – DOI
I will first make several points about Walach’s systematic review itself and then have a look at the primary studies that it included. Finally, I will try to draw some conclusions.
The review authors state in their introduction that “beneficial effects of this intervention [homeopathy] have been shown for various kinds of medical conditions, including child diarrhea, supportive care in cancer, fibromyalgia, or ADHD.” In other words, already in the introduction, they disclose their strong pro-homeopathy bias; it would, of course, not be difficult to find investigations that contradict their optimism.
Despite the stated inclusion/exclusion criteria, the authors did include the Frei-study that did not follow a parallel-group design (see also below).
The authors included two active-controlled studies both of which did not report the type of treatment received by the control group. In other words, these trials failed to report important data which was a stated exclusion criterium (see below).
In their discussion section, the authors state that “all included studies employed individualized homeopathy and were of comparable, solid quality, hence a lack of methodological rigor is unlikely the reason for the difference between homeopathy and controls…” This, I think, is grossly misleading; even according to the authors’ own assessments, one study was deemed to have a high risk of bias and in two studies the risk of bias was “unclear”.
The overall positive effect of homeopathy demonstrated by the review was determined almost exclusively by the study of Oberai et al (p-value = 0.000). In fact, the studies by Jones and by Jacobs were negative, and the one by Frei was borderline positive with a p-value of 0.46. The authors address this crucial issue repeatedly and claim that excluding Oberai et al would still generate an overall positive meta-analytic result. Yet, they do not mention that the overall result would no longer be clinically relevant.
Looking at the included primary studies, I should make the following points:
- The two Filbert studies, as mentioned, failed to report important data and should, according to the stated exclusion criteria, not have been included.
- The study by Jacobs was a pilot study and generated negative findings.
- The study by Jones is a non-peer-reviewed thesis. In my view, it should never have been included.
- The study by Frei was a cross-over trial. According to the exclusion/inclusion criteria of the authors, it should not have been included.
- The study by Oberai et al is the trial that has by far the largest effect size and thus is the driver of the overall result of the review. It is therefore important to have a closer look at it.
Here is the abstract:
Objective: To evaluate the usefulness of individualised homoeopathic medicines in treatment of Attention Deficit Hyperactivity Disorder (ADHD).
Design: Randomised placebo-controlled single-blind pilot trial.
Setting: Central Research Institute (Homoeopathy), Kottayam, Kerala, India from June 2009 to November 2011.
Participants: Children aged 6-15 years meeting the Diagnostic Statistical Manual of mental disorders (DSM-IV) criteria for ADHD.
Interventions: A total of 61 patients (Homoeopathy = 30, placebo = 31) were randomised to receive either individualised homoeopathic medicine in fifty millesimal (LM) potency or placebo for a period of one year.
Outcome measures: Conner’s Parent Rating Scale-Revised: Short (CPRS-R (S)), Clinical Global Impression-Severity Scale (CGI-SS), Clinical Global Impression- Improvement Scale (CGI-IS) and Academic performance.
Results: A total of 54 patients (homoeopathy = 27, placebo = 27) were analysed under modified intention to treat (ITT). All patients in homoeopathy group showed better outcome in baseline adjusted General Linear Model (GLM) repeated measures ANCOVA for oppositional, cognition problems, hyperactivity and ADHD Index (domains of CPRS-R (S)) and CGI-IS at T3, T6, T9 and T12 (P = 0.0001). The mean baseline-adjusted treatment difference between groups at month 12 from baseline for all individual outcome measures favoured homoeopathy group; Oppositional (−16.4, 95% CI – 20.5 to − 12.2, P = 0.0001), Cognition problems (−15.5, 95% CI − 19.2 to − 11.8, P = 0.0001), Hyperactivity (−20.6, 95% CI − 25.6 to − 15.4, P = 0.0001), ADHD I (−15.6, 95% CI − 19.5 to − 11.6, P = 0.0001), Academic performance 14.4%, 95% CI 8.3 to 20.5, P = 0.0001), CGISS (−1.6, 95% CI − 1.9 to − 1.2, P = 0.0001), CGIIS (−1.6, 95% CI − 2.3 to -0.9, P = 0.0001).
Conclusion: This pilot study provides evidence to support the therapeutic effects of individualised homoeopathic medicines in ADHD children. However, the results need to be validated in multi-center randomised double-blind placebo-controlled clinical trial.
Here are a few points of concern related to the Oberai et al:
- The trial was a mere pilot study.
- Despite the fact that it is now 9 years old, the authors never published a definitive trial.
- The study was published in an obscure journal that is not Medline-listed.
- The study is very poorly reported.
- It is unclear how the diagnosis of ADHD for including the patients was verified.
- The control patients were treated for one year with a placebo and no other therapies. In my view, this is not ethical.
- The method of randomization is unclear.
- The authors state that acute symptoms were treated throughout the study period with homeopathy, even in the control group. This seems odd and defies the principle of a placebo-controlled trial.
- The authors state that only the patients were blind, not the investigators. This opens the door wide for all sorts of biases. It is, for example, likely that it also de-blinded the patients (the verum could be adjusted and changed, while the placebo remained constant).
All in all, this paper is of poor quality, Its findings are far from trustworthy and were not meant to be definitive. According to the following exclusion criteria, it should have been excluded:
- It had several serious methodological flaws.
- It did not blind the investigators.
- It is likely that patients were de-blinded.
- It failed to report important data.
So, why did Walach and his co-authors include it?
Could it be because, without the Oberai-study, the overall findings of the review would at best have turned out to be borderline significant and not clinically relevant?
12 Responses to Walach’s new meta-analysis of homeopathy revisited
Leave a Reply to Mojo Cancel reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
…..and yet again plausible biological mechanisms for which homeopathy may help ADHD. Like everything else…….??????. Perhaps Prince Charles can help us out.
Great in-depth analysis!
Clearly, Walach’s meta-analysis is deeply flawed and ought never to have been published. Nevertheless, we can sadly look forward to pro-homeopathy dupes parading Walach’s paper for ever.
I would like to expand on a couple of points in the post and my comment below the other post on this topic, but first a disclaimer/explanation: I worked as a nurse in CAMHS (child and adolescent mental health) for 30 years; I was lead clinician in assessing many bairns for possible ADHD; I am well-versed in our UK guidelines on this topic.
Not having any clear indication of a diagnosis being reached is inherently flawed as it is not obvious that the study population is actually an ADHD population. Bland statements about “meeting DSM criteria” are not a diagnosis, especially as there is no indication of who made this judgement nor how (full assessment can be time-consuming and complex, not just looking at some ticky box – https://www.nice.org.uk/guidance/ng87), especially as it does not seem that anyone involved in the papers I’ve looked at has any expertise in CAMHS, paediatrics or even neurology.
At least one of the Filbert papers excludes children referred from a school or NHS services (that would most likely be CAMHS or paediatrics), making the likelihood of an actual diagnosis lower. In that paper “usual care” is not defined and, as CAMHS patients are excluded, that would preclude the use of the standard medications (methylphenidate or atomoxetine), as these should only be prescribed by a relevant specialist. So I’m not clear what comparison is used…
The other Filbert paper again does not confirm diagnosis. Neither does the Jones thesis.
With regard to outcome measures, these are sorely limited in most cases, using short versions of the parents’ Conners questionaire and non-ADHD rating scales. ADHD is a condition displayed across all areas of a child’s life in all settings (I’ve even gone so far as to seek reports from football coaches…), so measuring outcomes in just one area of someone’s life is inadequate, especially as it is not confirmed from other areas. The Filbert paper which does seek responses from teachers does not obtain consistent replies, again calling into question any global improvement. Conclusions about ADHD treatment cannot reasonably be drawn from this information.
Other papers use the Conners Global Index to indicate improvement in ADHD symptoms, which is a complete misuse of this, as improvements, assuming they are there, could be coming from the non-ADHD linked parts of Conners.
Add all of these things together and without getting into small samples, the high likelihood of parental bias in more than one of these studies or any other flaws, and we can easily see that none of these studies are capable of showing anything about ADHD and its treatment.
Oh, some of the authors also tell us that their work doesn’t do this.
From Jacobs et alia: “Results: There were no statistically significant differences between homeopathic remedy and placebo groups on the primary or secondary outcome variables. However, there were statistically and clinically significant improvements in both groups on many of the outcome measures.
“Conclusions: This pilot study provides no evidence to support a therapeutic effect of individually selected homeopathic remedies in children with ADHD. A therapeutic effect of the homeopathic encounter is suggested and warrants further evaluation. Future studies should be carried out over a longer period of time and should include a control group that does not receive the homeopathic consultation. Comparison to conventional stimulant medication for ADHD also should be considered.”
Note the trying to push homeopathy despite showing no or selective effects – authorial bias much?
Or something similar from Jones’ thesis – “On analysis, the results (Table 4.8, 4.9 and 4.10) showed no statistically significant effect of treatment (i.e. no difference between treatment and placebo
group), but across the whole trial and within each group (particularly the
treatment group) subjects had significant reductions in symptoms (i.e. the
reductions in symptoms were large enough that there was less than 5% chance
that they were random fluctuations/effects). This was seen in both the treatment
and placebo groups, as indicated by Table 4.11, 4.12 and 4.13, but more
significant reductions were seen in the treatment group, indicated by Table 4.14.
As discussed in Chapter 5, this by no means rules out the efficacy of
homoeopathic simillimum for the treatment of AD/HD.”
Sorry about the copy…
Overall, need to do better, need to design studies properly, need to make sure to study the correct population, need to use decent outcome measures, need to stop trying to cherry pick sub-sections of results which look like they say what you want (we are almost getting into what Orac calls “torturing the data until it confesses”), and need to stop trying to justify biologically implausible attempts at treatment.
I don’t have access to the full text. Do they say anything about how this study would be blinded? Would the subjects in this group have a nice 30-45 minute chat in which they are encouraged to talk about their feelings and likes and dislikes? Otherwise I would have thought that adequate blinding would be a bit tricky.
Dunno: I’m not coughing up for the full text either. That definitely is a potential problem, but I’m sure it’ll be hand-waved away, like all that nonsense in Jones’ thesis. How that rubbish was allowed by the awarding institution is beyond me – I wouldn’t accept nonsense like that in essays from students I was supervising.
Slight amendment: Jones’ thesis does mention “Participants had to be pre-diagnosed with AD/HD by a child
psychologist or paediatrician.” Which is still not standard practice in the UK.
And if you delve into the results I’m not seeing much real difference between treatment and placebo groups, both of which are so small as to be hardly worth it…
And then we see that so many different “remedies” were used and chopped and changed between different appointments as to make one wonder what they were playing at.
And look more closely at the Results section for some fine twisting and turning after admitting to no significant differences between the groups.
Ahhh, yes, the Oberai paper.
They defined three primary outcome measures and reported effect sizes of 0.22, 0.59 and 0.54.
How on earth can this condense to 1.436 as given by Gaertner et al. for the overall effect size of this trial?
Walach’s article was retracted.
https://www.nature.com/articles/s41390-022-02127-3#additional-information
It is a strange way to retract a publication, as there is no evidence of fraud or image manipulation. Strange that they did not ask the authors how they recalculated the effect sizes, why “overall allocation of Risk of Bias (ROB) was not in line with Cochrane guideline”? They don’t say. The only time I have read that “the Editor-in-Chief has substantial concerns regarding the validity of the results presented in this article” was in the case of Séralini’s article. This smells like Mr. Aust and other “skeptics” were behind it to push the retraction in Pediatrics (considering that Aust already months before expressed his desire to retract it with a publication in the Skeptikal Inquirer). As there is no fraud, and the “substantial concern” is reduced to a few points that are also not entirely clear, the conclusion of the meta-analysis stands.
if you think once can just push for a retraction you are more naive than I thought.
“As there is no fraud”
to misrepresent data and conduct dodgy analyses can amount to fraud!
So I take it, you would only retract a paper if fraud can be poven? What about errors and ignorance?
By the way: How would you prove fraud?
Or: There are two options only: The authors introduced the misleading data either willingly (=fraud) or unwillingly (=stupidity). Which one is more probable with the three authors being full fledged academics?
“As there is no fraud, and the “substantial concern” is reduced to a few points that are also not entirely clear, the conclusion of the meta-analysis stands.”
I would recommend, you reread the editor’s comments.