Reflexology is the treatment of reflex zones, usually on the sole of the feet, with manual massage and pressure. Reflexologists assume that certain zones correspond to certain organs, and that their treatment can influence the function of these organs. Thus reflexology is advocated for all sorts of conditions. Proponents are keen to point out that their approach has many advantages: it is pleasant (the patient feels well with the treatment and the therapist feels even better with the money), safe and cheap, particularly if the patient does the treatment herself.
Self-administered foot reflexology could be practical because it is easy to learn and not difficult to apply. But is it also effective? A recent systematic review evaluated the effectiveness of self-foot reflexology for symptom management.
Participants were healthy persons not diagnosed with a specific disease. The intervention was foot reflexology administered by participants, not by practitioners or healthcare providers. Studies with either between groups or within group comparison were included. The electronic literature searches utilized core databases (MEDLINE, EMBASE, Cochrane, and CINAHL Chinese (CNKI), Japanese (J-STAGE), and Korean databases (KoreaMed, KMbase, KISS, NDSL, KISTI, and OASIS)).
Three non-randomized trials and three before-and-after studies met the inclusion criteria. No RCTs were located. The results of these studies showed that self-administered foot reflexology resulted in significant improvement in subjective outcomes such as perceived stress, fatigue, and depression. However, there was no significant improvement in objective outcomes such as cortisol levels, blood pressure, and pulse rate. We did not find any randomized controlled trial.
The authors concluded that this study presents the effectiveness of self-administered foot reflexology for healthy persons’ psychological and physiological symptoms. While objective outcomes showed limited results, significant improvements were found in subjective outcomes. However, owing to the small number of studies and methodological flaws, there was insufficient evidence supporting the use of self-performed foot reflexology. Well-designed randomized controlled trials are needed to assess the effect of self-administered foot reflexology in healthy people.
I find this review quite interesting, but I would draw very different conclusions from its findings.
The studies that are available turned out to be of very poor methodological quality: they lack randomisation or rely on before/after comparisons. This means they are wide open to bias and false-positive results, particularly in regards to subjective outcome measures. Predictably, the findings of this review confirm that no effects are seen on objective endpoints. This is in perfect agreement with the hypothesis that reflexology is a pure placebo. Considering the biological implausibility of the underlying assumptions of reflexology, this makes sense.
My conclusions of this review would therefore be as follows: THE RESULTS ARE IN KEEPING WITH REFLEXOLOGY BEING A PURE PLACEBO.
Reiki healers believe they are able to channel ‘healing energy’ into patients’ body and thus enable them to get healthy. If Reiki were not such a popular treatment, one could brush such claims aside and think “let the lunatic fringe believe what they want”. But as Reiki so effectively undermines consumers’ sense of reality and rationality, I feel a responsibility to inform the public what Reiki truly amounts to.
This pilot study compared the effects of Reiki therapy with those of companionship on improvements in quality of life, mood, and symptom distress in cancer patients receiving chemotherapy. Thirty-six breast cancer patients received one of three treatments:
- usual care,
- Reiki + usual care,
- companionship + usual care.
First, data were collected from patients receiving usual care. Second, patients were randomized to either receive Reiki or a companionship during chemotherapy.
Questionnaires assessing quality of life, mood, symptom distress, and Reiki acceptability were completed at baseline and chemotherapy sessions 1, 2, and 4.
The results show that Reiki was rated relaxing with no side effects. Reiki and companionship groups both reported improvements in quality of life and mood that were greater than those seen in the usual care group.
The authors concluded that interventions during chemotherapy, such as Reiki or companionship, are feasible, acceptable, and may reduce side effects.
Yet another example of utterly bizarre conclusions from a fairly straight forward study and quite clear results. What they really demonstrate is the fact that Reiki is nothing more than a placebo; its perceived benefit relies entirely on non-specific effects. This view is also supported by our systematic review (its 1st author is a Reiki healer!): the evidence is insufficient to suggest that reiki is an effective treatment for any condition. Therefore the value of reiki remains unproven.
In other words, we do not need a trained Reiki master, nor the illusion of some mysterious ‘healing energy’. Simple companionship without woo or make-believe has exactly the same effect without undermining rationality. Or, to put it much more bluntly: REIKI IS NONSENSE ON STILTS.
Poor sleep quality during pregnancy is a frequent problem. Drug treatment can be problematic due to possible adverse effects for mother and embryo/foetus. Many pregnant women prefer natural treatments and assume that ‘natural’ equals harmless.
In the present study, the sedative effects of Bryophyllum pinnatum were investigated. This remedy is a phytotherapeutic medication predominantly used in anthroposophic medicine. In previous clinical studies on its tocolytic effect, B. pinnatum showed a promising risk/benefit ratio for mother and child. A recent analysis of the prescribing pattern for B. pinnatum in a network of anthroposophic physicians revealed sleep disorders as one of the most frequent diagnosis.
In this prospective, multi-centre, observational study, pregnant women suffering from sleep problems were treated with B. pinnatum (350mg tablets, 50% leaf press juice, Weleda AG, Arlesheim, dosage at physician’s consideration). Sleep quality, daily sleepiness and fatigue were assessed with the aid of standardised questionnaires, at the beginning of the treatment and after 2 weeks. Possible adverse effects perceived by the patients during the treatment were recorded.
The results show that the number of wake-ups, as well as the subjective quality of sleep was significantly improved at the end of the treatment with B. pinnatum. The Epworth Sleeping Scale decreased, indicating a reduction in tiredness during the day. There was, however, no evidence for a prolongation of the sleep duration, reduction in the time to fall asleep, as well as change in the Fatigue Severity Scale after B. pinnatum. No serious adverse drug reactions were detected.
From these data, the authors concluded that B. pinnatum is a suitable treatment of sleep problems in pregnancy. The data of this study encourage further clinical investigations on the use of B. pinnatum in sleep disorders.
Clinical trials of anthroposophic remedies, i.e. remedies which are based on the school of medicine founded by Rudolf Steiner, are very rare. Therefore this trial could be important.
B. pinnatum is a plant used in traditional Tai medicine against hypertension, and to some extend this makes sense: it contains cardiac glycosides which might help lowering elevated blood pressure. The reason for its use as a hypnotic, however, is not clear.
So, is B pinnatum really a ‘suitable treatment of sleep problems in pregnancy’? I doubt it for the following reasons:
- the effects documented in this study are far from convincing,
- we would need much more solid data to issue such a general recommendation,
- cardiac glycosides can cause very serious adverse effects,
- the sample size of the study is at least one dimension too small for assuming that it is safe,
- we know nothing about its potential to cause harm to the foetus.
Personally, I find it irresponsible to draw conclusions such as the ones above on the basis of data which are flimsy to the extreme. I ask myself, to what extend wishful thinking might be a regrettable characteristic for the entire field of anthroposophic medicine.
Highly diluted homeopathic remedies are pure placebos; at least this is what sceptics have been saying for about 200 years. This assumption is based on the fact that homeopathy’s plausibility is close to zero and that the totality of the reliable evidence fails to demonstrate that it works beyond placebo for any condition.
But, if this is true, why do so many patients swear by homeopathy and experience benefit from it? This question has been answered many times: THE BENEFIT IS NOT DUE TO THE REMEDY BUT TO NON-SPECIFIC EFFECTS OF THE CONSULTATION.
More confirmation for this conclusion comes from an unexpected source.
Indian homeopaths recently published a trial of individualized homeopathy in osteoarthritis. To be more precise, it was a prospective, parallel-arm, double-blind, randomized, placebo-controlled pilot study which was conducted from January to October 2014 involving 60 patients (homeopathy, n = 30; placebo, n = 30). All patients were suffering from acute painful episodes of knee osteoarthritis and visiting the outpatient clinic of Mahesh Bhattacharyya Homeopathic Medical College and Hospital, West Bengal, India.
The results show statistically significant reduction in 3 visual analogue scales (measuring pain, stiffness, and loss of function) and Osteoarthritis Research Society International scores in both groups over 2 weeks (P < .05). However, group differences were not significant (P > .05).
The authors conclude that, overall, homeopathy did not appear to be superior to placebo; still, further rigorous evaluation in this design involving a larger sample size seems feasible in future.
Considering what I wrote above, I would alter these conclusion to something much more reasonable: further studies of homeopathy are certainly feasible. However, they are neither necessary nor desirable.
TO PUT IT DIFFERENTLY: HOMEOPATHY BELONGS IN THE BOOKS OF MEDICAL HISTORY.
Homeopathy has many critics who claim that there is no good evidence for this type of therapy. Homeopaths invariably find this most unfair and point to a plethora of studies that show an effect. They are, of course, correct! There are plenty of trials that suggest that homeopathic remedies do work. The question, however, is HOW RELIABLE ARE THESE STUDIES?
Here is a brand new one which might stand for dozens of others.
In this study, homeopaths treated 50 multimorbid patients with homeopathic remedies identifies by a method called ‘polarity analysis’ (PA) and prospectively followed them over one year (PA enables homeopaths to calculate a relative healing probability, based on Boenninghausen’s grading of polar symptoms).
The 43 patients (86%) who completed the observation period experienced an average improvement of 91% in their initial symptoms. Six patients dropped out, and one did not achieve an improvement of 80%, and was therefore also counted as a treatment failure. The cost of homeopathic treatment was 41% of projected equivalent conventional treatment.
Good news then for enthusiasts of homeopathy? 91% improvement!
Yet, I am afraid that critics might not be bowled over. They might smell a whiff of selection bias, lament the lack of a control group or regret the absence of objective outcome measures. But I was prepared to go as far as stating that such results might be quite interesting… until I read the authors’ conclusions that is:
Polarity Analysis is an effective method for treating multimorbidity. The multitude of symptoms does not prevent the method from achieving good results. Homeopathy may be capable of taking over a considerable proportion of the treatment of multimorbid patients, at lower costs than conventional medicine.
Virtually nothing in these conclusions is based on the data provided. They are pure extrapolation and wild assumptions. Two questions seem to emerge from this:
- How on earth can we take this and so many other articles on homeopathy seriously?
- When does this sort of article cross the line between wishful thinking and scientific misconduct?
As promised, I will try with this post to explain my reservations regarding the new meta-analysis suggesting that individualised homeopathic remedies are superior to placebos. Before I start, however, I want to thank all those who have commented on various issues; it is well worth reading the numerous and diverse comments.
To remind us of the actual meta-analysis, it might be useful to re-publish its abstract (the full article is also available online):
A rigorous and focused systematic review and meta-analysis of randomised controlled trials (RCTs) of individualised homeopathic treatment has not previously been undertaken. We tested the hypothesis that the outcome of an individualised homeopathic treatment approach using homeopathic medicines is distinguishable from that of placebos.
The review’s methods, including literature search strategy, data extraction, assessment of risk of bias and statistical analysis, were strictly protocol-based. Judgment in seven assessment domains enabled a trial’s risk of bias to be designated as low, unclear or high. A trial was judged to comprise ‘reliable evidence’ if its risk of bias was low or was unclear in one specified domain. ‘Effect size’ was reported as odds ratio (OR), with arithmetic transformation for continuous data carried out as required; OR > 1 signified an effect favouring homeopathy.
Thirty-two eligible RCTs studied 24 different medical conditions in total. Twelve trials were classed ‘uncertain risk of bias’, three of which displayed relatively minor uncertainty and were designated reliable evidence; 20 trials were classed ‘high risk of bias’. Twenty-two trials had extractable data and were subjected to meta-analysis; OR = 1.53 (95% confidence interval (CI) 1.22 to 1.91). For the three trials with reliable evidence, sensitivity analysis revealed OR = 1.98 (95% CI 1.16 to 3.38).
Medicines prescribed in individualised homeopathy may have small, specific treatment effects. Findings are consistent with sub-group data available in a previous ‘global’ systematic review. The low or unclear overall quality of the evidence prompts caution in interpreting the findings. New high-quality RCT research is necessary to enable more decisive interpretation.
Since my team had published an RCTs of individualised homeopathy, it seems only natural that my interest focussed on why the study (even though identified by Mathie et al) had not been included in the meta-analysis. Our study had provided no evidence that adjunctive homeopathic remedies, as prescribed by experienced homeopathic practitioners, are superior to placebo in improving the quality of life of children with mild to moderate asthma in addition to conventional treatment in primary care.
I was convinced that this trial had been rigorous and thus puzzled why, despite receiving ‘full marks’ from the reviewers, they had not included it in their meta-analysis. I thus wrote to Mathie, the lead author of the meta-analysis, and he explained: For your trial (White et al. 2003), under domain V of assessment, we were unable to extract data for meta-analysis, and so it was attributed high risk of bias, as specified by the Cochrane judgmental criteria. Our designated main outcome was the CAQ, for which we needed to know (or could at least estimate) a mean and SD for both the baseline and the end-point of the study. Since your paper reported only the change from baseline in Table 3 or in the main text, it is not possible to derive the necessary end-point for analysis.
It took a while and several further emails until I understood: our study did report both the primary (Table 2 quality of life) and secondary outcome measure (Table 3 severity of symptoms). The primary outcome measure was reported in full detail such that a meta-analysis would have been possible. The secondary outcome measure was also reported but not in full detail, and the data provided by us would not lend themselves to meta-analyses. By electing not our primary but our secondary outcome measure for their meta-analysis, Mathie et al were able to claim that they were unable to use our study and reject it for their meta-analysis.
Why did they do that?
The answer is simple: in their methods section, they specify that they used outcome measures “based on a pre-specified hierarchical list in order of greatest to least importance, recommended by the WHO“. This, I would argue is deeply flawed: the most important outcome measure of a study is usually the one for which the study was designed, not the one that some guys at the WHO feel might be important (incidentally, the WHO list was never meant to be applied to meta-analyses in that way).
By following rigidly their published protocol, the authors of the meta-analysis managed to exclude our negative trial. Thus they did everything right – or did they?
Well, I think they committed several serious mistakes.
- Firstly, they wrote the protocol, which forced them to exclude our study. Following a protocol is not a virtue in itself; if the protocol is nonsensical it even is the opposite. Had they proceeded as is normal in such cases and used our primary outcome measure in their meta-analyses, it is most likely that their overall results would not have been in favour of homeopathy.
- Secondly, they awarded our study a malus point for the criterium ‘selective outcome reporting’. This is clearly a wrong decision: we did report the severity-outcome, albeit not in sufficient detail for their meta-analysis. Had they not committed this misjudgment, our RCT would have been the only one with an ‘A’ rating. This would have very clearly highlighted the nonsense of excluding the best-rated trial from meta-analysis.
There are several other oddities as well. For instance, Mathie et al judge our study to be NOT free of vested interest. I asked Mathie why they had done this and was told it is because we accepted free trial medication from a homeopathic pharmacy. I would argue that my team was far less plagued by vested interest than the authors of their three best (and of course positive) trials who, as I happen to know, are consultants for homeopathic manufacturers.
And all of this is just in relation to our own study. Norbert Aust has uncovered similar irregularities with other trials and I take the liberty of quoting his comments posted previously again here:
I have reason to believe that this review and metaanalysis in biased in favor of homeopathy. To check this, I compared two studies (1) Jacobs 1994 about the treatment of childhood diarrhea in Nicaragua, (2) Walach 1997 about homeopathic threatment of headaches. The Jacobs study is one of the three that provided ‘reliable evidence’, Walach’s study earned a poor C2.2 rating and was not included in the meta-analyses. Jacobs’ results were in favour of homeopathy, Walach’s not.
For the domains where the rating of Walach’s study was less than that of the Jacobs study, please find citations from the original studies or my short summaries for the point in question.
Domain I: Sequence generation:
“The remedy selected was then mailed to a notary public who held a stock of placebos. The notary threw a dice and mailed either the homeopathic remedy or an appropriate placebo. The notary was provided with a blank randomisation list.”
Rating: UNCLEAR (Medium risk of bias)
“For each of these medications, there was a box of tubes in sequentially numbered order which had been previously randomized into treatment or control medication using a random numbers table in blocks of four”
Rating: YES (Low risk of bias)
Domain IIIb: Blinding of outcome assessor
“The notary was provided with a blank randomization list which was an absolutely unique document. It was only handed out after the biometrician (WG) had deposited all coded original data as a printout at the notary’s office. (…) Data entry was performed blindly by personnel not involved in the study. ”
Rating: UNCLEAR (Medium risk of bias)
“All statistical analyses were done before breaking the randomisation code, using the program …”
Rating: YES (Low risk of bias)
Domain V: Selective outcome reporting
Study protocol was published in 1991 prior to enrollment of participants, all primary outcome variables were reported with respect to all participants and the endpoints.
Rating: NO (high risk of bias)
No prior publication of protocol, but a pilot study exists. However this was published in 1993 only after the trial was performed in 1991. Primary outcome defined (duration of diarrhea), reported but table and graph do not match, secondary outcome (number of unformed stools on day 3) seems defined post hoc, for this is the only one point in time, this outcome yielded a significant result.
Rating: YES (low risk of bias)
Domain VI: Other sources of bias:
Rating: NO (high risk of bias), no details given
Imbalance of group properties (size, weight and age of children), that might have some impact on course of disease, high impact of parallel therapy (rehydration) by far exceeding effect size of homeopathic treatment
Rating: YES (low risk of bias), no details given
In a nutshell: I fail to see the basis for the different ratings in the studies themselves. I assume bias of the authors of the review.
So, what about the question posed in the title of this article? The meta-analysis is clearly not a ‘proof of concept’. But is it proof for misconduct? I asked Mathie and he answered as follows: No, your statement does not reflect the situation at all. As for each and every paper, we selected the main outcome measure for your trial using the objective WHO classification approach (in which quality of life is clearly of lower rank than severity). This is all clearly described in our prospective protocol. Under no circumstances did we approach this matter retrospectively, in the way you are implying.
Some nasty sceptics might have assumed that the handful of rigorous studies with negative results were well-known to most researchers of homeopathy. In this situation, it would have been hugely tempting to write the protocol such that these studies must be excluded. I am thrilled to be told that the authors of the current new meta-analysis (who declared all sorts of vested interests at the end of the article) resisted this temptation.
As I have said on several occasions before: I am constantly on the lookout for new rigorous science that supports the claims of alternative medicine. Thus I was delighted to find a recent and potentially important article with some positive evidence.
Fish oil has been studied extensively in terms of its effects on health. We know that it has powerful anti-inflammatory properties and might thus benefit a wide range of conditions. However, the effects of FO in rheumatoid arthritis (RA) have not been examined in the context of contemporary treatment of early RA.
A new study has tried to fill this gap by examining the effects of high versus low dose FO in early RA employing a ‘treat-to-target’ protocol of combination disease-modifying anti-rheumatic drugs (DMARDs).
Patients with RA <12 months’ duration and who were DMARD-naïve were enrolled and randomised 2:1 to FO at a high dose or low dose (for masking). These groups, designated FO and control, were given 5.5 or 0.4 g/day, respectively, of the omega-3 fats, eicosapentaenoic acid + docosahexaenoic acid. All patients received methotrexate (MTX), sulphasalazine and hydroxychloroquine, and DMARD doses were adjusted according to an algorithm taking disease activity and toxicity into account. DAS28-erythrocyte sedimentation rate, modified Health Assessment Questionnaire (mHAQ) and remission were assessed three monthly. The primary outcome measure was failure of triple DMARD therapy.
In the FO group, failure of triple DMARD therapy was lower (HR=0.28 (95% CI 0.12 to 0.63; p=0.002) unadjusted and 0.24 (95% CI 0.10 to 0.54; p=0.0006) following adjustment for smoking history, shared epitope and baseline anti–cyclic citrullinated peptide. The rate of first American College of Rheumatology (ACR) remission was significantly greater in the FO compared with the control group (HRs=2.17 (95% CI 1.07 to 4.42; p=0.03) unadjusted and 2.09 (95% CI 1.02 to 4.30; p=0.04) adjusted). There were no differences between groups in MTX dose, DAS28 or mHAQ scores, or adverse events.
The authors concluded that FO was associated with benefits additional to those achieved by combination ‘treat-to-target’ DMARDs with similar MTX use. These included reduced triple DMARD failure and a higher rate of ACR remission.
So here we have a dietary supplement that actually might generate more good than harm! There is a mountain of data of good research on the subject. We understand the mechanism of action and we have encouraging clinical evidence. Some people might still say that we do not need to take supplements in order to benefit from the health effects of FO, consuming fatty fish regularly might have the same effects. This is true, of course, but the amount of fish that one would need to eat every day would probably be too large for most people’s taste.
The drawback (from the perspective of alternative medicine) in all this is, of course, that some experts might deny that FO has much to do with alternative medicine. Again: what do we call alternative medicine that works? We call it MEDICINE! And perhaps FO is an excellent example of exactly that.
Guest post by Pete Attkins
Commentator “jm” asked a profound and pertinent question: “What DOES it take for people to get real in this world, practice some common sense, and pay attention to what’s going on with themselves?” This question was asked in the context of asserting that personal experience always trumps the results of large-scale scientific experiments; and asserting that alt-med experts are better able to provide individulized healthcare than 21st Century orthodox medicine.
What does common sense and paying attention lead us to conclude about the following? We test a six-sided die for bias by rolling it 100 times. The number 1 occurs only once and the number 6 occurs many times, never on its own, but in several groups of consecutive sixes.
I think it is reasonable to say that common sense would, and should, lead everyone to conclude that the die is biased and not fit for its purpose as a source of random numbers.
In other words, we have a gut feeling that the die is untrustworthy. Gut instincts and common sense are geared towards maximizing our chances of survival in our complex and unpredictable world — these are innate and learnt behaviours that have enabled humans to survive despite the harshness of our ever changing habitat.
Only very recently in the long history of our species have we developed specialized tools that enable us to better understand our harsh and complex world: science and critical thinking. These tools are difficult to master because they still haven’t been incorporated into our primary and secondary formal education systems.
The vast majority of people do not have these skills therefore, when a scientific finding flies in the face of our gut instincts and/or common sense, it creates an overwhelming desire to reject the finding and classify the scientist(s) as being irrational and lacking basic common sense. It does not create an intense desire to accept the finding then painstakingly learn all of the science that went into producing the finding.
With that in mind, let’s rethink our common sense conclusion that the six-sided die is biased and untrustworthy. What we really mean is that the results have given all of us good reason to be highly suspicious of this die. We aren’t 100% certain that this die is biased, but our gut feeling and common sense are more than adequate to form a reasonable mistrust of it and to avoid using it for anything important to us. Reasons to keep this die rather than discard it might be to provide a source of mild entertainment or to use its bias for the purposes of cheating.
Some readers might be surprised to discover at this point that the results I presented from this apparently heavily-biased die are not only perfectly valid results obtained from a truly random unbiased die, they are to be fully expected. Even if the die had produced 100 sixes in that test, it would not confirm that the die is biased in any way whatsoever. Rolling a truly unbiased die once will produce one of six possible outcomes. Rolling the same die 100 times will produce one unique sequence out of the 6^100 (6.5 x 10^77) possible sequences: all of which are equally valid!
Gut feeling plus common sense rightfully informs us that the probability of a random die producing one hundred consecutive sixes is so incredibly remote that nobody will ever see it occur in reality. This conclusion is also mathematically sound: if there were 6.5 x 10^77 people on Earth, each performing the same test on truly random dice, there is no guarantee that anyone would observe a sequence of one hundred consecutive sixes.
When we observe a sequence such as 2 5 1 4 6 3 1 4 3 6 5 2… common sense informs us that the die is very likely random. If we calculate the arithmetic mean to be very close to 3.5 then common sense will lead us to conclude that the die is both random and unbiased enough to use it as a reliable source of random numbers.
Unfortunately, this is a perfect example of our gut feelings and common sense failing us abysmally. They totally failed to warn us that the 2 5 1 4 6 3 1 4 3 6 5 2… sequence we observed had exactly the same (im)probability of occurring as a sequence of one hundred 6s or any other sequence that one can think of that doesn’t look random to a human observer.
The 100-roll die test is nowhere near powerful enough to properly test a six-sided die, but this test is more than adequately powered to reveal some of our cognitive biases and some of the deficits in our personal mastery of science and critical thinking.
To properly test the die we need to provide solid evidence that it is both truly random and that its measured bias tends towards zero as the number of rolls tends towards infinity. We could use the services of one testing lab to conduct billions of test rolls, but this would not exclude errors caused by such things as miscalibrated equipment and experimenter bias. It is better to subdivide the testing across multiple labs then carefully analyse and appropriately aggregate the results: this dramatically reduces errors caused by equipment and humans.
In medicine, this testing process is performed via systematic reviews of multiple, independent, double-blind, placebo-controlled trials — every trial that is insufficiently powered to add meaningfully to the result is rightfully excluded from the aggregation.
Alt-med relies on a diametrically opposed testing process. It performs a plethora of only underpowered tests; presents those that just happen to show a positive result (just as a random die could’ve produced); and sweeps under the carpet the overwhelming number of tests that produced a negative result. It publishes only the ‘successes’, not its failures. By sweeping its failures under the carpet it feels justified in making the very bold claim: Our plethora of collected evidence shows clearly that it mostly ‘works’ and, when it doesn’t, it causes no harm.
One of the most acidic tests for a hypothesis and its supporting data (which is a mandatory test in a few branches of critical engineering) is to substitute the collected data for random data that has been carefully crafted to emulate the probability mass functions of the collected datasets. This test has to be run multiple times for reasons that I’ve attempted to explain in my random die example. If the proposer of the hypothesis is unable to explain the multiple failures resulting from this acid test then it is highly likely that the proposer either does not fully understand their hypothesis or that their hypothesis is indistinguishable from the null hypothesis.
Guest post by Jan Oude-Aost
ADHD is a common disorder among children. There are evidence based pharmacological treatments, the best known being methylphenidate (MPH). MPH has kind of a bad reputation, but is effective and reasonably safe. The market is also full of alternative treatments, pharmacological and others, some of them under investigation, some unproven and many disproven. So I was not surprised to find a study about Ginkgo biloba as a treatment for ADHD. I was surprised, however, to find this study in the German Journal of Child and Adolescent Psychiatry and Psychotherapy, officially published by the “German Society of Child and Adolescent Psychiatry and Psychotherapy“ (Deutsche Gesellschaft für Kinder- und Jugendpsychiatrie und Psychotherapie). The journal’s guidelines state that studies should provide new scientific results.
The study is called “Ginkgo biloba Extract EGb 761® in Children with ADHD“. EGb 761® is the key ingredient in “Tebonin®“, a herbal drug made by “Dr. Wilma Schwabe GmbH“. The abstract states:
“One possible treatment, at least for cognitive problems, might be the administration of Ginkgo biloba, though evidence is rare.This study tests the clinical efficacy of a Ginkgo biloba special extract (EGb 761®) (…) in children with ADHD (…).“
“Eine erfolgversprechende, bislang kaum untersuchte Möglichkeit zur Behandlung kognitiver Aspekte ist die Gabe von Ginkgo biloba. Ziel der vorliegenden Studie war die Prüfung klinischer Wirksamkeit (…) von Ginkgo biloba-Extrakt Egb 761® bei Kindern mit ADHS.“ (Taken from the English and German abstracts.)
The study sample (20!) was recruited among children who “did not tolerate or were unwilling“ to take MPH. The unwilling part struck me as problematic. There is likely a strong selection bias towards parents who are unwilling to give their children MPH. I guess it is not the children who are unwilling to take MPH, but the parents who are unwilling to administer it. At least some of these parents might be biased against MPH and might already favor CAMmodalities.
The authors state three main problems with “herbal therapy“ that require more empirical evidence: First of all the question of adverse reactions, which they claim occur in about 1% of cases with “some CAMs“ (mind you, not “herbal therapy“). Secondly, the question of drug interactions and thirdly, the lack of information physicians have about the CAMs their patients use.
A large part of the study is based on results of an EEG-protocol, which I choose to ignore, because the clinical results are too weak to give the EEG findings any clinical relevance.
Before looking at the study itself, let’s look at what is known about Ginkgo biloba as a drug. Ginkgo is best known for its use in patients with dementia, cognitive impairment und tinnitus. A Cochrane review from 2009 concluded:
“There is no convincing evidence that Ginkgo biloba is efficacious for dementia and cognitive impairment“ .
The authors of the current Study cite Sarris et al. (2011), a systematic review of complementary treatment of ADHD. Sarris et al. mention Salehi et al. (2010) who tested Ginkgo against MPH. MPH turned out to be much more effective than Ginkgo, but Sarris et al. argue that the duration of treatment (6 weeks) might have been too short to see the full effects of Ginkgo.
Given the above information it is unclear why Ginkgo is judged a “possible“ treatment, properly translated from German even “promising”, and why the authors state that Ginkgo has been “barely studied“.
In an unblinded, uncontrolled study with a sample likely to be biased toward the tested intervention, anything other than a positive result would be odd. In the treatment of autism there are several examples of implausible treatments that worked as long as parents knew that their children were getting the treatment, but didn’t after proper blinding (e.g. secretin).
This study’s aim was to test clinical efficacy, but the conclusion begins with how well tolerated Ginkgo was. The efficacy is mentioned subsequently: “Following administration, interrelated improvements on behavioral ratings of ADHD symptoms (…) were detected (…).“ But the way they where “detected“ is interesting. The authors used an established questionnaire (FBB-HKS) to let parents rate their children. Only the parents. The children and their teachers where not given the FBB-HKS-questionnaires, inspite of this being standard clinical practice (and inspite of giving children questionnaires to determine changes in quality of life, which were not found).
None of the three problems that the authors describe as important (adverse reactions, drug interactions, lack of information) can be answered by this study. I am no expert in statistics but it seems unlikely to me to meaningfully determine adverse effects in just 20 patients especially when adverse effects occur at a rate of 1%. The authors claim they found an incidence rate of 0,004% in “700 observation days“. Well, if they say so.
The authors conclude:
“Taken together, the current study provides some preliminary evidence that Ginkgo biloba Egb 761® seems to be well tolerated in the short term and may be a clinically useful treatment for children with ADHD. Double-blind randomized trials are required to clarify the value of the presented data.“
Given the available information mentioned earlier, one could have started with that conclusion and conducted a double blind RCT in the first place!
The trends of this preliminary open study may suggest that Ginkgo biloba Egb 761® might be considered as a complementary or alternative medicine for treating children with ADHD.“
So, why do I care? If preliminary evidence “may suggest“ that something “might be considered“ as a treatment? Because I think that this study does not answer any important questions or give us any new or useful knowledge. Following the journal’s guidelines, it should therefore not have been published. I also think it is an example of bad science. Bad not just because of the lack of critical thinking. It also adds to the misinformation about possible ADHD treatments spreading through the internet. The study was published in September. In November I found a website citing the study and calling it “clinical proof“ when it is not. But child psychiatrists will have to explain that to many parents, instead of talking about their children’s health.
I somehow got the impression that this study was more about marketing than about science. I wonder if Schwabe will help finance the necessary double-blind randomized trial…
Acupuncture seems to be as popular as never before – many conventional pain clinics now employ acupuncturists, for instance. It is probably true to say that acupuncture is one of the best-known types of all alternative therapies. Yet, experts are still divided in their views about this treatment – some proclaim that acupuncture is the best thing since sliced bread, while others insist that it is no more than a theatrical placebo. Consumers, I imagine, are often left helpless in the middle of these debates. Here are 7 important bits of factual information that might help you make up your mind, in case you are tempted to try acupuncture.
- Acupuncture is ancient; some enthusiast thus claim that it has ‘stood the test of time’, i. e. that its long history proves its efficacy and safety beyond reasonable doubt and certainly more conclusively than any scientific test. Whenever you hear such arguments, remind yourself that the ‘argumentum ad traditionem’ is nothing but a classic fallacy. A long history of usage proves very little – think of how long blood letting was used, even though it killed millions.
- We often think of acupuncture as being one single treatment, but there are many different forms of this therapy. According to believers in acupuncture, acupuncture points can be stimulated not just by inserting needles (the most common way) but also with heat, electrical currents, ultrasound, pressure, etc. Then there is body acupuncture, ear acupuncture and even tongue acupuncture. Finally, some clinicians employ the traditional Chinese approach based on the assumption that two life forces are out of balance and need to be re-balanced, while so-called ‘Western’ acupuncturists adhere to the concepts of conventional medicine and claim that acupuncture works via scientifically explainable mechanisms that are unrelated to ancient Chinese philosophies.
- Traditional Chinese acupuncturists have not normally studied medicine and base their practice on the Taoist philosophy of the balance between yin and yang which has no basis in science. This explains why acupuncture is seen by traditional acupuncturists as a ‘cure all’ . In contrast, medical acupuncturists tend to cite neurophysiological explanations as to how acupuncture might work. However, it is important to note that, even though they may appear plausible, these explanations are currently just theories and constitute no proof for the validity of acupuncture as a medical intervention.
- The therapeutic claims made for acupuncture are legion. According to the traditional view, acupuncture is useful for virtually every condition affecting mankind; according to the more modern view, it is effective for a relatively small range of conditions only. On closer examination, the vast majority of these claims can be disclosed to be based on either no or very flimsy evidence. Once we examine the data from reliable clinical trials (today several thousand studies of acupuncture are available – see below), we realise that acupuncture is associated with a powerful placebo effect, and that it works better than a placebo only for very few (some say for no) conditions.
- The interpretation of the trial evidence is far from straight forward: most of the clinical trials of acupuncture originate from China, and several investigations have shown that very close to 100% of them are positive. This means that the results of these studies have to be taken with more than a small pinch of salt. In order to control for patient-expectations, clinical trials can be done with sham needles which do not penetrate the skin but collapse like miniature stage-daggers. This method does, however, not control for acupuncturists’ expectations; blinding of the therapists is difficult and therefore truly double (patient and therapist)-blind trials of acupuncture do hardly exist. This means that even the most rigorous studies of acupuncture are usually burdened with residual bias.
- Few acupuncturists warn their patients of possible adverse effects; this may be because the side-effects of acupuncture (they occur in about 10% of all patients) are mostly mild. However, it is important to know that very serious complications of acupuncture are on record as well: acupuncture needles can injure vital organs like the lungs or the heart, and they can introduce infections into the body, e. g. hepatitis. About 100 fatalities after acupuncture have been reported in the medical literature – a figure which, due to lack of a monitoring system, may disclose just the tip of an iceberg.
- Given that, for the vast majority of conditions, there is no good evidence that acupuncture works beyond a placebo response, and that acupuncture is associated with finite risks, it seems to follow that, in most situations, the risk/benefit balance for acupuncture fails to be convincingly positive.