scientific misconduct
As promised, I will try with this post to explain my reservations regarding the new meta-analysis suggesting that individualised homeopathic remedies are superior to placebos. Before I start, however, I want to thank all those who have commented on various issues; it is well worth reading the numerous and diverse comments.
To remind us of the actual meta-analysis, it might be useful to re-publish its abstract (the full article is also available online):
BACKGROUND:
A rigorous and focused systematic review and meta-analysis of randomised controlled trials (RCTs) of individualised homeopathic treatment has not previously been undertaken. We tested the hypothesis that the outcome of an individualised homeopathic treatment approach using homeopathic medicines is distinguishable from that of placebos.
METHODS:
The review’s methods, including literature search strategy, data extraction, assessment of risk of bias and statistical analysis, were strictly protocol-based. Judgment in seven assessment domains enabled a trial’s risk of bias to be designated as low, unclear or high. A trial was judged to comprise ‘reliable evidence’ if its risk of bias was low or was unclear in one specified domain. ‘Effect size’ was reported as odds ratio (OR), with arithmetic transformation for continuous data carried out as required; OR > 1 signified an effect favouring homeopathy.
RESULTS:
Thirty-two eligible RCTs studied 24 different medical conditions in total. Twelve trials were classed ‘uncertain risk of bias’, three of which displayed relatively minor uncertainty and were designated reliable evidence; 20 trials were classed ‘high risk of bias’. Twenty-two trials had extractable data and were subjected to meta-analysis; OR = 1.53 (95% confidence interval (CI) 1.22 to 1.91). For the three trials with reliable evidence, sensitivity analysis revealed OR = 1.98 (95% CI 1.16 to 3.38).
CONCLUSIONS:
Medicines prescribed in individualised homeopathy may have small, specific treatment effects. Findings are consistent with sub-group data available in a previous ‘global’ systematic review. The low or unclear overall quality of the evidence prompts caution in interpreting the findings. New high-quality RCT research is necessary to enable more decisive interpretation.
Since my team had published an RCTs of individualised homeopathy, it seems only natural that my interest focussed on why the study (even though identified by Mathie et al) had not been included in the meta-analysis. Our study had provided no evidence that adjunctive homeopathic remedies, as prescribed by experienced homeopathic practitioners, are superior to placebo in improving the quality of life of children with mild to moderate asthma in addition to conventional treatment in primary care.
I was convinced that this trial had been rigorous and thus puzzled why, despite receiving ‘full marks’ from the reviewers, they had not included it in their meta-analysis. I thus wrote to Mathie, the lead author of the meta-analysis, and he explained: For your trial (White et al. 2003), under domain V of assessment, we were unable to extract data for meta-analysis, and so it was attributed high risk of bias, as specified by the Cochrane judgmental criteria. Our designated main outcome was the CAQ, for which we needed to know (or could at least estimate) a mean and SD for both the baseline and the end-point of the study. Since your paper reported only the change from baseline in Table 3 or in the main text, it is not possible to derive the necessary end-point for analysis.
It took a while and several further emails until I understood: our study did report both the primary (Table 2 quality of life) and secondary outcome measure (Table 3 severity of symptoms). The primary outcome measure was reported in full detail such that a meta-analysis would have been possible. The secondary outcome measure was also reported but not in full detail, and the data provided by us would not lend themselves to meta-analyses. By electing not our primary but our secondary outcome measure for their meta-analysis, Mathie et al were able to claim that they were unable to use our study and reject it for their meta-analysis.
Why did they do that?
The answer is simple: in their methods section, they specify that they used outcome measures “based on a pre-specified hierarchical list in order of greatest to least importance, recommended by the WHO“. This, I would argue is deeply flawed: the most important outcome measure of a study is usually the one for which the study was designed, not the one that some guys at the WHO feel might be important (incidentally, the WHO list was never meant to be applied to meta-analyses in that way).
By following rigidly their published protocol, the authors of the meta-analysis managed to exclude our negative trial. Thus they did everything right – or did they?
Well, I think they committed several serious mistakes.
- Firstly, they wrote the protocol, which forced them to exclude our study. Following a protocol is not a virtue in itself; if the protocol is nonsensical it even is the opposite. Had they proceeded as is normal in such cases and used our primary outcome measure in their meta-analyses, it is most likely that their overall results would not have been in favour of homeopathy.
- Secondly, they awarded our study a malus point for the criterium ‘selective outcome reporting’. This is clearly a wrong decision: we did report the severity-outcome, albeit not in sufficient detail for their meta-analysis. Had they not committed this misjudgment, our RCT would have been the only one with an ‘A’ rating. This would have very clearly highlighted the nonsense of excluding the best-rated trial from meta-analysis.
There are several other oddities as well. For instance, Mathie et al judge our study to be NOT free of vested interest. I asked Mathie why they had done this and was told it is because we accepted free trial medication from a homeopathic pharmacy. I would argue that my team was far less plagued by vested interest than the authors of their three best (and of course positive) trials who, as I happen to know, are consultants for homeopathic manufacturers.
And all of this is just in relation to our own study. Norbert Aust has uncovered similar irregularities with other trials and I take the liberty of quoting his comments posted previously again here:
I have reason to believe that this review and metaanalysis in biased in favor of homeopathy. To check this, I compared two studies (1) Jacobs 1994 about the treatment of childhood diarrhea in Nicaragua, (2) Walach 1997 about homeopathic threatment of headaches. The Jacobs study is one of the three that provided ‘reliable evidence’, Walach’s study earned a poor C2.2 rating and was not included in the meta-analyses. Jacobs’ results were in favour of homeopathy, Walach’s not.
For the domains where the rating of Walach’s study was less than that of the Jacobs study, please find citations from the original studies or my short summaries for the point in question.
Domain I: Sequence generation:
Walach:
“The remedy selected was then mailed to a notary public who held a stock of placebos. The notary threw a dice and mailed either the homeopathic remedy or an appropriate placebo. The notary was provided with a blank randomisation list.”
Rating: UNCLEAR (Medium risk of bias)
Jacobs:
“For each of these medications, there was a box of tubes in sequentially numbered order which had been previously randomized into treatment or control medication using a random numbers table in blocks of four”
Rating: YES (Low risk of bias)
Domain IIIb: Blinding of outcome assessor
Walach:
“The notary was provided with a blank randomization list which was an absolutely unique document. It was only handed out after the biometrician (WG) had deposited all coded original data as a printout at the notary’s office. (…) Data entry was performed blindly by personnel not involved in the study. ”
Rating: UNCLEAR (Medium risk of bias)
Jacobs:
“All statistical analyses were done before breaking the randomisation code, using the program …”
Rating: YES (Low risk of bias)
Domain V: Selective outcome reporting
Walach:
Study protocol was published in 1991 prior to enrollment of participants, all primary outcome variables were reported with respect to all participants and the endpoints.
Rating: NO (high risk of bias)
Jacobs:
No prior publication of protocol, but a pilot study exists. However this was published in 1993 only after the trial was performed in 1991. Primary outcome defined (duration of diarrhea), reported but table and graph do not match, secondary outcome (number of unformed stools on day 3) seems defined post hoc, for this is the only one point in time, this outcome yielded a significant result.
Rating: YES (low risk of bias)
Domain VI: Other sources of bias:
Walach:
Rating: NO (high risk of bias), no details given
Jacobs:
Imbalance of group properties (size, weight and age of children), that might have some impact on course of disease, high impact of parallel therapy (rehydration) by far exceeding effect size of homeopathic treatment
Rating: YES (low risk of bias), no details given
In a nutshell: I fail to see the basis for the different ratings in the studies themselves. I assume bias of the authors of the review.
Conclusion
So, what about the question posed in the title of this article? The meta-analysis is clearly not a ‘proof of concept’. But is it proof for misconduct? I asked Mathie and he answered as follows: No, your statement does not reflect the situation at all. As for each and every paper, we selected the main outcome measure for your trial using the objective WHO classification approach (in which quality of life is clearly of lower rank than severity). This is all clearly described in our prospective protocol. Under no circumstances did we approach this matter retrospectively, in the way you are implying.
Some nasty sceptics might have assumed that the handful of rigorous studies with negative results were well-known to most researchers of homeopathy. In this situation, it would have been hugely tempting to write the protocol such that these studies must be excluded. I am thrilled to be told that the authors of the current new meta-analysis (who declared all sorts of vested interests at the end of the article) resisted this temptation.
One thing that has often irritated me – alright, I admit it: sometimes it even infuriated me – is the pseudoscientific language of authors writing about alternative medicine. Reading publications in this area often seems to me like being in the middle of a game of ‘bullshit bingo’ (I am afraid that some of the commentators on this blog have importantly contributed to this phenomenon). In an article of 2004, I once discussed this issue in some detail and concluded that “… pseudo-scientific language … can be seen as an attempt to present nonsense as science…this misleads patients and can thus endanger their health…” For this paper, I had focussed on examples from the ‘bioresonance’- literature – more by coincidence than by design, I should add. I could have selected any other alternative treatment or diagnostic method; the use of pseudoscientific language is truly endemic in alternative medicine.
To give you a little flavour, here is the section of my 2004 paper where I used 5 quotes from recent articles on bioresonance and added a brief comment after each of them.
Quote No. 1
‘The biophysical control processes are superordinate to the biochemical processes. In the same way as the atomic processes result in chemical compounds the ultrafine biocommunication results in the biochemical processes. Control signals have an electromagnetic quality. Disturbing signals or ‘disturbing energies’ also have an electromagnetic quality. This is the reason why they can, for example, be conducted through cables and transformed into therapy signals by means of sophisticated electronic devices. The purpose is to clear the pathological part of the signals.’
Here the author uses highly technical language which, at first, sounds very complicated and scientific. However, after a second read, one is bound to discover that the words hide more than they reveal. In particular, the scientific tone distracts from the lack of logic in the argument. The basic message, once the pseudoscientific veneer is stripped away, seems to be the following. Living systems display electromagnetic phenomena. The electromagnetic energies that they rely upon can make us ill. The energies can also be transferred into an electronic instrument where they can be changed so that they don’t cause any more harm.
Quote No. 2
‘A very important advantage of the BICOM device as compared to the original form of the MORA-therapy in paediatry is the possibility to reduce the oscillation, a fact which meets much better the reaction pattern of the child and gives better results’ [3].
This paragraph essentially states that the BICOM instrument can change (the frequency or amplitude of) some sort of (electromagnetic) wave. We are told that, for children, this is preferable because of the way children tend to react. This would then be more effective.
Quote No. 3
‘The question how causative the Bioresonanz-Therapy can be must be answered in a differentiated way. The BR is in the first place effective on the informative level, which means on the ultrafine biokybernetical regulation level of the organism. This also includes the time factor and with that the functional aspect, and thus it influences the material-biochemical area of the body. The BRT is in comparison to other therapy procedures very high on the scale of causativeness, but it still remains in the physical level, and does not reach into the spiritual area. The freeing of the patient from his diseases can self evidently also lead to a change and improvement of conduct and attitudes and to a general wellbeing of the patient’ [4].
This amazing statement is again not easy to understand. If my reading is correct, the author essentially wants to tell us that BR interferes with the flow of information within organisms. The process is time-dependent and therefore affects function, physical and biochemical properties. Compared to other treatments, BR is more causative without affecting our spiritual sphere. As BR cures a disease, it can also change behaviour, attitudes and wellbeing.
Quote No. 4
‘MORA therapy is an auto-iso-therapy using the patient’s own vibrations in a wide range of the electromagnetic spectrum. Strictly speaking, we have hyperwaves in a six-dimensional cosmos with two hidden parameters (as predicted by Albert Einstein and others). Besides the physical plane there are six other planes of existence and the MORA therapy works in the biological plane, a region called the M-field, according to Sheldrake and Burkhard Heim’ [5].
Here we seem to be told that the MORA therapy is a selftreatment using the body’s own resources, namely a broad range of electromagnetic waves. These waves are hyperwaves in 6 dimensions and their existence has already been predicted by Einstein. Six (or 7?) planes of existence seem to have been discovered and the MORA therapy is operative in one of them.
Quote No. 5
‘The author presents an overall medical conception of the world between mass maximum and masslessness and completes it with the pair of concepts of subjectivity/objectivity. Three test procedures of the bioelectronic function diagnostics are presented and incorporated in addition to other procedures in this conception of the world. Therefore, in the sense of a holistic medicine, there is a useful indication for every medical procedure, because there are different objectives associated with each procedure. A one-sided assessment of the procedures does not do justice to the human being as a whole’ [6].
This author introduces a new concept of the world between maxima and minima of mass or objectivity. He has developed 3 tests of BR diagnosis that fit into the new concept. Therefore, holistically speaking, any therapy is good for something because each may have a different aim. One-sided assessments of such holistic treatments are too narrow bearing in mind the complexity of a human being.
The danger of pseudoscientific language in health care is obvious: it misleads patients, consumers, journalists, politicians, and everyone else (perhaps even some of the original authors?) into believing that nonsense is credible; to express it more bluntly: it is a method of cheating the unsuspecting public. Yes, the way I see it, it is a form of health fraud. Thus it leads to wrong therapeutic decisions and endangers public health.
I could easily get quite cross with the many authors who publish such drivel. But let’s not allow them to spoil our day; let’s take a different approach: let’s try to have some fun.
I herewith invite my readers to post quotes in the comments section of the most extraordinary excesses of pseudoscientific language that they have come across. If the result is sufficiently original, I might try to design a new BULLSHIT BINGO with it.
Rigorous research into the effectiveness of a therapy should tell us the truth about the ability of this therapy to treat patients suffering from a given condition — perhaps not one single study, but the totality of the evidence (as evaluated in systematic reviews) should achieve this aim. Yet, in the realm of alternative medicine (and probably not just in this field), such reviews are often highly contradictory.
A concrete example might explain what I mean.
There are numerous systematic reviews assessing the effectiveness of acupuncture for fibromyalgia syndrome (FMS). It is safe to assume that the authors of these reviews have all conducted comprehensive searches of the literature in order to locate all the published studies on this subject. Subsequently, they have evaluated the scientific rigor of these trials and summarised their findings. Finally they have condensed all of this into an article which arrives at a certain conclusion about the value of the therapy in question. Understanding this process (outlined here only very briefly), one would expect that all the numerous reviews draw conclusions which are, if not identical, at least very similar.
However, the disturbing fact is that they are not remotely similar. Here are two which, in fact, are so different that one could assume they have evaluated a set of totally different primary studies (which, of course, they have not).
One recent (2014) review concluded that acupuncture for FMS has a positive effect, and acupuncture combined with western medicine can strengthen the curative effect.
Another recent review concluded that a small analgesic effect of acupuncture was present, which, however, was not clearly distinguishable from bias. Thus, acupuncture cannot be recommended for the management of FMS.
How can this be?
By contrast to most systematic reviews of conventional medicine, systematic reviews of alternative therapies are almost invariably based on a small number of primary studies (in the above case, the total number was only 7 !). The quality of these trials is often low (all reviews therefore end with the somewhat meaningless conclusion that more and better studies are needed).
So, the situation with primary studies of alternative therapies for inclusion into systematic reviews usually is as follows:
- the number of trials is low
- the quality of trials is even lower
- the results are not uniform
- the majority of the poor quality trials show a positive result (bias tends to generate false positive findings)
- the few rigorous trials yield a negative result
Unfortunately this means that the authors of systematic reviews summarising such confusing evidence often seem to feel at liberty to project their own pre-conceived ideas into their overall conclusion about the effectiveness of the treatment. Often the researchers are in favour of the therapy in question – in fact, this usually is precisely the attitude that motivated them to conduct a review in the first place. In other words, the frequently murky state of the evidence (as outlined above) can serve as a welcome invitation for personal bias to do its effect in skewing the overall conclusion. The final result is that the readers of such systematic reviews are being misled.
Authors who are biased in favour of the treatment will tend to stress that the majority of the trials are positive. Therefore the overall verdict has to be positive as well, in their view. The fact that most trials are flawed does not usually bother them all that much (I suspect that many fail to comprehend the effects of bias on the study results); they merely add to their conclusions that “more and better trials are needed” and believe that this meek little remark is sufficient evidence for their ability to critically analyse the data.
Authors who are not biased and have the necessary skills for critical assessment, on the other hand, will insist that most trials are flawed and therefore their results must be categorised as unreliable. They will also emphasise the fact that there are a few reliable studies and clearly point out that these are negative. Thus their overall conclusion must be negative as well.
In the end, enthusiasts will conclude that the treatment in question is at least promising, if not recommendable, while real scientists will rightly state that the available data are too flimsy to demonstrate the effectiveness of the therapy; as it is wrong to recommend unproven treatments, they will not recommend the treatment for routine use.
The difference between the two might just seem marginal – but, in fact, it is huge: IT IS THE DIFFERENCE BETWEEN MISLEADING PEOPLE AND GIVING RESPONSIBLE ADVICE; THE DIFFERENCE BETWEEN VIOLATING AND ADHERING TO ETHICAL STANDARDS.
‘Healing, hype or harm? A critical analysis of complementary or alternative medicine’ is the title of a book that I edited and that was published in 2008. Its publication date coincided with that of ‘Trick or Treatment?’ and therefore the former was almost completely over-shadowed by the latter. Consequently few people know about it. This is a shame, I think, and this post is dedicated to encouraging my readers to have a look at ‘Healing, hype or harm?’
One reviewer commented on Amazon about this book as follows: Vital and informative text that should be read by everyone alongside Ben Goldacre’s ‘Bad Science’ and Singh and Ernt’s ‘Trick or Treatment’. Everyone should be able to made informed choices about the treatments that are peddled to the desperate and gullible. As Tim Minchin famously said ‘What do you call Alternative Medicine that has been proved to work? . . . Medicine!’
This is high praise indeed! But I should not omit the fact that others have commented that they were appalled by our book and found it “disappointing and unsettling”. This does not surprise me in the least; after all, alternative medicine has always been a divisive subject.
The book was written by a total of 17 authors and covers many important aspects of alternative medicine. Some of its most famous contributors are Michael Baum, Gustav Born, David Colquhoun, James Randi and Nick Ross. Some of the most important subjects include:
- Compassion
- Quackademia
- Impartiality
- Ethics
- Politics
- Holism
- Vitalism
- Placebo
As already mentioned, our book is already 6 years old; however, this does not mean that it is now out-dated. The subject areas were chosen such that it will be timely for a long time to come. Nor does this book reflect one single point of view; as it was written by over a dozen different experts with vastly different backgrounds, it offers an entire spectrum of views and attitudes. It is, in a word, a book that stimulates critical thinking and thoughtful analysis.
I sincerely think you should have a look at it… and, in case you think I am hoping to maximise my income by telling you all this: all the revenues from this book go to charity.
After the usually challenging acute therapy is behind them, cancer patients are often desperate to find a therapy that might improve their wellbeing. At that stage they may suffer from a wide range of symptoms which can seriously limit their quality of life. Any treatment that can be shown to restore them to their normal mental and physical health would be more than welcome.
Most homeopaths believe that their remedies can do just that, particularly if they are tailored not to the disease but to the individual patient. Sadly, the evidence that this might be so is almost non-existent. Now, a new trial has become available; it was conducted by Jennifer Poole, a chartered psychologist and registered homeopath, and researcher and teacher at Nemeton Research Foundation, Romsey.
The aim of this study was to explore the benefits of a three-month course of individualised homeopathy (IH) for survivors of cancer. Fifteen survivors of any type of cancer were recruited from a walk-in cancer support centre. Conventional treatment had to have taken place within the last three years. Patients saw a homeopath who prescribed IH. After three months of IH, they scored their total, physical and emotional wellbeing using the Functional Assessment of Chronic Illness Therapy for Cancer (FACIT-G). The results show that 11 of the 14 women had statistically positive outcomes for emotional, physical and total wellbeing.
The conclusions of the author are clear: Findings support previous research, suggesting CAM or IH could be beneficial for survivors of cancer.
This article was published in the NURSING TIMES, and the editor added a footnote informing us that “This article has been double-blind “.
I find this surprising. A decent peer-review should have picked up the point that a study of that nature cannot possibly produce results which tell us anything about the benefits of IH. The reasons for this are fairly obvious:
- there was no control group,
- therefore the observed outcomes are most likely due to 1) natural history, 2) placebo, 3) regression towards the mean and 4) social desirability; it seems most unlikely that IH had anything to do with the result
- the sample size was tiny,
- the patients elected to receive IH which means that had high expectations of a positive outcome,
- only subjective outcome measures were used,
- there is no good previous research suggesting that IH benefits cancer patients.
On the last point, a recent systematic review showed that the studies available on this topic had mixed results either showing a significantly greater improvement in QOL in the intervention group compared to the control group, or no significant difference between groups. The authors concluded that there existed significant gaps in the evidence base for the effectiveness of CAM on QOL in cancer survivors. Further work in this field needs to adopt more rigorous methodology to help support cancer survivors to actively embrace self-management and effective CAMs, without recommending inappropriate interventions which are of no proven benefit.
All this new study might tell us is that IH did not seem to harm these patients – but even this finding is not certain; to be sure, we would need to include many more patients. Any conclusions about the effectiveness of IH are totally unwarranted. But are there ANY generalizable conclusions that can be drawn from this article? Yes, I can think of a few:
- Some cancer patients can be persuaded to try the most implausible treatments.
- Some journals will publish any rubbish.
- Some peer-reviewers fail to spot the most obvious defects.
- Some ‘researchers’ haven’t got a clue.
- The attempts of misleading us about the value of homeopathy are incessant.
One might argue that this whole story is too trivial for words; who cares what dodgy science is published in the NURSING TIMES? But I think it does matter – not so much because of this one silly article itself, but because similarly poor research with similarly ridiculous conclusions is currently published almost every day. Subsequently it is presented to the public as meaningful science heralding important advances in medicine. It matters because this constant drip of bogus research eventually influences public opinion and determines far-reaching health care decisions.
Dodgy science abounds in alternative medicine; this is perhaps particularly true for homeopathy. A brand-new trial seems to confirm this view.
The aim of this study was to test the hypothesis that homeopathy (H) enhances the effects of scaling and root planing (SRP) in patients with chronic periodontitis (CP).
The researchers, dentists from Brazil, randomised 50 patients with CP to one of two treatment groups: SRP (C-G) or SRP + H (H-G). Assessments were made at baseline and after 3 and 12 months of treatment. The local and systemic responses to the treatments were evaluated after one year of follow-up. The results showed that both groups displayed significant improvements, however, the H-G group performed significantly better than C-G group.
The authors concluded that homeopathic medicines, as an adjunctive to SRP, can provide significant local and systemic improvements for CP patients.
Really? I am afraid, I disagree!
Homeopathic medicines might have nothing whatsoever to do with this result. Much more likely is the possibility that the findings are caused by other factors such as:
- placebo-effects,
- patients’ expectations,
- improved compliance with other health-related measures,
- the researchers’ expectations,
- the extra attention given to the patients in the H-G group,
- disappointment of the C-G patients for not receiving the additional care,
- a mixture of all or some of the above.
I should stress that it would not have been difficult to plan the study in such a way that these factors were eliminated as sources of bias or confounding. But this study was conducted according to the A+B versus B design which we have discussed repeatedly on this blog. In such trials, A is the experimental treatment (homeopathy) and B is the standard care (scaling and root planning). Unless A is an overtly harmful therapy, it is simply not conceivable that A+B does not generate better results than B alone. The simplest way to comprehend this argument is to imagine A and B are two different amounts of money: it is impossible that A+B is not more that B!
It is unclear to me what relevant research question such a study design actually does answer (if anyone knows, please tell me). It seems obvious, however, that it cannot test the hypothesis that homeopathy (H) enhances the effects of scaling and root planing (SRP). This does not necessarily mean that the design is necessarily useless. But at the very minimum, one would need an adequate research question (one that matches this design) and adequate conclusions based on the findings.
The fact that the conclusions drawn from a dodgy trial are inadequate and misleading could be seen as merely a mild irritation. The facts that, in homeopathy, such poor science and misleading conclusions emerge all too regularly, and that journals continue to publish such rubbish are not just mildly irritating; they are annoying and worrying – annoying because such pseudo-science constitutes an unethical waste of scarce resources; worrying because it almost inevitably leads to wrong decisions in health care.
There must be well over 10 000 clinical trials of acupuncture; Medline lists ~5 000, and many more are hidden in the non-Medline listed literature. That should be good news! Sadly, it isn’t.
It should mean that we now have a pretty good idea for what conditions acupuncture is effective and for which illnesses it does not work. But we don’t! Sceptics say it works for nothing, while acupuncturists claim it is a panacea. The main reason for this continued controversy is that the quality of the vast majority of these 10 000 studies is not just poor, it is lousy.
“Where is the evidence for this outraging statement???” – I hear the acupuncture-enthusiasts shout. Well, how about my own experience as editor-in-chief of FACT? No? Far too anecdotal?
How about looking at Cochrane reviews then; they are considered to be the most independent and reliable evidence in existence? There are many such reviews (most, if not all [co-]authored by acupuncturists) and they all agree that the scientific rigor of the primary studies is fairly awful. Here are the crucial bits of just the last three; feel free to look for more:
All of the studies had a high risk of bias…
All included trials had a high risk of bias…
The studies were not judged to be free from bias…
Or how about providing an example? Good idea! Here is a new trial which could stand for numerous others:
This study was performed to compare the efficacy of acupuncture versus corticosteroid injection for the treatment of Quervain’s tendosynovitis (no, you do not need to look up what condition this is for understanding this post). Thirty patients were treated in two groups. The acupuncture group received 5 acupuncture sessions of 30 minutes duration. The injection group received one methylprednisolone acetate injection in the first dorsal compartment of the wrist. The degree of disability and pain was evaluated by using the Quick Disabilities of the Arm, Shoulder, and Hand (Q-DASH) scale and the Visual Analogue Scale (VAS) at baseline and at 2 weeks and 6 weeks after the start of treatment. The baseline means of the Q-DASH and the VAS scores were 62.8 and 6.9, respectively. At the last follow-up, the mean Q-DASH scores were 9.8 versus 6.2 in the acupuncture and injection groups, respectively, and the mean VAS scores were 2 versus 1.2. Thus there were short-term improvements of pain and function in both groups.
The authors drew the following conclusions: Although the success rate was somewhat higher with corticosteroid injection, acupuncture can be considered as an alternative option for treatment of De Quervain’s tenosynovitis.
The flaws of this study are exemplary and numerous:
- This should have been a study that compares two treatments – the technical term is ‘equivalence trial – and such studies need to be much larger to produce a meaningful result. Small sample sizes in equivalent trials will always make the two treatments look similarly effective, even if one is a pure placebo.
- There is no gold standard treatment for this condition. This means that a comparative trial makes no sense at all. In such a situation, one ought to conduct a placebo-controlled trial.
- There was no blinding of patients; therefore their expectation might have distorted the results.
- The acupuncture group received more treatments than the injection group; therefore the additional attention might have distorted the findings.
- Even if the results were entirely correct, one cannot conclude from them that acupuncture was effective; the notion that it was similarly ineffective as the injections is just as warranted.
These are just some of the most fatal flaws of this study. The sad thing is that similar criticisms can be made for most of the 10 000 trials of acupuncture. But the point here is not to nit-pick nor to quack-bust. My point is a different and more serious one: fatally flawed research is not just a ‘poor show’, it is unethical because it is a waste of scarce resources and, even more importantly, an abuse of patients for meaningless pseudo-science. All it does is it misleads the public into believing that acupuncture might be good for this or that condition and consequently make wrong therapeutic decisions.
In acupuncture (and indeed in most alternative medicine) research, the problem is so extremely wide-spread that it is high time to do something about it. Journal editors, peer-reviewers, ethics committees, universities, funding agencies and all others concerned with such research have to work together so that such flagrant abuse is stopped once and for all.
When someone has completed a scientific project, it is customary to publish it [‘unpublished science is no science’, someone once told me many years ago]. To do so, he needs to write it up and submit it to a scientific journal. The editor of this journal will then submit it to a process called ‘peer review’.
What does ‘peer review’ entail? Well, it means that 2-3 experts are asked to critically assess the paper in question, make suggestions as to how it can be improved and submit a recommendation as to whether or not the article deserves to be published.
Peer review has many pitfalls but, so far, nobody has come up with a solution that is convincingly better. Many scientists are under pressure to publish [‘publish or perish’], and therefore some people resort to cheating. A most spectacular case of fraudulent peer review has been reported recently in this press release:
SAGE statement on Journal of Vibration and Control
London, UK (08 July 2014) – SAGE announces the retraction of 60 articles implicated in a peer review and citation ring at the Journal of Vibration and Control (JVC). The full extent of the peer review ring has been uncovered following a 14 month SAGE-led investigation, and centres on the strongly suspected misconduct of Peter Chen, formerly of National Pingtung University of Education, Taiwan (NPUE) and possibly other authors at this institution.
In 2013 the then Editor-in-Chief of JVC, Professor Ali H. Nayfeh,and SAGE became aware of a potential peer review ring involving assumed and fabricated identities used to manipulate the online submission system SAGE Track powered by ScholarOne Manuscripts™. Immediate action was taken to prevent JVC from being exploited further, and a complex investigation throughout 2013 and 2014 was undertaken with the full cooperation of Professor Nayfeh and subsequently NPUE.
In total 60 articles have been retracted from JVC after evidence led to at least one author or reviewer being implicated in the peer review ring. Now that the investigation is complete, and the authors have been notified of the findings, we are in a position to make this statement.
While investigating the JVC papers submitted and reviewed by Peter Chen, it was discovered that the author had created various aliases on SAGE Track, providing different email addresses to set up more than one account. Consequently, SAGE scrutinised further the co-authors of and reviewers selected for Peter Chen’s papers, these names appeared to form part of a peer review ring. The investigation also revealed that on at least one occasion, the author Peter Chen reviewed his own paper under one of the aliases he had created.
Unbelievable? Perhaps, but sadly it is true; some scientists seem to be criminally ingenious when it comes to getting their dodgy articles into peer-reviewed journals.
And what does this have to do with ALTERNATIVE MEDICINE, you may well ask. The Journal of Vibration and Control is not even medical and certainly would never consider publishing articles on alternative medicine. Such papers go to one of the many [I estimate more that 1000] journals that cover either alternative medicine in general or any of the modalities that fall under this wide umbrella. Most of these journals, of course, pride themselves with being peer-reviewed – and, at least nominally, that is correct.
I have been on the editorial board of most of the more important journals in alternative medicine, and I cannot help thinking that their peer review process is not all that dissimilar from the fraudulent scheme set up by Peter Chen and disclosed above. What happens in alternative medicine is roughly as follows:
- a researcher submits a paper for publication,
- the editor sends it out for peer review,
- the peer reviewers are either those suggested by the original author or members of the editorial board of the journal,
- in either case, the reviewers are more than likely to be uncritical and recommend publication,
- in the end, peer review turns out to be a farcical window dressing exercise with no consequence,
- thus even very poor research and pseudo-research are being published abundantly.
The editorial boards of journals of alternative medicine tend to be devoid of experts who are critical about the subject at hand. If you think that I am exaggerating, have a look at the editorial board members of ‘HOMEOPATHY’ (or any other journal of alternative medicine) and tell me who might qualify as a critic of homeopathy. When the editor, Peter Fisher, recently fired me from his board because he felt I had tarnished the image of homeopathy, this panel lost the only person who understood the subject matter and, at the same time, was critical about it (the fact that the website still lists me as an editorial board member is merely a reflection of how slow things are in the world of homeopathy: Fisher fired me more than a year ago).
The point I am trying to make is simple: peer review is never a perfect method but when it is set up to be deliberately uncritical, it cannot possibly fulfil its function to prevent the publication of dodgy research. In this case, the quality of the science will be inadequate and generate false-positive messages that mislead the public.
A remarkable article about homeopathy and immunisation entitled THE IMMUNISATION DILEMMA came to my attention recently. Its abstract promised: “evidence quantifying the effectiveness of vaccination and HP (homeoprophylaxis) will be examined. New international research describing and analysing HP interventions will be reported. An evidence-based conclusion will be reached.”
Sounds interesting? Let’s see what the article really offers. Here is the relevant text:
…evidence does exist to support claims regarding the effectiveness of homeopathic immunisation is undeniable.
I was first invited to visit Cuba in December 2008 to present at an international conference hosted by the Finlay Institute, which is a W. H. O.-accredited vaccine manufacturer. The Cubans described their use of HP to control an outbreak of leptospirosis (Weilʼs syndrome – a potentially fatal, water-born bacterial disease) in 2007 among the residents of the three eastern provinces which were most severely damaged by a severe hurricane – over 2.2 million people [7]. 2008 was an even worse year involving three hurricanes, and the countryʼs food production was only just recovering at the time of the conference. The HP program had been repeated in 2008, but data was not available at the conference regarding that intervention.
I revisited Cuba in 2010 and 2012, each time to work with the leader of the HP interventions, Dr. Bracho, to analyse the data available. Dr. Bracho is not a homeopath; he is a published and internationally recognised expert in the manufacture of vaccine adjuvants. He worked in Australia at Flinders University during 2004 with a team trying to develop an antimalarial vaccine.
In 2012 we accessed the raw leptospirosis surveillance data, comprising weekly reports from 15 provinces over 9 years (2000 to 2008) reporting 21 variables. This yielded a matrix with 147 420 possible entries. This included data concerning possible confounders, such as vaccination and chemoprophylaxis, which allowed a careful examination of possible distorting effects. With the permission of the Cubans, I brought this data back to Australia and it is being examined by mathematicians at an Australian university to see what other information can be extracted. Clearly, there is objective data supporting claims regarding the effectiveness of HP.
The 2008 result was remarkable, and could only be explained by the effectiveness of the HP intervention. Whilst the three hurricanes caused immense damage throughout the country it was again worse in the east, yet the three homeopathically immunised provinces experienced a negligible increase in cases whilst the rest of the country showed significant increases until the dry season in January 2009 [8].
This is but one example – there are many more. It is cited to show that there is significant data available, and that orthodox scientists and doctors have driven the HP interventions, in the Cuban case. Many people internationally now know this, so once again claims by orthodox authorities that there is no evidence merely serves to show that either the authorities are making uninformed/unscientific statements, or that they are aware but are intentionally withholding information. Either way, confidence is destroyed and leads to groups of people questioning what they are told…
Final Conclusions
The attacks against homeopathy in general and HP in particular will almost certainly continue. If we can achieve a significant level of agreement then we would be able to answer challenges to HP with a single, cohesive, evidence-based, and generally united response. This would be a significant improvement to the existing situation.
Reference 7 is the following article: Bracho G, Varela E, Fernández R et al. Large-scale application of highly-diluted bacteria for Leptospirosis epidemic control. Homeopathy 2010; 99: 156-166. The crucial bit if this paper are as follows:
A homeoprophylactic formulation was prepared from dilutions of four circulating strains of Leptospirosis. This formulation was administered orally to 2.3 million persons at high risk in an epidemic in a region affected by natural disasters. The data from surveillance were used to measure the impact of the intervention by comparing with historical trends and non-intervention regions.
After the homeoprophylactic intervention a significant decrease of the disease incidence was observed in the intervention regions. No such modifications were observed in non-intervention regions. In the intervention region the incidence of Leptospirosis fell below the historic median. This observation was independent of rainfall.
The homeoprophylactic approach was associated with a large reduction of disease incidence and control of the epidemic. The results suggest the use of HP as a feasible tool for epidemic control, further research is warranted.
The paper thus describes little more than an observational study. It shows that one region was less affected than another. I think it is quite clear that this could have many reasons which are unrelated to the homeopathic immunisation. Even the authors are cautious and speak in their conclusions not of a causal effect but of an “association”.
The 2012 data cited in the text remains unpublished; until it is available for public scrutiny, it is impossible to confirm that it is sound and meaningful.
Reference 8 refers to this article: Golden I, Bracho G. Adaptability of homœoprophylaxis in endemic, epidemic and stable background conditions. Homœopathic Links 2009; 22: 211-213. I have no access to this paper (if someone does, please fill us in) but, judging from both its title and the way it is described in the text, it does not seem to show reliable data about the efficacy of homeopathic immunisation.
So, is it true that “evidence does exist to support claims regarding the effectiveness of homeopathic immunisation”?
I do not think so!
Immunisation is by no means a trivial matter; wrong decisions in this area have the potential to cost the lives of millions. Therefore proofs of efficacy need to be published in peer-reviewed journals of high standing. These findings need then be criticised, replicated and re-criticised and re-replicated. Only when there is a wide consensus about the efficacy/safety or lack of efficacy/safety of a new form of immunisation, can it be generally accepted and implemented into clinical practice.
The current consensus about homeopathic immunisation is that it is nothing less than dangerous phantasy. Those who promote this quackery should be publicly exposed as charlatans of the worst kind.
Some practitioners of alternative medicine (doctors, naturopaths, chiropractors and others) earn a lot of money with the claim that chelation therapy (an effective mainstream treatment for acute heavy metal poisoning) is an effective means to treat cardiovascular disease. However, the notion is controversial and implausible. Several systematic reviews of the best evidence concluded less than optimistically:
The available data do not support the use of chelation in cardiovascular diseases.
More recently, important new evidence has emerged. The largest study of chelation therapy (TACT) ever conducted cost ~ $ 30 million and concluded that among stable patients with a history of MI, use of an intravenous chelation regimen with disodium EDTA, compared with placebo, modestly reduced the risk of adverse cardiovascular outcomes, many of which were revascularization procedures. These results provide evidence to guide further research but are not sufficient to support the routine use of chelation therapy for treatment of patients who have had an MI.
At the time, the TACT trial was heavily and rightly criticised for a whole host of reasons. For instance, because of the result of the FDA inspection of the highest accruing TACT site:
- The investigators didn’t conduct the investigation in accordance with the signed statement and investigational plan. Several examples were given of shoddy procedures, prefilled forms, and failure to train personnel.
- Failure to report promptly to the IRB all unanticipated problems involving risk to human subjects or others. Examples are given, including failure to report the deaths of patients on the study in a timely fashion (in one case the death wasn’t reported to the IRB until four months later; in another case it was never reported at all). In other cases, adverse event reports were not submitted to the IRB.
- Failure to prepare or maintain adequate case histories with respect to observations and data pertinent to the investigation.
- Investigational drug disposition records are not adequate with respect to dates, quantity, and use by subjects.
Despite these problems, the study was published in JAMA, albeit with a very critical editorial:
Differential dropout in TACT suggests unmasking, but the problem of intentional unblinding is more concerning. The sponsors of the trial, the National Heart, Lung, and Blood Institute (NHLBI) and the National Center for Complementary and Alternative Medicine (NCCAM), were unblinded throughout the trial. The National Institutes of Health policy unwisely allows the sponsor access to unblinded trial data, and both organizations sent observers to the closed sessions of the data monitoring committee. This gave them access to confidential data during each of the 11 interim analyses. The unblinding of the study sponsor represents a serious deviation from acceptable standards of conduct for supervision of clinical trials. If a pharmaceutical company sponsoring a trial were allowed access to actual outcome data during the study, there would be major objections. Like any sponsor, the NHLBI and NCCAM cannot be considered unbiased observers. These agencies made major financial commitments to the trial and may intentionally or inadvertently influence study conduct if inappropriately unblinded during the study…
Given the numerous concerns with this expensive, federally funded clinical trial, including missing data, potential investigator or patient unmasking, use of subjective end points, and intentional unblinding of the sponsor, the results cannot be accepted as reliable and do not demonstrate a benefit of chelation therapy. The findings of TACT should not be used as a justification for increased use of this controversial therapy.
Orac, makes several further critical points about the published trial:
First, the primary endpoint (i.e., the aggregated serious cardiovascular events) did indeed show a modest difference, namely 30% of placebo subjects versus 26.5% of the EDTA chelation subjects (hazard ratio 0.82 for chelation). However, one notes that the result is just barely statistically significant, p = 0.035, with the 99% confidence interval for the hazard ratio ranging from 0.69 to 0.99. (The predetermined level for statistical significance for purposes of this study was 0.036; so this is statistically significant by the barest margin.) More importantly, if you look at the individual endpoints that make up that aggregate, there was no statistically significant difference in death, myocardial infarction, stroke, coronary revascularization, and hospitalization for angina. Subgroup analysis (always a questionable analysis that requires replication, even when preplanned, as in TACT) purported to show a much greater benefit for diabetics, with a hazard ratio of 0.61 (p=0.002), while patients without diabetes showed no statistically significant difference in any of the outcome measures, including the aggregated total bad outcomes.
Now a paper that has just emerged describes the intent-to-treat comparison of this trial in patients with diabetes.
This was a double-blind, placebo-controlled, 2 × 2 factorial multicenter randomized trial of 1,708 post-myocardial infarction (MI) patients ≥50 years of age and with creatinine ≤2.0 mg/dL randomized to receive 40 EDTA chelation or placebo infusions plus 6 caplets daily of a 28-component multivitamin-multimineral mixture or placebo. The primary end point was a composite of total mortality, MI, stroke, coronary revascularization, or hospitalization for angina.
Median age was 65 years, 18% were female, 94% were Caucasian, 37% were diabetic, 83% had prior coronary revascularization, and 73% were on statins. Five-year Kaplan-Meier estimates for the primary end point was 31.9% in the chelation + high-dose vitamin group, 33.7% in the chelation + placebo vitamin group, 36.6% in the placebo infusion + active vitamin group, and 40.2% in the placebo infusions + placebo vitamin group. The reduction in primary end point by double active treatment compared with double placebo was significant (hazard ratio 0.74, 95% CI 0.57-0.95, P = .016). In patients with diabetes, the primary end point reduction of double active compared with double placebo was more pronounced (hazard ratio 0.49, 95% CI 0.33-0.75, P < .001).
The authors conclude that in stable post-MI patients on evidence-based medical therapy, the combination of oral high-dose vitamins and chelation therapy compared with double placebo reduced clinically important cardiovascular events to an extent that was both statistically significant and of potential clinical relevance.
I fear that these conclusions are erroneous and misleading: the marginally positive finding might have nothing to do with chelation per se; most likely they are due to the fact that the ‘vitamin’ mixture administered along with chelation contained ingredients like heparin and procaine which are potentially beneficial for cardiovascular conditions. Moreover, the placebo contained a considerable amount of glucose which could easily explain the better outcome of the diabetic subgroup receiving the verum – in other words, the verum generated better results not because it was effective but because the ‘placebo’ had detrimental effects.