MD, PhD, FMedSci, FSB, FRCP, FRCPEd

false positive

It is almost 10 years ago that Prof Kathy Sykes’ BBC series entitled ALTERNATIVE MEDICINE was aired. I had been hired by the BBC as their advisor for the programme and had tried my best to iron out the many mistakes that were about to be broadcast. But the scope for corrections turned out to be narrow and, at one stage, the errors seemed too serious and too far beyond repair to continue with my task. I had thus offered my resignation from this post. Fortunately this move led to some of my concerns being addressed after all, and they convinced me to remain in post.

The first part of the series was on acupuncture, and Kathy presented the opening scene of a young women undergoing open heart surgery with the aid of acupuncture. All the BBC had ever shown me and asked me to advise on was the text – I had never seen the images. Kathy’s text included the statement that the patient was having the surgery “with only needles to control the pain.”  I had not objected to this statement in the firm belief that the images of the film would back up this extraordinary claim. As it turned out, it did not; the patient clearly had all sorts of other treatments given through intra-venous lines and, in the film, these were openly in the view of Kathy Sykes.

This overt contradiction annoyed not just me but several other people as well. One of them was Simon Singh who filed an official complaint against the BBC for misleading the public, and eventually won his case.

The notion that acupuncture can serve as an alternative to anaesthesia or other surgical conditions crops up with amazing regularity. It is important not least because is often used as a promotional tool with the implication that, IF ACUPUNCTURE CAN ACHIVE SUCH DRAMATIC EFFECTS, IT MUST BE AN INCREDIBLY USEFUL TREATMENT! It is therefore relevant to ask what the scientific evidence tells us about this issue.

This was the question we wanted to address in a recent publication. Specifically, our aim was to summarise recent systematic reviews of acupuncture for surgical conditions.

Thirteen electronic databases were searched for relevant reviews published since 2000. Data were extracted by two independent reviewers according to predefined criteria. Twelve systematic reviews met our inclusion criteria. They related to the prevention or treatment of post-operative nausea and vomiting as well as to surgical or post-operative pain. The reviews drew conclusions which were far from uniform; specifically for surgical pain the evidence was not convincing. We concluded that “the evidence is insufficient to suggest that acupuncture is an effective intervention in surgical settings.”

So, Kathy Sykes’ comment was misguided in more than just one way: firstly, the scene she described in the film did not support what she was saying; secondly, the scientific evidence fails to support the notion that acupuncture can be used as an alternative to analgesia during surgery.

This story has several positive outcomes all the same. After seeing the BBC programme, Simon Singh contacted me to learn my views on the matter. This prompted me to support his complaint against the BBC and helped him to win this case. Furthermore, it led to a co-operation and friendship which produced our book TRICK OR TREATMENT.

The news that the use of Traditional Chinese Medicine (TCM) positively affects cancer survival might come as a surprise to many readers of this blog; but this is exactly what recent research has suggested. As it was published in one of the leading cancer journals, we should be able to trust the findings – or shouldn’t we?

The authors of this new study used the Taiwan National Health Insurance Research Database to conduct a retrospective population-based cohort study of patients with advanced breast cancer between 2001 and 2010. The patients were separated into TCM users and non-users, and the association between the use of TCM and patient survival was determined.

A total of 729 patients with advanced breast cancer receiving taxanes were included. Their mean age was 52.0 years; 115 patients were TCM users (15.8%) and 614 patients were TCM non-users. The mean follow-up was 2.8 years, with 277 deaths reported to occur during the 10-year period. Multivariate analysis demonstrated that, compared with non-users, the use of TCM was associated with a significantly decreased risk of all-cause mortality (adjusted hazards ratio [HR], 0.55 [95% confidence interval, 0.33-0.90] for TCM use of 30-180 days; adjusted HR, 0.46 [95% confidence interval, 0.27-0.78] for TCM use of > 180 days). Among the frequently used TCMs, those found to be most effective (lowest HRs) in reducing mortality were Bai Hua She She Cao, Ban Zhi Lian, and Huang Qi.

The authors of this paper are initially quite cautious and use adequate terminology when they write that TCM-use was associated with increased survival. But then they seem to get carried away by their enthusiasm and even name the TCM drugs which they thought were most effective in prolonging cancer survival. It is obvious that such causal extrapolations are well out of line with the evidence they produced (oh, how I wished that journal editors would finally wake up to such misleading language!) .

Of course, it is possible that some TCM drugs are effective cancer cures – but the data presented here certainly do NOT demonstrate anything like such an effect. And before such a far-reaching claim is being made, much more and much better research would be necessary.

The thing is, there are many alternative and plausible explanations for the observed phenomenon. For instance, it is conceivable that users and non-users of TCM in this study differed in many ways other than their medication, e.g. severity of cancer, adherence to conventional therapies, life-style, etc. And even if the researchers have used clever statistical methods to control for some of these variables, residual confounding can never be ruled out in such case-control studies.

Correlation is not causation, they say. Neglect of this elementary axiom makes for very poor science – in fact, it produces dangerous pseudoscience which could, like in the present case, lead a cancer patient straight up the garden path towards a premature death.

Do you think that chiropractic is effective for asthma? I don’t – in fact, I know it isn’t because, in 2009, I have published a systematic review of the available RCTs which showed quite clearly that the best evidence suggested chiropractic was ineffective for that condition.

But this is clearly not true, might some enthusiasts reply. What is more, they can even refer to a 2010 systematic review which indicates that chiropractic is effective; its conclusions speak a very clear language: …the eight retrieved studies indicated that chiropractic care showed improvements in subjective measures and, to a lesser degree objective measures… How on earth can this be?

I would not be surprised, if chiropractors claimed the discrepancy is due to the fact that Prof Ernst is biased. Others might point out that the more recent review includes more studies and thus ought to be more reliable. The newer review does, in fact, have about twice the number of studies than mine.

How come? Were plenty of new RCTs published during the 12 months that lay between the two publications? The answer is NO. But why then the discrepant conclusions?

The answer is much less puzzling than you might think. The ‘alchemists of alternative medicine’ regularly succeed in smuggling non-evidence into such reviews in order to beautify the overall picture and confirm their wishful thinking. The case of chiropractic for asthma does by no means stand alone, but it is a classic example of how we are being misled by charlatans.

Anyone who reads the full text of the two reviews mentioned above will find that they do, in fact, include exactly the same amount of RCTs. The reason why they arrive at different conclusions is simple: the enthusiasts’ review added NON-EVIDENCE to the existing RCTs. To be precise, the authors included one case series, one case study, one survey, two randomized controlled trials (RCTs), one randomized patient and observer blinded cross-over trial, one single blind cross study design, and one self-reported impairment questionnaire.

Now, there is nothing wrong with case reports, case series, or surveys – except THEY TELL US NOTHING ABOUT EFFECTIVENESS. I would bet my last shirt that the authors know all of that; yet they make fairly firm and positive conclusions about effectiveness. As the RCT-results collectively happen to be negative, they even pretend that case reports etc. outweigh the findings of RCTs.

And why do they do that? Because they are interested in the truth, or because they don’t mind using alchemy in order to mislead us? Your guess is as good as mine.

The efficacy or effectiveness of medical interventions is, of course, best tested in clinical trials. The principle of a clinical trial is fairly simple: typically, a group of patients is divided (preferably at random) into two subgroups, one (the ‘verum’ group) is treated with the experimental treatment and the other (the ‘control’ group) with another option (often a placebo), and the eventual outcomes of the two groups is compared. If done well, such studies are able to exclude biases and confounding factors such that their findings allow causal inference. In other words, they can tell us whether an outcome was caused by the intervention per se or by some other factor such as the natural history of the disease, regression towards the mean etc.

A clinical trial is a research tool for testing hypotheses; strictly speaking, it tests the ‘null-hypothesis’: “the experimental treatment generates the same outcomes as the treatment of the control group”. If the trial shows no difference between the outcomes of the two groups, the null-hypothesis is confirmed. In this case, we commonly speak of a negative result. If the experimental treatment was better than the control treatment, the null-hypothesis is rejected, and we commonly speak of a positive result. In other words, clinical trials can only generate positive or negative results, because the null-hypothesis must either be confirmed or rejected – there are no grey tones between the black of a negative and the white of a positive study.

For enthusiasts of alternative medicine, this can create a dilemma, particularly if there are lots of published studies with negative results. In this case, the totality of the available trial evidence is negative which means the treatment in question cannot be characterised as effective. It goes without saying that such an overall conclusion rubs the proponents of that therapy the wrong way. Consequently, they might look for ways to avoid this scenario.

One fairly obvious way of achieving this aim is to simply re-categorise the results. What, if we invented a new category? What, if we called some of the negative studies by a different name? What about NON-CONCLUSIVE?

That would be brilliant, wouldn’t it. We might end up with a simple statistic where the majority of the evidence is, after all, positive. And this, of course, would give the impression that the ineffective treatment in question is effective!

How exactly do we do this? We continue to call positive studies POSITIVE; we then call studies where the experimental treatment generated worst results than the control treatment (usually a placebo) NEGATIVE; and finally we call those studies where the experimental treatment created outcomes which were not different from placebo NON-CONCLUSIVE.

In the realm of alternative medicine, this ‘non-conclusive result’ method has recently become incredibly popular . Take homeopathy, for instance. The Faculty of Homeopathy proudly claim the following about clinical trials of homeopathy: Up to the end of 2011, there have been 164 peer-reviewed papers reporting randomised controlled trials (RCTs) in homeopathy. This represents research in 89 different medical conditions. Of those 164 RCT papers, 71 (43%) were positive, 9 (6%) negative and 80 (49%) non-conclusive.

This misleading nonsense was, of course, warmly received by homeopaths. The British Homeopathic Association, like many other organisations and individuals with an axe to grind lapped up the message and promptly repeated it: The body of evidence that exists shows that much more investigation is required – 43% of all the randomised controlled trials carried out have been positive, 6% negative and 49% inconclusive.

Let’s be clear what has happened here: the true percentage figures seem to show that 43% of studies (mostly of poor quality) suggest a positive result for homeopathy, while 57% of them (on average the ones of better quality) were negative. In other words, the majority of this evidence is negative. If we conducted a proper systematic review of this body of evidence, we would, of course, have to account for the quality of each study, and in this case we would have to conclude that homeopathy is not supported by sound evidence of effectiveness.

The little trick of applying the ‘NON-CONCLUSIVE’ method has thus turned this overall result upside down: black has become white! No wonder that it is so popular with proponents of all sorts of bogus treatments.

Whenever a new trial of an alternative intervention emerges which fails to confirm the wishful thinking of the proponents of that therapy, the world of alternative medicine is in turmoil. What can be done about yet another piece of unfavourable evidence? The easiest solution would be to ignore it, of course - and this is precisely what is often tried. But this tactic usually proves to be unsatisfactory; it does not neutralise the new evidence, and each time someone brings it up, one has to stick one’s head back into the sand. Rather than denying its existence, it would be preferable to have a tool which invalidates the study in question once and for all.

The ‘fatal flaw’ solution is simpler than anticipated! Alternative treatments are ‘very special’, and this notion must be emphasised, blown up beyond all proportions and used cleverly to discredit studies with unfavourable outcomes: the trick is simply to claim that studies with unfavourable results have a ‘fatal flaw’ in the way the alternative treatment was applied. As only the experts in the ‘very special’ treatment in question are able to judge the adequacy of their therapy, nobody is allowed to doubt their verdict.

Take acupuncture, for instance; it is an ancient ‘art’ which only the very best will ever master – at least that is what we are being told. So, all the proponents need to do in order to invalidate a trial, is read the methods section of the paper in full detail and state ‘ex cathedra’ that the way acupuncture was done in this particular study is completely ridiculous. The wrong points were stimulated, or the right points were stimulated but not long enough [or too long], or the needling was too deep [or too shallow], or the type of stimulus employed was not as recommended by TCM experts, or the contra-indications were not observed etc. etc.

As nobody can tell a correct acupuncture from an incorrect one, this ‘fatal flaw’ method is fairly fool-proof. It is also ever so simple: acupuncture-fans do not necessarily study hard to find the ‘fatal flaw’, they only have to look at the result of a study – if it was favourable, the treatment was obviously done perfectly by highly experienced experts; if it was unfavourable, the therapists clearly must have been morons who picked up their acupuncture skills in a single weekend course. The reasons for this judgement can always be found or, if all else fails, invented.

And the end-result of the ‘fatal flaw’ method is most satisfactory; what is more, it can be applied to all alternative therapies – homeopathy, herbal medicine, reflexology, Reiki healing, colonic irrigation…the method works for all of them! What is even more, the ‘fatal flaw’ method is adaptable to other aspects of scientific investigations such that it fits every conceivable circumstance.

An article documenting the ‘fatal flaw’ has to be published, of course - but this is no problem! There are dozens of dodgy alternative medicine journals which are only too keen to print even the most far-fetched nonsense as long as it promotes alternative medicine in some way. Once this paper is published, the proponents of the therapy in question have a comfortable default position to rely on each time someone cites the unfavourable study: “WHAT NOT THAT STUDY AGAIN! THE TREATMENT HAS BEEN SHOWN TO BE ALL WRONG. NOBODY CAN EXPECT GOOD RESULTS FROM A THERAPY THAT WAS NOT CORRECTLY ADMINISTERED. IF YOU DON’T HAVE BETTER STUDIES TO SUPPORT YOUR ARGUMENTS, YOU BETTER SHUT UP.”

There might, in fact, be better studies – but chances are that the ‘other side’ has already documented a ‘fatal flaw’ in them too.

It is usually BIG PHARMA who stands accused of being less than honest with the evidence, particularly when it runs against commercial interests; and the allegations prove to be correct with depressing regularity. In alternative medicine, commercial interests exist too, but there is usually much less money at stake. So, a common assumption is that conflicts of interest are less relevant in alternative medicine. Like so many assumptions in this area, this notion is clearly and demonstrably erroneous.

The sums of money are definitely smaller, but non-commercial conflicts of interest are potentially more important than the commercial ones. I am thinking of the quasi-religious beliefs that are so very prevalent in alternative medicine. Belief can move mountains, they say – it can surely delude people and make them do the most extraordinary things. Belief can transform advocates of alternative medicine into ‘ALCHEMISTS OF ALTERNATIVE EVIDENCE’ who turn negative/unfavourable into positive/favourable evidence.

The alchemists’ ‘tricks of the trade’ are often the same as used by BIG PHARMA; they include:

  • drawing conclusions which are not supported by the data
  • designing studies such that they will inevitably generate a favourable result
  • cherry-picking the evidence
  • hiding unfavourable findings
  • publishing favourable results multiple times
  • submitting data-sets to multiple statistical tests until a positive result emerges
  • defaming scientists who publish unfavourable findings
  • bribing experts
  • prettify data
  • falsifying data

As I said, these methods, albeit despicable, are well-known to pseudoscientists in all fields of inquiry. To assume that they are unknown in alternative medicine is naïve and unrealistic, as many of my previous posts confirm.

In addition to these ubiquitous ‘standard’ methods of scientific misconduct and fraud, there are a few techniques which are more or less unique to and typical for the alchemists of alternative medicine. In the following parts of this series of articles, I will try to explain these methods in more detail.

Cancer patients are bombarded with information about supplements which allegedly are effective for their condition. I estimate that 99.99% of this information is unreliable and much of it is outright dangerous. So, there is an urgent need for trustworthy, objective information. But which source can we trust?

The authors of a recent article in ‘INTEGRATIVE CANCER THARAPIES’ (the first journal to spearhead and focus on a new and growing movement in cancer treatment. The journal emphasizes scientific understanding of alternative medicine and traditional medicine therapies, and their responsible integration with conventional health care. Integrative care includes therapeutic interventions in diet, lifestyle, exercise, stress care, and nutritional supplements, as well as experimental vaccines, chrono-chemotherapy, and other advanced treatments) review the issue of dietary supplements in the treatment of cancer patients. They claim that the optimal approach is to discuss both the facts and the uncertainty with the patient, in order to reach a mutually informed decision. This sounds promising, and we might thus trust them to deliver something reliable.

In order to enable doctors and other health care professionals to have such discussion, the authors then report on the work of the ‘Clinical Practice Committee’ of ‘The Society of Integrative Oncology’. This panel undertook the challenge of providing basic information to physicians who wish to discuss these issues with their patients. A list of supplements that have the best suggestions of benefit was constructed by leading researchers and clinicians who have experience in using these supplements:

  1. curcumin,
  2. glutamine,
  3. vitamin D,
  4. maitake mushrooms,
  5. fish oil,
  6. green tea,
  7. milk thistle,
  8. astragalus,
  9. melatonin,
  10. probiotics.

The authors claim that their review includes basic information on each supplement, such as evidence on effectiveness and clinical trials, adverse effects, and interactions with medications. The information was constructed to provide an up-to-date base of knowledge, so that physicians and other health care providers would be aware of the supplements and be able to discuss realistic expectations and potential benefits and risks (my emphasis).

At first glance, this task looks ambitious but laudable; however, after studying the paper in some detail, I must admit that I have considerable problems taking it seriously – and here is why.

The first question I ask myself when reading the abstract is: Who are these “leading researchers and clinicians”? Surely such a consensus exercise crucially depends on who is being consulted. The article itself does not reveal who these experts are, merely that they are all members of the ‘Society of Integrative Oncology’. A little research reveals this organisation to be devoted to integrating all sorts of alternative therapies into cancer care. If we assume that the experts are identical with the authors of the review; one should point out that most of them are proponents of alternative medicine. This lack of critical input seems more than a little disconcerting.

My next questions are: How did they identify the 10 supplements and how did they evaluate the evidence for or against them? The article informs us that a 5-step procedure was employed:

1. Each clinician in this project was requested to construct a list of supplements that they tend to use frequently in their practice.

2. An initial list of close to 25 supplements was constructed. This list included supplements that have suggestions of some possible benefit and likely to carry minimal risk in cancer care.

3. From that long list, the group agreed on the 10 leading supplements that have the best suggestions of benefit.

4. Each participant selected 1 to 2 supplements that they have interest and experience in their use and wrote a manuscript related to the selected supplement in a uniformed and agreed format. The agreed format was constructed to provide a base of knowledge, so physicians and other health care providers would be able to discuss realistic expectations and potential benefits and risks with patients and families that seek that kind of information.

5. The revised document was circulated among participants for revisions and comments.

This method might look fine to proponents of alternative medicine, but from a scientific point of view, it is seriously wanting. Essentially, they asked those experts who are in favour of a given supplement to write a report to justify his/her preference. This method is not just open bias, it formally invites bias.

Predictably then, the reviews of the 10 chosen supplements are woefully inadequate. These is no evidence of a systematic approach; the cited evidence is demonstrably cherry-picked; there is a complete lack of critical analysis; for several supplements, clinical data are virtually absent without the authors finding this embarrassing void a reason for concern; dosage recommendations are often vague and naïve, to say the least (for instance, for milk thistle: 200 to 400 mg per day – without indication of what the named weight range refers to, the fresh plant, dried powder, extract…?); safety data are incomplete and nobody seems to mind that supplements are not subject to systematic post-marketing surveillance; the text is full of naïve thinking and contradictions (e.g.”There are no reported side effects of the mushroom extracts or the Maitake D-fraction. As Maitake may lower blood sugar, it should be used with caution in patients with diabetes“); evidence suggesting that a given supplement might reduce the risk of cancer is presented as though this means it is an effective treatment for an existing cancer; cancer is usually treated as though it is one disease entity without any differentiation of different cancer types.

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. But I do wonder, isn’t being in favour of integrating half-baked nonsense into cancer care and being selected for one’s favourable attitude towards certain supplements already a conflict of interest?

In any case, the review is in my view not of sufficient rigor to form the basis for well-informed discussions with patients. The authors of the review cite a guideline by the ‘Society of Integrative Oncology’ for the use of supplements in cancer care which states: For cancer patients who wish to use nutritional supplements, including botanicals for purported antitumor effects, it is recommended that they consult a trained professional. During the consultation, the professional should provide support, discuss realistic expectations, and explore potential benefits and risks. It is recommended that use of those agents occur only in the context of clinical trials, recognized nutritional guidelines, clinical evaluation of the risk/benefit ratio based on available evidence, and close monitoring of adverse effects. It seems to me that, with this review, the authors have not adhered to their own guideline.

Criticising the work of others is perhaps not very difficult, however, doing a better job usually is. So, can I offer anything that is better than the above criticised review? The answer is YES. Our initiative ‘CAM cancer’ provides up-to-date, concise and evidence-based systematic reviews of many supplements and other alternative treatments that cancer patients are likely to hear about. Their conclusions are not nearly as uncritically positive as those of the article in ‘INTEGRATIVE CANCER THERAPIES’.

I happen to believe that it is important for cancer patients to have access to reliable information and that it is unethical to mislead them with biased accounts about the value of any treatment.

There is not a discussion about homeopathy where an apologist would eventually state: HOMEOPATHY CANNOT BE A PLACEBO, BECAUSE IT WORKS IN ANIMALS!!! Those who are not well-versed in this subject tend to be impressed, and the argument has won many consumers over to the dark side, I am sure. But is it really correct?

The short answer to this question is NO.

Pavlov discovered the phenomenon of ‘conditioning’ in animals, and ‘conditioning’ is considered to be a major part of the placebo-response. So, depending on the circumstances, animals do respond to placebo (my dog, for instance, used to go into a distinct depressive mood when he saw me packing a suitcase).

Then there is the fact that the animal’s response might be less important than the owner’s reaction to homeopathic treatment. This is particularly important with pets, of course. Homeopathy-believing pet owners might over-interpret the pet’s response and report that the homeopathic remedy has worked wonders when, in fact, it has made no difference.

Finally, there may be some situations where neither of the above two phenomena can play a decisive role. Homeopaths like to cite studies where entire herds of cows were treated homeopathically to prevent mastitis, a common problem in milk-cows. It is unlikely that conditioning or wishful thinking of the owner are decisive in such a study. Let’s see whether homeopathy-promoters will also be fond of this new study of exactly this subject.

New Zealand vets compared clinical and bacteriological cure rates of clinical mastitis following treatment with either antimicrobials or homeopathic preparations. They used 7 spring-calving herds from the Waikato region of New Zealand to source cases of clinical mastitis (n=263 glands) during the first 90 days following calving. Duplicate milk samples were collected for bacteriology from each clinically infected gland at diagnosis and 25 (SD 5.3) days after the initial treatment. Affected glands were treated with either an antimicrobial formulation or a homeopathic remedy. Generalised linear models with binomial error distribution and logit link were used to analyse the proportion of cows that presented clinical treatment cures and the proportion of glands that were classified as bacteriological cures, based on initial and post-treatment milk samples.

The results show that the mean cumulative incidence of clinical mastitis was 7% (range 2-13% across herds) of cows. Streptococcus uberis was the most common pathogen isolated from culture-positive samples from affected glands (140/209; 67%). The clinical cure rate was higher for cows treated with antimicrobials (107/113; 95%) than for cows treated with homeopathic remedies (72/114; 63%) (p<0.001) based on the observance of clinical signs following initial treatment. Across all pathogen types bacteriological cure rate at gland level was higher for those cows treated with antimicrobials (75/102; 74%) than for those treated with a homeopathic preparation (39/107; 36%) (p<0.001).

The authors conclude that homeopathic remedies had significantly lower clinical and bacteriological cure rates compared with antimicrobials when used to treat post-calving clinical mastitis where S. uberis was the most common pathogen. The proportion of cows that needed retreatment was significantly higher for the homeopathic treated cows. This, combined with lower bacteriological cure rates, has implications for duration of infection, individual cow somatic cell count, costs associated with treatment and animal welfare.

Yes, I know, this is just one single study, and we need to consider the totality of the reliable evidence. Currently, there are 203 clinical trials of homeopathic treatments of animals; and they are being reviewed at the very moment (unfortunately by a team that is not known for its objective stance on homeopathy). So, we will have to wait and see. When, in 1999, A. Vickers reviewed all per-clinical studies, including those on animals, he concluded that there is a lack of independent replication of any pre-clinical research in homoeopathy. In the few instances where a research team has set out to replicate the work of another, either the results were negative or the methodology was questionable.

All this is to say that, until truly convincing evidence to the contrary is available, the homeopaths’ argument ‘HOMEOPATHY CANNOT BE A PLACEBO, BECAUSE IT WORKS IN ANIMALS!!!’ is, in my view, as weak as the dilution of their remedies.

There are dozens of observational studies of homeopathy which seem to suggest – at least to homeopaths – that homeopathic treatments generate health benefits. As these investigations lack a control group, their results can be all to easily invalidated by pointing out that factors like ‘regression towards the mean‘ (RTM, a statistical artefact caused by the phenomenon that a variable that is extreme on its first measurement tends to be closer to the average on its second measurement) might be the cause of the observed change. Thus the debate whether such observational data are reliable or not has been raging for decades. Now, German (pro-homeopathy) investigators have published a paper which potentially could resolve this dispute.

With this re-analysis of an observational study, the investigators wanted to evaluate whether the observed changes in previous cohort studies are due to RTM and to estimate RTM adjusted effects. SF-36 quality-of-life (QoL) data from a cohort of 2827 chronically diseased adults treated with homeopathy were reanalysed using a method described in 1991 by Mee and Chua’s. RTM adjusted effects, standardized by the respective standard deviation at baseline, were 0.12 (95% CI: 0.06-0.19, P < 0.001) in the mental and 0.25 (0.22-0.28, P < 0.001) in the physical summary score of the SF-36. Small-to-moderate effects were confirmed for most individual diagnoses in physical, but not in mental component scores. Under the assumption that the true population mean equals the mean of all actually diseased patients, RTM adjusted effects were confirmed for both scores in most diagnoses.

The authors reached the following conclusion: “In our paper we showed that the effects on quality of life observed in patients receiving homeopathic care in a usual care setting are small or moderate at maximum, but cannot be explained by RTM alone. Due to the uncontrolled study design they may, however, completely be due to nonspecific effects. All our analyses made a restrictive and conservative assumption, so the true treatment effects might be larger than shown.” 

Of course, the analysis heavily relies on the validity of Mee and Chua’s modified t-test. It requires the true mean in the target population to be known, a requirement that seldom can be fulfilled. The authors therefore took the SF-36 mean summary scores from the 1998 German health survey as proxies. I am not a statistician and therefore unable to tell how reliable this method might be (- if there is someone out there who can give us some guidance here, please post your comment).

In order to make sense of these data, we need to consider that, during the study period, about half of the patients admitted to have had additional visits to non-homeopathic doctors, and 27% also received conventional drugs. In addition, they would have benefitted from:

  • the benign history of the conditions they were suffering from,
  • a placebo-effect,
  • the care and attention they received
  • and all sorts of other non-specific effects.

So, considering these factors, what does this interesting re-analysis really tell us? My interpretation is as follows: the type of observational study that homeopaths are so fond of yields false-positive results. If we correct them – as the authors have done here for just one single factor, the RTM – the effect size gets significantly smaller. If we were able to correct them for some of the other factors mentioned above, the effect size would shrink more and more. And if we were able to correct them for all confounders, their results would almost certainly concur with those of rigorously controlled trials which demonstrate that homeopathic remedies are pure placebos.

I am quite sure that this interpretation is unpopular with homeopaths, but I am equally certain that it is correct.

Advocates of alternative medicine are incredibly fond of supporting their claims with anecdotes, or ‘case-reports’ as they are officially called. There is no question, case-reports can be informative and important, but we need to be aware of their limitations.

A recent case-report from the US might illustrated this nicely. It described a 65-year-old male patient who had had MS for 20 years when he decided to get treated with Chinese scalp acupuncture. The motor area, sensory area, foot motor and sensory area, balance area, hearing and dizziness area, and tremor area were stimulated once a week for 10 weeks, then once a month for 6 further sessions.

After the 16 treatments, the patient showed remarkable improvements. He was able to stand and walk without any problems. The numbness and tingling in his limbs did not bother him anymore. He had more energy and had not experienced incontinence of urine or dizziness after the first treatment. He was able to return to work full time. Now the patient has been in remission for 26 months.

The authors of this case-report conclude that Chinese scalp acupuncture can be a very effective treatment for patients with MS. Chinese scalp acupuncture holds the potential to expand treatment options for MS in both conventional and complementary or integrative therapies. It can not only relieve symptoms, increase the patient’s quality of life, and slow and reverse the progression of physical disability but also reduce the number of relapses and help patients.

There is absolutely nothing wrong with case-reports; on the contrary, they can provide extremely valuable pointers for further research. If they relate to adverse effects, they can give us crucial information about the risks associated with treatments. Nobody would ever argue that case-reports are useless, and that is why most medical journals regularly publish such papers. But they are valuable only, if one is aware of their limitations. Medicine finally started to make swift progress, ~150 years ago, when we gave up attributing undue importance to anecdotes, began to doubt established wisdom and started testing it scientifically.

Conclusions such as the ones drawn above are not just odd, they are misleading to the point of being dangerous. A reasonable conclusion might have been that this case of a MS-patient is interesting and should be followed-up through further observations. If these then seem to confirm the positive outcome, one might consider conducting a clinical trial. If this study proves to yield encouraging findings, one might eventually draw the conclusions which the present authors drew from their single case.

To jump at conclusions in the way the authors did, is neither justified nor responsible. It is unjustified because case-reports never lend themselves to such generalisations. And it is irresponsible because desperate patients, who often fail to understand the limitations of case-reports and tend to believe things that have been published in medical journals, might act on these words. This, in turn, would raise false hopes or might even lead to patients forfeiting those treatments that are evidence-based.

It is high time, I think, that proponents of alternative medicine give up their love-affair with anecdotes and join the rest of the health care professions in the 21st century.

Categories