MD, PhD, FMedSci, FSB, FRCP, FRCPEd

bias

Even after all these years of full-time research into alternative medicine and uncounted exchanges with enthusiasts involved in this sector, I find the logic that is often applied in this field bewildering and the unproductiveness of the dialogue disturbing.

To explain what I mean, it be might best to publish a (fictitious, perhaps slightly exaggerated) debate between a critical thinker or scientist (S) and an uncritical proponent (P) of one particular form of alternative medicine.

P: Did you see this interesting study demonstrating that treatment X is now widely accepted, even by highly critical GPs at the cutting edge of health care?

S: This was a survey, not a ‘study’, and I never found the average GP “highly critical”. Surveys of this nature are fairly useless and they “demonstrate” nothing of real value.

P: Whatever, but it showed that GPs accept treatment X. This can only mean that they realise how safe and effective it is.

S: Not necessarily, GPs might just give in to consumer demand, or the sample was cleverly selected, or the question was asked in a leading manner, etc.

P: Hardly, because there is plenty of good evidence for treatment X.

S: Really? Show me.

P: There is this study here which proves that treatment X works and is risk-free.

S: The study was far too small to demonstrate safety, and it is wide open to multiple sources of bias. Therefore it does not conclusively show efficacy either.

P: You just say this because you don’t like its result! You have a closed mind!

In any case, it was merely an example! There are plenty more positive studies; do your research properly before you talk such nonsense.

S: I did do some research and I found a recent, high quality systematic review that arrived at a negative conclusion about the value of treatment X.

P: That review was done by sceptics who clearly have an axe to grind. It is based on studies which do not account for the intrinsic subtleties of treatment X. Therefore they are unfair tests of treatment X. These trials don’t really count at all. Every insider knows that! The fact that you cite it merely confirms that you do not understand what you are talking about.

S: It seems to me, that you like scientific evidence only when it confirms your belief. This, I am afraid, is what quacks tend to do!

P: I strongly object to being insulted in this way.

S: I did not insult you, I merely made a statement of fact.

P: If you like facts, you have to see that one needs to have sufficient expertise in treatment X in order to apply it properly and effectively. This important fact is neglected in all of those trials that report negative results; and that’s why they are negative. Simple! I really don’t understand why you are too stupid to understand this. Such studies do not show that treatment X is ineffective, but they demonstrate that the investigators were incompetent or hired with the remit to discredit treatment X.

S: I would have thought they are negative because they minimised bias and the danger of generating a false positive result.

P: No, by minimising bias, as you put it, these trials eliminated the factors that are important elements of treatment X.

S: Such as the placebo-effect?

P: That’s what you call it because you irrationally believe in reductionist science.

S: Science requires no belief, I think you are the believer here.

P: The fact is that scientists of your ilk negate all factors related to human interactions. Patients are no machines, you know, they need compassion; we clinicians know that because we work at the coal face of health care. Scientists in their ivory towers have no idea about patient care and just want science for science sake. This is not how you help patients. Show some compassion man!

S: I do know about the importance of compassion and care, but here we are discussing an entirely different topic, namely tests the efficacy or effectiveness of treatments, not patient-care. Let’s focus on one issue at a time.

P: You cannot separate things in this way. We have to take a holistic view. Patients are whole individuals, and you cannot do them justice by running artificial experiments. Every patient is different; clinical trials fail to account for this fact and are therefore fairly irrelevant to us and to our patients. Real life is very different from your imagined little experiments, you know.

S: These are platitudes that are nonsensical in this context and do not contribute anything meaningful to the present discussion. You do not seem to understand the methodology or purpose of a clinical trial.

P: That is typical! Whenever you run out of arguments, you try to change the subject or throw a few insults at me.

S: Not at all, I thought we were talking about clinical trials evaluating the effectiveness of treatment X.

P: That’s right; and they do show that it is effective, provided you consider those which are truly well-done by experts who know about treatment X and believe in it.

S: Not true. Only if you cherry-pick the data will you be able to produce an overall positive result for treatment X.

P: In any case, the real world results of clinical practice show very clearly that it works. It would not have survived for so long, if it didn’t. Nobody can deny that, and nobody should claim that silly little trials done in artificial circumstances are more meaningful than a wealth of experience.

S: Experience has little to do with reliable evidence.

P: To deny the value of experience is just stupid and clearly puts you in the wrong. I have shown you plenty of reliable evidence but you just ignore everything I say that does not go along with your narrow-minded notions about science; science is not the only way of knowing or comprehending things! Stop being obsessed with science.

S: No, you show me rubbish data and have little understanding of science, I am afraid.

P: Here we go again! I have had about enough of that and your blinkered arguments. We are going in circles because you are ignorant and arrogant. I have tried my best to show you the light, but your mind is closed. I offer true insight and you pay me back with insults. You and your cronies are in the pocket of BIG PHARMA. You are cynical, heartless and not interested in the wellbeing of patients. Next you will tell me to vaccinate my kids!

S: I think this is a waste of time.

P: Precisely! Everyone who has followed this debate will see very clearly that you are obsessed with reductionist science and incapable of considering the suffering of whole individuals. You want to deny patients a treatment that  really helps them simply because you do not understand how treatment X works. Shame on you!!!

Neck pain is a common problem which is often far from easy to treat. Numerous therapies are being promoted but few are supported by good evidence. Could yoga be the solution?

The aim of a brand-new RCT was to evaluate the effectiveness of Iyengar yoga for chronic non-specific neck pain. Patients were randomly assigned to either yoga or exercise. The yoga group attended a 9-week yoga course, while the exercise group received a self-care manual on home-based exercises for neck pain. The primary outcome measure was neck pain. Secondary outcome measures included functional disability, pain at motion, health-related quality of life, cervical range of motion, proprioceptive acuity, and pressure pain threshold. Fifty-one patients participated in the study: yoga (n = 25), exercise (n = 26). At the end of the treatment phase, patients in the yoga group reported significantly less neck pain compared as well as less disability and better mental quality of life compared with the exercise group. Range of motion and proprioceptive acuity were improved and the pressure pain threshold was elevated in the yoga group.

The authors draw the following conclusion: “Yoga was more effective in relieving chronic nonspecific neck pain than a home-based exercise program. Yoga reduced neck pain intensity and disability and improved health-related quality of life. Moreover, yoga seems to influence the functional status of neck muscles, as indicated by improvement of physiological measures of neck pain.”

I’d love to agree with the authors and would be more than delighted, if an effective treatment for neck pain had been identified. Yoga is fairly safe and inexpensive; it promotes a generally healthy life-style, and is attractive to many patients; it has thus the potential to help thousands of suffering individuals. However, I fear that things might not be quite as rosy as the authors of this trial seem to believe.

The principle of an RCT essentially is that two groups of patients receive two different therapies and that any difference in outcome after the treatment phase is attributable to the therapy in question. Unfortunately, this is not the case here. One does not need to be an expert in critical thinking to realise that, in the present study, the positive outcome might be unrelated to yoga. For instance, it could be that the unsupervised home exercises were carried out wrongly and thus made the neck pain worse. In this case, the difference between the two treatment groups might not have been caused by yoga at all. A second possibility is that the yoga-group benefited not from the yoga itself but from the attention given to these patients which the exercise-group did not have. A third explanation could be that the yoga teachers were very kind to their patients, and that the patients returned their kindness by pretending to have less symptoms or exaggerating their improvements. In my view the most likely cause of the results seen in this study is a complex mixture of all the options just mentioned.

This study thus teaches us two valuable lessons: 1) whenever possible, RCTs should be designed such that a clear attribution of cause and effect is possible, once the results are on the table; 2) if cause and effect cannot be clearly defined, it is unwise to draw conclusions that are definite and have the potential to mislead the public.

This post has an odd title and addresses an odd subject. I am sure some people reading it will ask themselves “has he finally gone potty; is he a bit xenophobic, chauvinistic, or what?” I can assure you none of the above is the case.

Since many years, I have been asked to peer-review Chinese systematic reviews and meta-analyses of TCM-trials submitted to various journals and to the Cochrane Collaboration for publication, and I estimate that around 300 such articles are available today. Initially, I thought they were a valuable contribution to our knowledge, particularly for the many of us who cannot read Chinese languages. I hoped they might provide reliable information about this huge and potentially important section of the TCM-evidence. After doing this type of work for some time, I became more and more frustrated; now I have decided not to accept this task any longer – not because it is too much trouble, but because I have come to the conclusion that these articles are far less helpful than I had once assumed; in fact, I now fear that they are counter-productive.

In order to better understand what I mean, it might be best to use an example; this recent systematic review seems as good for that purpose as any.

Its Chinese authors “hypothesized that the eligible trials would provide evidence of the effect of Chinese herbs on bone mineral density (BMD) and the therapeutic benefits of Chinese medicine treatment in patients with bone loss. Randomized controlled trials (RCTs) were thus retrieved for a systematic review from Medline and 8 Chinese databases. The authors identified 12 RCTs involving a total of 1816 patients. The studies compared Chinese herbs with placebo or standard anti-osteoporotic therapy. The pooled data from these RCTs showed that the change of BMD in the spine was more pronounced with Chinese herbs compared to the effects noted with placebo. Also, in the femoral neck, Chinese herbs generated significantly higher increments of BMD compared to placebo. Compared to conventional anti-osteoporotic drugs, Chinese herbs generated greater BMD changes.

In their abstract, the part on the paper that most readers access, the authors reached the following conclusions: “Our results demonstrated that Chinese herb significantly increased lumbar spine BMD as compared to the placebo or other standard anti-osteoporotic drugs.” In the article itself, we find this more detailed conclusion: “We conclude that Chinese herbs substantially increased BMD of the lumbar spine compared to placebo or anti-osteoporotic drugs as indicated in the current clinical reports on osteoporosis treatment. Long term of Chinese herbs over 12 months of treatment duration may increase BMD in the hip more effectively. However, further studies are needed to corroborate the positive effect of increasing the duration of Chinese herbs on outcome as the results in this analysis are based on indirect comparisons. To date there are no studies available that compare Chinese herbs, Chinese herbs plus anti-osteoporotic drugs, and anti-osteoporotic drug versus placebo in a factorial design. Consequently, we are unable to draw any conclusions on the possible superiority of Chinese herbs plus anti-osteoporotic drug versus anti-osteoporotic drug or Chinese herb alone in the context of BMD.

Most readers will feel that this evidence is quite impressive and amazingly solid; they might therefore advocate routinely using Chinese herbs for the common and difficult to treat problem of osteoporosis. The integration of TCM might avoid lots of human suffering, prolong the life of many elderly patients, and save us all a lot of money. Why then am I not at all convinced?

The first thing to notice is the fact that we do not really know which of the ~7000 different Chinese herbs should be used. The article tells us surprisingly little about this crucial point. And even, if we manage to study this question in more depth, we are bound to get thoroughly confused; there are simply too many herbal mixtures and patent medicines to easily identify the most promising candidates.

The second and more important hurdle to making sense of these data is the fact that most of the primary studies originate from inaccessible Chinese journals and were published in Chinese languages which, of course, few people in the West can understand. This is entirely our fault, some might argue, but it does mean that we have to believe the authors, take their words at face value, and cannot check the original data. You may think this is fine, after all, the paper has gone through a rigorous peer-review process where it has been thoroughly checked by several top experts in the field. This, however, is a fallacy; like you and me, the peer-reviewers might not read Chinese either! (I don’t, and I reviewed quite a few of these papers; in some instances, I even asked for translations of the originals to do the job properly but this request was understandably turned down) In all likelihood, the above paper and most similar articles have not been properly peer-reviewed at all.

The third and perhaps most crucial point can only be fully appreciated, if we were able to access and understand the primary studies; it relates to the quality of the original RCTs summarised in such systematic reviews. The abstract of the present paper tells us nothing at all about this issue. In the paper, however, we do find a formal assessment of the studies’ risk of bias which shows that the quality of the included RCTs was poor to very poor. We also find a short but revealing sentence: “The reports of all trials mentioned randomization, but only seven described the method of randomization.” This remark is much more significant than it may seem: we have shown that such studies use such terminology in a rather adventurous way; reviewing about 2000 of these allegedly randomised trials, we found that many Chinese authors call a trial “randomised” even in the absence of a control group (one cannot randomise patients and have no control group)! They seem to like the term because it is fashionable and makes publication of their work easier. We thus have good reason to fear that some/many/most of the studies were not RCTs at all.

The fourth issue that needs mentioning is the fact that very close to 100% of all Chinese TCM-trials report positive findings. This means that either TCM is effective for every indication it is tested for (most unlikely, not least because there are many negative non-Chinese trials of TCM), or there is something very fundamentally wrong with Chinese research into TCM. Over the years, I have had several Chinese co-workers in my team and was invariably impressed by their ability to work hard and efficiently; we often discussed the possible reasons for the extraordinary phenomenon of 0% negative Chinese trials. The most plausible answer they offered was this: it would be most impolite for a Chinese researcher to produce findings which contradict the opinion of his/her peers.

In view of these concerns, can we trust the conclusions of such systematic reviews? I don’t think so – and this is why I have problems with research of this nature. If there are good reasons to doubt their conclusions, these reviews might misinform us systematically, they might not further but hinder progress, and they might send us up the garden path. This could well be in the commercial interest of the Chinese multi-billion dollar TCM-industry, but it would certainly not be in the interest of patients and good health care.

Many people who have arrived at a certain age have knee osteoarthritis and most of them suffer pain, lack of mobility etc. because of it. There are many effective treatments for this condition, of course, but some have serious side-effects, others are tedious to follow and therefore not popular, and none of the existing options totally cure the problem. In many cases, surgery is the best solution; a knee-endoprosthesis can restore everything almost back to normal. But surgery carries risks and will cause considerable pain and rehabilitation-effort. This is perhaps why we are still looking for a treatment that is both effective and risk-free. Personally, I doubt that such a therapy will ever be found, but that does, of course, not stop alternative medicine enthusiasts from claiming that this or that treatment is what the world has been waiting for. The newest kid on this block is leech therapy. Did I just write “newest”? Leeches are not new at all; they are a treatment from the dark ages of medicine – but are they about to experience a come-back?

A recent systematic review and meta-analysis evaluated the effectiveness of medical leech therapy for osteoarthritis of the knee. Five electronic databases were screened to identify randomized (RCTs) and non randomized controlled clinical trials (CCTs) comparing leech therapy to any type of control condition. The main outcome measures were pain, functional impairment, and joint stiffness. Three RCTs and 1 CCT with a total of 237 patients with osteoarthritis were included. Three trials had, according to the review-authors, a low risk of bias. They claimed to have found strong evidence for immediate and short-term pain reduction, immediate improvement in patients’ physical function, and both immediate and long-term improvement in their joint stiffness. Moderate evidence was found for leech therapy’s short-term effects on physical function and long-term effects on pain. Leech therapy was not associated with any serious adverse events. The authors reached the following conclusion: ” Given the low number of reported adverse events, leech therapy may be a useful approach in treating this condition. Further high-quality RCTs are required for the conclusive judgment of its effectiveness and safety.”

When, about 35 years ago, I worked as a young doctor in the homeopathic hospital in Munich, I was taught how to apply leeches to my patients. We got the animals from a specialised supplier, put them on the patient’s skin and waited until they had bitten a little hole and started sucking the patient’s blood. Once they were full they spontaneously fell off and were then disposed off. Many patients were too disgusted with the prospect of leech therapy to agree to this intervention. Those who did were very impressed with the procedure; it occurred to me then that this therapy must be associated with an enormous placebo-effect simply because it is exotic, impressive and a treatment that no patient will ever forget.

The bite of the leech is not normally painful because the leech has a local anaesthetic which it applies in order to suck blood without being noticed. The leech furthermore injects a powerful anticoagulant into its victim’s body which is necessary for preventing the blood from clotting. Through the injection of these pharmacologically active substances, leeches can clearly be therapeutic and they are thus not entirely unknown in conventional medicine; in plastic surgery, for instance, they are sometimes being used to generate optimal results for micro surgical wounds. Their anticoagulant has long been identified and is sometimes being used therapeutically. The use of leeches for the management of osteoarthritis, however, is not a conventional concept. So, how convincing are the above data? Should we agree with the authors’ conclusion that “leech therapy may be a useful approach in treating this condition“? I think not, and here is why:

1) The collective evidence for efficacy is far from convincing. The few studies which were summarised in this systematic review are mostly those of the research group that also authored the review. Critical thinkers would insist on an independent assessment of those trials. Moreover, none of the trials was patient-blind (which would not be all that difficult to do), and thus the enormous placebo-effect of applying a leech might be the cause of all or most of the observed effect.

2) The authors claim that the treatment is safe. On the basis of just 250 patients treated under highly controlled conditions, this claim has almost no evidential basis.

3) As already mentioned above, there are many treatments which are more effective for improving pain and function than leeches.

4) Leech therapy is time-consuming, relatively expensive and quite unpractical as a regular, long-term therapy.

5) In my experience, patients will run a mile to avoid having something as ‘disgusting’ as leeches sucking blood from their body.

6) The animals need to be destroyed after the treatment to avoid infections.

7) As multiple leeches applied regularly will suck a significant volume of blood, the treatment might lead to anaemia and would be contra-indicated in patients with low haemoglobin levels.

8) Like most other treatments for osteoarthritis, leech therapy would not be curative but might just alleviate the symptoms temporarily.

On balance therefore, I very much doubt that the leech will have a come-back in the realm of osteoarthritis therapy. In fact, I think that, in this particular context, leeches are just a chapter from the dark ages of medicine. Their re-introduction into osteoarthritis care seems like a significant step into the wrong direction.

Reiki is a form of  healing which rests on the assumption that some form “energy” determines our health. In this context, I tend to put energy in inverted commas because it is not the energy a physicist might have in mind. It is a much more mystical entity, a form of vitality that is supposed to be essential for life and keep us going. Nobody has been able to define or quantify this “energy”, it defies scientific measurement and is biologically implausible. These circumstances render Reiki one of the least plausible therapies in the tool kit of alternative medicine.

Reiki-healers (they prefer to be called “masters”) would channel “energy” into his or her patient which, in turn, is thought to stimulate the healing process of whatever condition is being treated. In the eyes of those who believe in this sort of thing, Reiki is therefore a true panacea: it can heal everything.

The clinical evidence for or against Reiki is fairly clear – as one would expect after realising how ‘far out’ its underlying concepts are. Numerous studies are available, but most are of very poor quality. Their results tend to suggest that patients experience benefit after having Reiki but they rarely exclude the possibility that this is due to placebo or other non-specific effects. Those that are rigorous show quite clearly that Reiki is a placebo. Our own review therefore concluded that “the evidence is insufficient to suggest that Reiki is an effective treatment for any condition… the value of Reiki remains unproven.”

Since the publication of our article, a number of new investigations have become available. In a brand-new study, for instance, the researchers wanted to explore a Reiki therapy-training program for the care-givers of paediatric patients. A series of Reiki training classes were offered by a Reiki-master. At the completion of the program, interviews were conducted to elicit participant’s feedback regarding its effectiveness.

Seventeen families agreed to participate and 65% of them attended three Reiki training sessions. They reported that Reiki had benefited their child by improving their comfort (76%), providing relaxation (88%) and pain relief (41%). All caregivers thought that becoming an active participant in their child’s care was a major gain. The authors of this investigation conclude that “a hospital-based Reiki training program for caregivers of hospitalized pediatric patients is feasible and can positively impact patients and their families. More rigorous research regarding the benefits of Reiki in the pediatric population is needed.

Trials like this one abound in the parallel world of “energy” medicine. In my view, such investigations do untold damage: they convince uncritical thinkers that “energy” healing is a rational and effective approach – so much so that even the military is beginning to use it.

The flaws in trials as the one above are too obvious to mention. Like most studies in this area, this new investigation proves nothing except the fact that poor quality research will mislead those who believe in its findings.

Some might say, so what? If a patient experiences benefit from a bogus yet harmless therapy, why not? I would strongly disagree with this increasingly popular view. Reiki and similarly bizarre forms of “energy” healing are well capable of causing harm.

Some fanatics might use these placebo-treatments as a true alternative to effective therapies. This would mean that the condition at hand remains untreated which, in a worst case scenario, might even lead to the death of patients. More important, in my view, is an entirely different risk: making people believe in mystic “energies” undermines rationality in a much more general sense. If this happens, the harm to society would be incalculable and extends far beyond health care.

Believe it or not, but my decision – all those years ago – to study medicine was to a significant degree influenced by a somewhat naive desire to, one day, be able to save lives. In my experience, most medical students are motivated by this wish – “to save lives” in this context stands not just for the dramatic act of administering a life-saving treatment to a moribund patient but it is meant as a synonym for helping patients in a much more general sense.

I am not sure whether, as a young clinician, I ever did manage to save many lives. Later, I had a career-change and became a researcher. The general view about researchers seems to be that they are detached from real life, sit in ivory towers and write clever papers which hardly anyone understands and few people will ever read. Researchers therefore cannot save lives, can they?

So, what happened to those laudable ambitions of the young Dr Ernst? Why did I decide to go into research, and why alternative medicine; why did I not conduct research in more the promotional way of so many of my colleagues (my life would have been so much more hassle-free, and I even might have a knighthood by now); why did I feel the need to insist on rigorous assessments and critical thinking, often at high cost? For my many detractors, the answers to these questions seem to be more than obvious: I was corrupted by BIG PHARMA, I have an axe to grind against all things alternative, I have an insatiable desire to be in the lime-light, I defend my profession against the concurrence from alternative practitioners etc. However, for me, the issues are a little less obvious (today, I will, for the first time, disclose the bribe I received from BIG PHARMA for criticising alternative medicine: the precise sum was zero £ and the same amount again in $).

As I am retiring from academic life and doing less original research, I do have the time and the inclination to brood over such questions. What precisely motivated my research agenda in alternative medicine, and why did I remain unimpressed by the number of powerful enemies I made pursuing it?

If I am honest – and I know this will sound strange to many, particularly to those who are convinced that I merely rejoice in being alarmist – I am still inspired by this hope to save lives. Sure, the youthful naivety of the early days has all but disappeared, yet the core motivation has remained unchanged.

But how can research into alternative medicine ever save a single life?

Since about 20 years, I am regularly pointing out that the most important research questions in my field relate to the risks of alternative medicine. I have continually published articles about these issues in the medical literature and, more recently, I have also made a conscious effort to step out of the ivory towers of academia and started writing for a much wider lay-audience (hence also this blog). Important landmarks on this journey include:

– pointing out that some forms of alternative medicine can cause serious complications, including deaths,

– disclosing that alternative diagnostic methods are unreliable and can cause serious problems,

– demonstrating that much of the advice given by alternative practitioners can cause serious harm to the patients who follow it,

– that the advice provided in books or on the Internet can be equally dangerous,

– and that even the most innocent yet ineffective therapy becomes life-threatening, once it is used to replace effective treatments for serious conditions.

Alternative medicine is cleverly, heavily and incessantly promoted as being natural and hence harmless. Several of my previous posts and the ensuing discussions on this blog strongly suggest that some chiropractors deny that their neck manipulations can cause a stroke. Similarly, some homeopaths are convinced that they can do no harm; some acupuncturists insist that their needles are entirely safe; some herbalists think that their medicines are risk-free, etc. All of them tend to agree that the risks are non-existent or so small that they are dwarfed by those of conventional medicine, thus ignoring that the potential risks of any treatment must be seen in relation to their proven benefit.

For 20 years, I have tried my best to dispel these dangerous myths and fallacies. In doing so, I had to fight many tough battles  (sometimes even with the people who should have protected me, e.g. my peers at Exeter university), and I have the scars to prove it. If, however, I did save just one life by conducting my research into the risks of alternative medicine and by writing about it, the effort was well worth it.

There probably is no area in health care that produces more surveys than alternative medicine. I estimate that about 500 surveys are published every year; this amounts to about two every working day which is substantially more than the number of clinical trials in this field.

I have long been critical of this ‘survey-mania’. The reason is simple: most of these articles are of such poor quality that they tell us nothing of value.

The vast majority of these surveys attempts to evaluate the prevalence of use of alternative medicine, and it is this type of investigation that I intend to discuss here.

For a typical prevalence survey, a team of enthusiastic researchers might put together a few questions and design a questionnaire to find out what percentage of a group of individuals have tried alternative medicine in the past. Subsequently, the investigators might get one or two hundred responses. They then calculate simple descriptive statistics and demonstrate that xy% (let’s assume it is 45%) use alternative medicine. This finding eventually gets published in one of the many alternative medicine journals, and everyone is happy – well, almost everybody.

How can I be such a spoil-sport and claim that this result tells us nothing of value? At the very minimum, some might argue, it shows that enthusiasts of alternative medicine are interested in and capable of conducting research. I beg to differ: this is not research, it is pseudo-research which ignores most of the principles of survey-design.

The typical alternative medicine prevalence survey has none of the features that would render it a scientific investigation:

1) It lacks an accepted definition of what is being surveyed. There is no generally accepted definition of alternative medicine, and even if the researchers address specific therapies, they run into huge problems. Take prayer, for instance – some see this as alternative medicine, while others would, of course, argue that it is a religious pursuit. Or take herbal medicine – many consumers confuse it with homeopathy, some might think that drinking tea is herbal medicine, while others would probably disagree.

2) The questionnaires used for such surveys are almost never validated. Essentially, this means that we cannot be sure they evaluate what we think they evaluate. We all know that the way we formulate a question can determine the answer. There are many potential sources of bias here, and they are rarely taken into consideration.

3) Enthusiastic researchers of alternative medicine usually use a  small convenience sample of participants for their surveys. This means they ask a few people who happen to be around to fill their questionnaire. As a consequence, there is no way the survey is representative of the population in question.

4) The typical survey has a low response rate; sometimes the response rate is not even provided or remains unknown even to the investigators. This means we do not know how the majority of patients/consumers who received but did not fill the questionnaire would have answered. Often there is good reason to suspect that those who have a certain attitude did respond, while those with a different opinion did not. This self-selection process is likely to produce misleading findings.

And why I am so sure about all of theses limitations? To my embarrassment, I know about them not least because I have made most these mistakes myself at some time in my career. You might also ask why this is important: what’s the harm in publishing a few flimsy surveys?

In my view, these investigations are regrettably counter-productive because:

they tend to grossly over-estimate the popularity of alternative medicine,

they distract money, manpower and attention from the truly important research questions in this field,

they give a false impression of a buoyant research activity,

and their results are constantly misused.

The last point is probably the most important one. The argument that is all too often spun around such survey data goes roughly as follows: a large percentage of the population uses alternative medicine; people pay out of their own pocket for these treatments; they are satisfied with them (if not, they would not pay for them). BUT THIS IS GROSSLY UNFAIR! Why should only those individuals who are rich enough to afford alternative medicine benefit from it? ALTERNATIVE MEDICINE SHOULD BE MADE AVAILABLE FOR ALL.

I rest my case.

Clinical trials of acupuncture can be quite challenging. In particular, it is often difficult to make sure that any observed outcome is truly due to the treatment and not caused by some other factor(s). How tricky this can be, shows a recently published study.

A new RCT has all (well, almost all) the features of a rigorous study. It tested the effects of acupuncture in patients suffering from hay fever. The German investigators recruited 46 specialized physicians in 6 hospital clinics and 32 private outpatient clinics. In total, 422 patients with IgE sensitization to birch and grass pollen were randomized into three groups: 1) acupuncture plus rescue medication (RM) (n= 212), 2) sham acupuncture plus RM (n= 102), or 3) RM alone (n= 108). Twelve acupuncture sessions were provided in groups 1 and 2 over 8 weeks. The outcome measures included changes in the Rhinitis Quality of Life Questionnaire (RQLQ) overall score and the RM score (RMs) from baseline to weeks 7, 8 and 16 in the first year as well as week 8 in the second year after randomization.

Compared with sham acupuncture and with RM, acupuncture was associated with improvement in RQLQ score and RMS. There were no differences after 16 weeks in the first year. After the 8-week follow-up phase in the second year, small improvements favoring real acupuncture over  sham were noted.

Based on these results, the authors concluded that “acupuncture led to statistically significant improvements in disease-specific quality of life and antihistamine use measures after 8 weeks of treatment compared with sham acupuncture and with RM alone, but the improvements may not be clinically significant.

The popular media were full of claims that this study proves the efficacy of acupuncture. However, I am not at all convinced that this conclusion is not hopelessly over-optimistic.

It might not have been the acupuncture itself that led to the observed improvements; they could well have been caused by several factors unrelated to the treatment itself. To understand my concern, we need to look closer at the actual interventions employed by the investigators.

The real acupuncture was done on acupuncture points thought to be indicated for hay fever. The needling was performed as one would normally do it, and the acupuncturists were asked to treat the patients in  group 1 in such a way that they were likely to experience the famous ‘de-qi’ feeling.

The sham acupuncture, by contrast, was performed on non-acupuncture points; acupuncturists were asked to use shallow needling only and they were instructed to try not to produce ‘de-qi’.

This means that the following factors in combination or alone could have caused [and in my view probably did cause] the observed differences in outcomes between the acupuncture and the sham group:

1) verbal or non-verbal communication between the acupuncturists and the patient [previous trials have shown this factor to be of crucial importance]

2) the visibly less deep needling in the sham-group

3) the lack of ‘de-qi’ experience in the sham-group.

Sham-treatments in clinical trials serve the purpose of a placebo. They are thus meant to be indistinguishable from the verum. If that is not the case [as in the present study], the trial cannot be accepted as being patient-blind. If a trial is not patient-blind, the expectations of patients will most certainly influence the results.

Therefore I believe that the marginal differences noted in this study were not due to the effects of acupuncture per se, but were an artifact caused through de-blinding of the patients. De facto, neither the patients nor the acupuncturists were blinded in this study.

If that is true, the effects were not just not clinically relevant, as noted by the authors, they also had nothing to do with acupuncture. In other words, acupuncture is not of proven efficacy for this condition – a verdict which is also supported by our systematic review of the subject which concluded that “the evidence for the effectiveness of acupuncture for the symptomatic treatment or prevention of allergic rhinitis is mixed. The results for seasonal allergic rhinitis failed to show specific effects of acupuncture…”

Once again, we have before us a study which looks impressive at first glance. At closer scrutiny, we find, however, that it had important design flaws which led to false positive results and conclusions. In my view, it would have been the responsibility of the authors to discuss these limitations in full detail and to draw conclusions that take them into account. Moreover, it would have been the duty of the peer-reviewers and journal editors to pick up on these points. Instead the editors even commissioned an accompanying editorial which displays an exemplary lack of critical thinking.

Having failed to do any of this, they are in my opinion all guilty of misleading the world media who reported extensively and often uncritically on this new study thus misleading us all. Sadly, the losers in this bonanza of incompetence are the many hay fever sufferers who will now be trying (and paying for) useless treatments.

Still in the spirit of ACUPUNCTURE AWARENESS WEEK, I have another critical look at a recent paper. If you trust some of the conclusions of this new article, you might think that acupuncture is an evidence-based treatment for coronary heart disease. I think this would be a recipe for disaster.

This condition affects millions and eventually kills a frighteningly large percentage of the population. Essentially, it is caused by the fact that, as we get older, the blood vessels supplying the heart also change, become narrower and get partially or even totally blocked. This causes lack of oxygen in the heart which causes pain known as angina pectoris. Angina is a most important warning sign indicating that a full blown heart attack might be not far.

The treatment of coronary heart disease consists in trying to let more blood flow through the narrowed coronaries, either by drugs or by surgery. At the same time, one attempts to reduce the oxygen demand of the heart, if possible. Normalisation of risk factors like hypertension and hypercholesterolaemia are key preventative strategies. It is not immediate clear to me how acupuncture might help in all this – but I have been wrong before!

The new meta-analysis included 16 individual randomised clinical trials. All had a high or moderate risk of bias. Acupuncture combined with conventional drugs (AC+CD) turned out to be superior to conventional drugs alone in reducing the incidence of acute myocardial infarction (AMI). AC+CD was superior to conventional drugs in reducing angina symptoms as well as in improving electrocardiography (ECG). Acupuncture by itself was also superior to conventional drugs for angina symptoms and ECG improvement. AC+CD was superior to conventional drugs in shortening the time to onset of angina relief. However, the time to onset was significantly longer for acupuncture treatment than for conventional treatment alone.

From these results, the authors [who are from the Chengdu University of Traditional Chinese Medicine in Sichuan, China] conclude that “AC+CD reduced the occurrence of AMI, and both acupuncture and AC+CD relieved angina symptoms and improved ECG. However, compared with conventional treatment, acupuncture showed a longer delay before its onset of action. This indicates that acupuncture is not suitable for emergency treatment of heart attack. Owing to the poor quality of the current evidence, the findings of this systematic review need to be verified by more RCTs to enhance statistical power.”

As in the meta-analysis discussed in my previous post, the studies are mostly Chinese, flawed, and not obtainable for an independent assessment. As in the previous article, I fail to see a plausible mechanism by which acupuncture might bring about the effects. This is not just a trivial or coincidental observation – I could cite dozens of systematic reviews for which the same criticism applies.

What is different, however, from the last post on gout is simple and important: if you treat gout with a therapy that is ineffective, you have more pain and eventually might opt for an effective one. If you treat coronary heart disease with a therapy that does not work, you might not have time to change, you might be dead.

Therefore I strongly disagree with the authors of this meta-analysis; “the findings of this systematic review need NOT to be verified by more RCTs to enhance statistical power” — foremost, I think, the findings need to be interpreted with much more caution and re-written. In fact, the findings show quite clearly that there is no good evidence to use acupuncture for coronary heart disease. To pretend otherwise is, in my view, not responsible.

There might be an important lesson here: A SEEMINGLY SLIGHT CORRECTION OF CONCLUSIONS OF SUCH SYSTEMATIC REVIEWS MIGHT SAVE LIVES.

This week is acupuncture awareness week, and I will use this occasion to continue focusing on this therapy. This first time ever event is supported by the British Acupuncture Council who state that it aims to “help better inform people about the ancient practice of traditional acupuncture. With 2.3 million acupuncture treatments carried out each year, acupuncture is one of the most popular complementary therapies practised in the UK today.

Right, let’s inform people about acupuncture then! Let’s show them that there is often more to acupuncture research than meets the eye.

My team and I have done lots of research into acupuncture and probably published more papers on this than any other subject. We had prominent acupuncturists on board from the UK, Korea, China and Japan, we ran conferences, published books and are proud to have been innovative and productive in our multidisciplinary research. But here I do not intend to dwell on our own achievements, rather I will highlight several important new papers in this area.

Korean authors just published a meta-analysis to assess the effectiveness of acupuncture as  therapy for gouty arthritis. Ten RCTs involving 852 gouty arthritis patients were included. Six studies of 512 patients reported a significant decrease in uric acid in the treatment group compared with a control group, while two studies of 120 patients reported no such effect. The remaining four studies of 380 patients reported a significant decrease in pain in the treatment group.

The authors conclude “that acupuncture is efficacious as complementary therapy for gouty arthritis patients”.

We should be delighted with such a positive and neat result! Why then do I hesitate and have doubts?

I believe that this paper reveals several important issues in relation to systematic reviews of Chinese acupuncture trials and studies of other TCM interventions. In fact, this is my main reason for discussing the new meta-analysis here. The following three points are crucial, in my view:

1) All the primary studies were from China, and 8 of the 10 were only available in Chinese.

2) All of them had major methodological flaws.

3) It has been shown repeatedly that all acupuncture-trials from China are positive.

Given this situation, the conclusions of any review for which there are only Chinese acupuncture studies might as well be written before the actual research has started. If the authors are pro-acupuncture, as the ones of the present article clearly are, they will conclude that “acupuncture is efficacious“. If the research team has some critical thinkers on board, the same evidence will lead to an entirely different conclusion, such as “due to the lack of rigorous trials, the evidence is less than compelling.

Systematic reviews are supposed to be the best type of evidence we currently have; they are supposed to guide therapeutic decisions. I find it unacceptable that one and the same set of data could be systematically analysed to generate such dramatically different outcomes. This is confusing and counter-productive!

So what is there to do? How can we prevent being misled by such articles? I think that medical journals should refuse to publish systematic reviews which so clearly lack sufficient critical input. I also believe that reviewers of predominantly Chinese studies should provide English translations of these texts so that they can be independently assessed by those who are not able to read Chinese – and for the sake of transparency, journal editors should insist on this point.

And what about the value of acupuncture for gouty arthritis? I think I let the readers draw their own conclusion.

Recent Comments

Note that comments can now be edited for up to five minutes after they are first submitted.


Click here for a comprehensive list of recent comments.

Categories