There probably is no area in health care that produces more surveys than alternative medicine. I estimate that about 500 surveys are published every year; this amounts to about two every working day which is substantially more than the number of clinical trials in this field.
I have long been critical of this ‘survey-mania’. The reason is simple: most of these articles are of such poor quality that they tell us nothing of value.
The vast majority of these surveys attempts to evaluate the prevalence of use of alternative medicine, and it is this type of investigation that I intend to discuss here.
For a typical prevalence survey, a team of enthusiastic researchers might put together a few questions and design a questionnaire to find out what percentage of a group of individuals have tried alternative medicine in the past. Subsequently, the investigators might get one or two hundred responses. They then calculate simple descriptive statistics and demonstrate that xy% (let’s assume it is 45%) use alternative medicine. This finding eventually gets published in one of the many alternative medicine journals, and everyone is happy – well, almost everybody.
How can I be such a spoil-sport and claim that this result tells us nothing of value? At the very minimum, some might argue, it shows that enthusiasts of alternative medicine are interested in and capable of conducting research. I beg to differ: this is not research, it is pseudo-research which ignores most of the principles of survey-design.
The typical alternative medicine prevalence survey has none of the features that would render it a scientific investigation:
1) It lacks an accepted definition of what is being surveyed. There is no generally accepted definition of alternative medicine, and even if the researchers address specific therapies, they run into huge problems. Take prayer, for instance – some see this as alternative medicine, while others would, of course, argue that it is a religious pursuit. Or take herbal medicine – many consumers confuse it with homeopathy, some might think that drinking tea is herbal medicine, while others would probably disagree.
2) The questionnaires used for such surveys are almost never validated. Essentially, this means that we cannot be sure they evaluate what we think they evaluate. We all know that the way we formulate a question can determine the answer. There are many potential sources of bias here, and they are rarely taken into consideration.
3) Enthusiastic researchers of alternative medicine usually use a small convenience sample of participants for their surveys. This means they ask a few people who happen to be around to fill their questionnaire. As a consequence, there is no way the survey is representative of the population in question.
4) The typical survey has a low response rate; sometimes the response rate is not even provided or remains unknown even to the investigators. This means we do not know how the majority of patients/consumers who received but did not fill the questionnaire would have answered. Often there is good reason to suspect that those who have a certain attitude did respond, while those with a different opinion did not. This self-selection process is likely to produce misleading findings.
And why I am so sure about all of theses limitations? To my embarrassment, I know about them not least because I have made most these mistakes myself at some time in my career. You might also ask why this is important: what’s the harm in publishing a few flimsy surveys?
In my view, these investigations are regrettably counter-productive because:
they tend to grossly over-estimate the popularity of alternative medicine,
they distract money, manpower and attention from the truly important research questions in this field,
they give a false impression of a buoyant research activity,
and their results are constantly misused.
The last point is probably the most important one. The argument that is all too often spun around such survey data goes roughly as follows: a large percentage of the population uses alternative medicine; people pay out of their own pocket for these treatments; they are satisfied with them (if not, they would not pay for them). BUT THIS IS GROSSLY UNFAIR! Why should only those individuals who are rich enough to afford alternative medicine benefit from it? ALTERNATIVE MEDICINE SHOULD BE MADE AVAILABLE FOR ALL.
I rest my case.
Clinical trials of acupuncture can be quite challenging. In particular, it is often difficult to make sure that any observed outcome is truly due to the treatment and not caused by some other factor(s). How tricky this can be, shows a recently published study.
A new RCT has all (well, almost all) the features of a rigorous study. It tested the effects of acupuncture in patients suffering from hay fever. The German investigators recruited 46 specialized physicians in 6 hospital clinics and 32 private outpatient clinics. In total, 422 patients with IgE sensitization to birch and grass pollen were randomized into three groups: 1) acupuncture plus rescue medication (RM) (n= 212), 2) sham acupuncture plus RM (n= 102), or 3) RM alone (n= 108). Twelve acupuncture sessions were provided in groups 1 and 2 over 8 weeks. The outcome measures included changes in the Rhinitis Quality of Life Questionnaire (RQLQ) overall score and the RM score (RMs) from baseline to weeks 7, 8 and 16 in the first year as well as week 8 in the second year after randomization.
Compared with sham acupuncture and with RM, acupuncture was associated with improvement in RQLQ score and RMS. There were no differences after 16 weeks in the first year. After the 8-week follow-up phase in the second year, small improvements favoring real acupuncture over sham were noted.
Based on these results, the authors concluded that “acupuncture led to statistically significant improvements in disease-specific quality of life and antihistamine use measures after 8 weeks of treatment compared with sham acupuncture and with RM alone, but the improvements may not be clinically significant.”
The popular media were full of claims that this study proves the efficacy of acupuncture. However, I am not at all convinced that this conclusion is not hopelessly over-optimistic.
It might not have been the acupuncture itself that led to the observed improvements; they could well have been caused by several factors unrelated to the treatment itself. To understand my concern, we need to look closer at the actual interventions employed by the investigators.
The real acupuncture was done on acupuncture points thought to be indicated for hay fever. The needling was performed as one would normally do it, and the acupuncturists were asked to treat the patients in group 1 in such a way that they were likely to experience the famous ‘de-qi’ feeling.
The sham acupuncture, by contrast, was performed on non-acupuncture points; acupuncturists were asked to use shallow needling only and they were instructed to try not to produce ‘de-qi’.
This means that the following factors in combination or alone could have caused [and in my view probably did cause] the observed differences in outcomes between the acupuncture and the sham group:
1) verbal or non-verbal communication between the acupuncturists and the patient [previous trials have shown this factor to be of crucial importance]
2) the visibly less deep needling in the sham-group
3) the lack of ‘de-qi’ experience in the sham-group.
Sham-treatments in clinical trials serve the purpose of a placebo. They are thus meant to be indistinguishable from the verum. If that is not the case [as in the present study], the trial cannot be accepted as being patient-blind. If a trial is not patient-blind, the expectations of patients will most certainly influence the results.
Therefore I believe that the marginal differences noted in this study were not due to the effects of acupuncture per se, but were an artifact caused through de-blinding of the patients. De facto, neither the patients nor the acupuncturists were blinded in this study.
If that is true, the effects were not just not clinically relevant, as noted by the authors, they also had nothing to do with acupuncture. In other words, acupuncture is not of proven efficacy for this condition – a verdict which is also supported by our systematic review of the subject which concluded that “the evidence for the effectiveness of acupuncture for the symptomatic treatment or prevention of allergic rhinitis is mixed. The results for seasonal allergic rhinitis failed to show specific effects of acupuncture…”
Once again, we have before us a study which looks impressive at first glance. At closer scrutiny, we find, however, that it had important design flaws which led to false positive results and conclusions. In my view, it would have been the responsibility of the authors to discuss these limitations in full detail and to draw conclusions that take them into account. Moreover, it would have been the duty of the peer-reviewers and journal editors to pick up on these points. Instead the editors even commissioned an accompanying editorial which displays an exemplary lack of critical thinking.
Having failed to do any of this, they are in my opinion all guilty of misleading the world media who reported extensively and often uncritically on this new study thus misleading us all. Sadly, the losers in this bonanza of incompetence are the many hay fever sufferers who will now be trying (and paying for) useless treatments.
Still in the spirit of ACUPUNCTURE AWARENESS WEEK, I have another critical look at a recent paper. If you trust some of the conclusions of this new article, you might think that acupuncture is an evidence-based treatment for coronary heart disease. I think this would be a recipe for disaster.
This condition affects millions and eventually kills a frighteningly large percentage of the population. Essentially, it is caused by the fact that, as we get older, the blood vessels supplying the heart also change, become narrower and get partially or even totally blocked. This causes lack of oxygen in the heart which causes pain known as angina pectoris. Angina is a most important warning sign indicating that a full blown heart attack might be not far.
The treatment of coronary heart disease consists in trying to let more blood flow through the narrowed coronaries, either by drugs or by surgery. At the same time, one attempts to reduce the oxygen demand of the heart, if possible. Normalisation of risk factors like hypertension and hypercholesterolaemia are key preventative strategies. It is not immediate clear to me how acupuncture might help in all this – but I have been wrong before!
The new meta-analysis included 16 individual randomised clinical trials. All had a high or moderate risk of bias. Acupuncture combined with conventional drugs (AC+CD) turned out to be superior to conventional drugs alone in reducing the incidence of acute myocardial infarction (AMI). AC+CD was superior to conventional drugs in reducing angina symptoms as well as in improving electrocardiography (ECG). Acupuncture by itself was also superior to conventional drugs for angina symptoms and ECG improvement. AC+CD was superior to conventional drugs in shortening the time to onset of angina relief. However, the time to onset was significantly longer for acupuncture treatment than for conventional treatment alone.
From these results, the authors [who are from the Chengdu University of Traditional Chinese Medicine in Sichuan, China] conclude that “AC+CD reduced the occurrence of AMI, and both acupuncture and AC+CD relieved angina symptoms and improved ECG. However, compared with conventional treatment, acupuncture showed a longer delay before its onset of action. This indicates that acupuncture is not suitable for emergency treatment of heart attack. Owing to the poor quality of the current evidence, the findings of this systematic review need to be verified by more RCTs to enhance statistical power.”
As in the meta-analysis discussed in my previous post, the studies are mostly Chinese, flawed, and not obtainable for an independent assessment. As in the previous article, I fail to see a plausible mechanism by which acupuncture might bring about the effects. This is not just a trivial or coincidental observation – I could cite dozens of systematic reviews for which the same criticism applies.
What is different, however, from the last post on gout is simple and important: if you treat gout with a therapy that is ineffective, you have more pain and eventually might opt for an effective one. If you treat coronary heart disease with a therapy that does not work, you might not have time to change, you might be dead.
Therefore I strongly disagree with the authors of this meta-analysis; “the findings of this systematic review need NOT to be verified by more RCTs to enhance statistical power” — foremost, I think, the findings need to be interpreted with much more caution and re-written. In fact, the findings show quite clearly that there is no good evidence to use acupuncture for coronary heart disease. To pretend otherwise is, in my view, not responsible.
There might be an important lesson here: A SEEMINGLY SLIGHT CORRECTION OF CONCLUSIONS OF SUCH SYSTEMATIC REVIEWS MIGHT SAVE LIVES.
This week is acupuncture awareness week, and I will use this occasion to continue focusing on this therapy. This first time ever event is supported by the British Acupuncture Council who state that it aims to “help better inform people about the ancient practice of traditional acupuncture. With 2.3 million acupuncture treatments carried out each year, acupuncture is one of the most popular complementary therapies practised in the UK today.“
Right, let’s inform people about acupuncture then! Let’s show them that there is often more to acupuncture research than meets the eye.
My team and I have done lots of research into acupuncture and probably published more papers on this than any other subject. We had prominent acupuncturists on board from the UK, Korea, China and Japan, we ran conferences, published books and are proud to have been innovative and productive in our multidisciplinary research. But here I do not intend to dwell on our own achievements, rather I will highlight several important new papers in this area.
Korean authors just published a meta-analysis to assess the effectiveness of acupuncture as therapy for gouty arthritis. Ten RCTs involving 852 gouty arthritis patients were included. Six studies of 512 patients reported a significant decrease in uric acid in the treatment group compared with a control group, while two studies of 120 patients reported no such effect. The remaining four studies of 380 patients reported a significant decrease in pain in the treatment group.
The authors conclude “that acupuncture is efficacious as complementary therapy for gouty arthritis patients”.
We should be delighted with such a positive and neat result! Why then do I hesitate and have doubts?
I believe that this paper reveals several important issues in relation to systematic reviews of Chinese acupuncture trials and studies of other TCM interventions. In fact, this is my main reason for discussing the new meta-analysis here. The following three points are crucial, in my view:
1) All the primary studies were from China, and 8 of the 10 were only available in Chinese.
2) All of them had major methodological flaws.
3) It has been shown repeatedly that all acupuncture-trials from China are positive.
Given this situation, the conclusions of any review for which there are only Chinese acupuncture studies might as well be written before the actual research has started. If the authors are pro-acupuncture, as the ones of the present article clearly are, they will conclude that “acupuncture is efficacious“. If the research team has some critical thinkers on board, the same evidence will lead to an entirely different conclusion, such as “due to the lack of rigorous trials, the evidence is less than compelling.“
Systematic reviews are supposed to be the best type of evidence we currently have; they are supposed to guide therapeutic decisions. I find it unacceptable that one and the same set of data could be systematically analysed to generate such dramatically different outcomes. This is confusing and counter-productive!
So what is there to do? How can we prevent being misled by such articles? I think that medical journals should refuse to publish systematic reviews which so clearly lack sufficient critical input. I also believe that reviewers of predominantly Chinese studies should provide English translations of these texts so that they can be independently assessed by those who are not able to read Chinese – and for the sake of transparency, journal editors should insist on this point.
And what about the value of acupuncture for gouty arthritis? I think I let the readers draw their own conclusion.
“They would say that, wouldn’t they?” is the quote attributed to Mandy Rice-Davies giving witness in the Profumo affair. I think, it aptly highlights some of the issues related to conflicts of interest in health care.
These days, when a researcher publishes a paper, he will in all likelihood have to disclose all conflicts of interest he might have. The aim of this exercise is to be as transparent as possible; if someone has received support from a commercial company, for example, this fact does not necessarily follow that his paper is biased, but it is important to lay open the fact so that the readers can make up their own minds.
The questionnaires that authors have to complete prior to publication of their article focus almost exclusively on financial issues. For instance, one has to disclose any sponsorship, fees, travel support or shares that one might own in a company. In conventional medicine, these matters are deemed to be the most important sources for potential conflicts of interest.
In alternative medicine, financial issues are generally thought to be far less critical; it is generally seen as an area where there is so little money that it is hardly worth bothering. Perhaps this is the reason why few journals in this field insist on declarations of conflicts of interests and few authors disclose them.
After having been a full-time researcher of alternative medicine for two decades, I have become convinced that conflicts of interest are at least as prevalent and powerful in this field as in any other area of health care. Sure, there is less money at stake, but this fact is more than compensated by non-financial issues. Quasi-evangelic convictions abound in alternative medicine and it is, I think, obvious that they can amount to significant conflicts of interest.
During their training, alternative practitioners are being taught many things which are unproven, have no basis in fact or are just plainly wrong. Eventually this schooling can create a belief system which often is adhered to regardless of the scientific evidence and which tends to be defended at all cost. As some of my readers are bound to object to this remark, I better cite an example: during their training, students of chiropractic develop a more and more firm stance against immunization which in all likelihood is due to the type of information they receive at the chiropractic college. There is no question in my mind that creeds can represent an even more powerful conflict of interest than financial matters.
Moreover, this belief is indivisibly intertwined with existential issues. In alternative medicine, there may not be huge amounts of money at stake but practitioners’ livelihoods are perceived to be at risk. If an acupuncturist, for instance, argues in favour of his therapy, he also consciously or sub-consciously is trying to protect his income.
Some might say that this not different from conventional medicine, but I disagree: if we take away one specific therapy from a doctor because it turns out to be useless or unsafe, he will be able to use another one; if we take the acupuncture needle away from an acupuncturist, we have deprived him of his livelihood.
This is why conflicts of interest in alternative medicine tend to be very acute, powerful and personal. And this is why enthusiasts of alternative medicine are incapable or unwilling to look upon any type of critical assessment of their area as anything else than an attack on their income, their beliefs, their status, their training or their person. If anyone should doubt it, I recommend studying the comments I received to previous posts of this blog.
When Mandi Rice-Davies gave evidence during the trial of Stephen Ward, the osteopath who had introduced her to influential clients, the prosecuting council noted that Lord Astor denied having had an affair with her. Mrs Rice-Davies allegedly replied “Well, he would say that, wouldn’t he?” (Actually, she did not say these exact words but something rather similar) When I read the comments following my posts on this blog, I am often reminded of this now classical quote.
When chiropractors deny that neck manipulations carry a risk, when herbalists insist that traditional herbalism is based on good evidence, when homeopaths claim that their remedies are more than placebos, I believe we should ask who, in these debates, might have a conflict of interest.
Is there a circumstance of one party in the discussion where personal interests might benefit from the argument? Who is more likely to be objective, the person whose livelihood is endangered or the independent expert who studied the subject in depth but has no axe to grind? If you ask these questions, you might conclude as I frequently do: “they would say that, wouldn’t they?”