study design

Conventional cough syrups do not have the best of reputations – but the repute of homeopathic cough syrups is certainly not encouraging. So what should one do with such a preparation? Forget about it? No, one conducts a clinical trial, of course! Not just any old trial but one where science, ethics and common sense are absent. Here are the essentials of a truly innovative study that, I think, has all of these remarkable qualities:

The present prospective observational study investigated children affected by wet acute cough caused by non-complicated URTIs, comparing those who received the homeopathic syrup versus those treated with the homeopathic syrup plus antibiotic. The aims were: 1) to assess whether the addition of antibiotics to a symptomatic treatment had a role in reducing the severity and duration of acute cough in a pediatric population, as well as in improving cough resolution; 2) to verify the safety of the two treatments. Eighty-five children were enrolled in an open study: 46 children received homeopathic syrup alone for 10 days and 39 children received homeopathic syrup for 10 days plus oral antibiotic treatment (amoxicillin/clavulanate, clarithromycin, and erythromycin) for 7 days. To assess cough severity we used a subjective verbal category-descriptive (VCD) scale. Cough VCD score was significantly (P < 0.001) reduced in both groups starting from the second day of treatment (−0.52 ± 0.66 in the homeopathic syrup group and −0.56 ± 0.55 in children receiving homeopathic syrup plus oral antibiotic treatment). No significant differences in cough severity or resolution were found between the two groups of children in any of the 28 days of the study. After the first week (day 8) cough was completely resolved in more than one-half of patients in both groups. Two children (4.3 %) reported adverse effects in the group treated with the homeopathic syrup alone, versus 9 children (23.1 %) in the group treated with the homeopathic syrup plus antibiotics (P = 0.020).


Our data confirm that the homeopathic treatment in question has potential benefits for cough in children as well, and highlight the strong safety profile of this treatment. Additional antibiotic prescription was not associated with a greater cough reduction, and presented more adverse events than the homeopathic syrup alone.

Let us be clear about what has happened here. I think, the events can be summarised as follows:

  • the researchers come across a homeopathic syrup (anyone who understands respiratory problems and/or therapeutics would be more than a little suspicious of this product, but this team is exceptional),
  • they decide to do a trial with it (a decision which would make some ethicists already quite nervous, but the ethics committee is exceptional too),
  • the question raises, what should the researchers give to the control group?
  • someone has the idea, why not compare our dodgy syrup against something that is equally dodgy, perhaps even a bit unsafe?
  • the researchers are impressed and ask: but what precisely could we use?
  • let’s take antibiotics; they are often used for acute coughs, but the best evidence fails to show that they are helpful and they have, of course, risks,
  • another member of the team adds: let’s use children, they and their mothers are unlikely to understand what we are up to,
  • the team is in agreement,
  • Boiron, the world’s largest producer of homeopathic products, accepts to finance the study,
  • a protocol is written,
  • ethics approval is obtained,
  • the trial is conducted and even published by a journal with the help of peer-reviewers who are less than critical.

And the results of the trial? Contrary to the authors’ conclusion copied above, they show that two bogus treatments are worse that one.



In my last post, I claimed that researchers of alternative medicine tend to be less than rigorous. I did not link this statement to any evidence at all. Perhaps I should have at least provided an example!? As it happens, I just came across a brand new paper which nicely demonstrates what I meant.

According to its authors, this non-interventional study was performed to generate data on safety and treatment effects of a complex homeopathic drug. They treated 1050 outpatients suffering from common cold with a commercially available homeopathic remedy for 8 days. The study was conducted in 64 German outpatient practices of medical doctors trained in CAM. Tolerability, compliance and the treatment effects were assessed by the physicians and by patient diaries. Adverse events were collected and assessed with specific attention to homeopathic aggravation and proving symptoms. Each adverse effect was additionally evaluated by an advisory board of experts.

The physicians detected 60 adverse events from 46 patients (4.4%). Adverse drug reactions occurred in 14 patients (1.3%). Six patients showed proving symptoms (0.57%) and only one homeopathic aggravation (0.1%) appeared. The rate of compliance was 84% for all groups. The global assessment of the treatment effects resulted in the verdict “good” and “very good” in 84.9% of all patients.

The authors concluded that the homeopathic complex drug was shown to be safe and effective for children and adults likewise. Adverse reactions specifically related to homeopathic principles are very rare. All observed events recovered quickly and were of mild to moderate intensity.

So why do I think this is ‘positively barmy’?

The study had no control group. This means that there is no way anyone can attribute the observed ‘treatment effects’ to the homeopathic remedy. There are many other phenomena that may have caused or contributed to it, e. g.:

  • a placebo effect
  • the natural history of the condition
  • regression to the mean
  • other treatments which the patients took but did not declare
  • the empathic encounter with the physician
  • social desirability

To plan a study with the aim as stated above and to draw the conclusion as cited above is naïve and unprofessional (to say the least) on the part of the researchers (I often wonder where, in such cases, the boundary between incompetence and research misconduct might lie). To pass such a paper through the peer review process is negligent on the part of the reviewers. To publish the article is irresponsible on the part of the editor.


Twenty years ago, I published a short article in the British Journal of Rheumatology. Its title was ALTERNATIVE MEDICINE, THE BABY AND THE BATH WATER. Reading it again today – especially in the light of the recent debate (with over 700 comments) on acupuncture – indicates to me that very little has since changed in the discussions about alternative medicine (AM). Does that mean we are going around in circles? Here is the (slightly abbreviated) article from 1995 for you to judge for yourself:

“Proponents of alternative medicine (AM) criticize the attempt of conducting RCTs because they view this is in analogy to ‘throwing out the baby with the bath water’. The argument usually goes as follows: the growing popularity of AM shows that individuals like it and, in some way, they benefit through using it. Therefore it is best to let them have it regardless of its objective effectiveness. Attempts to prove or disprove effectiveness may even be counterproductive. Should RCTs prove that a given intervention is not superior to a placebo, one might stop using it. This, in turn, would be to the disadvantage of the patient who, previous to rigorous research, has unquestionably been helped by the very remedy. Similar criticism merely states that AM is ‘so different, so subjective, so sensitive that it cannot be investigated in the same way as mainstream medicine’. Others see reasons to change the scientific (‘reductionist’) research paradigm into a broad ‘philosophical’ approach. Yet others reject the RCTs because they think that ‘this method assumes that every person has the same problems and there are similar causative factors’.

The example of acupuncture as a (popular) treatment for osteoarthritis, demonstrates the validity of such arguments and counter-arguments. A search of the world literature identified only two RCTs on the subject. When acupuncture was tested against no treatment, the experimental group of osteoarthritis sufferers reported a 23% decrease of pain, while the controls suffered a 12% increase. On the basis of this result, it might seem highly unethical to withhold acupuncture from pain-stricken patients—’if a patient feels better for whatever reason and there are no toxic side effects, then the patient should have the right to get help’.

But what about the placebo effect? It is notoriously difficult to find a placebo indistinguishable to acupuncture which would allow patient-blinded studies. Needling non-acupuncture points may be as close as one can get to an acceptable placebo. When patients with osteoarthritis were randomized into receiving either ‘real acupuncture or this type of sham acupuncture both sub-groups showed the same pain relief.

These findings (similar results have been published for other AMs) are compatible only with two explanations. Firstly acupuncture might be a powerful placebo. If this were true, we need to establish how safe acupuncture is (clearly it is not without potential harm); if the risk/benefit ratio is favourable and no specific, effective form of therapy exists one might still consider employing this form as a ‘placebo therapy’ for easing the pain of osteoarthritis sufferers. One would also feel motivated to research this powerful placebo and identify its characteristics or modalities with the aim of using the knowledge thus generated to help future patients.

Secondly, it could be the needling, regardless of acupuncture points and philosophy, that decreases pain. If this were true, we could henceforward use needling for pain relief—no special training in or equipment for acupuncture would be required, and costs would therefore be markedly reduced. In addition, this knowledge would lead us to further our understanding of basic mechanisms of pain reduction which, one day, might evolve into more effective analgesia. In any case the published research data, confusing as they often are, do not call for a change of paradigm; they only require more RCTs to solve the unanswered problems.

Conducting rigorous research is therefore by no means likely to ‘throw out the baby with the bath water’. The concept that such research could harm the patient is wrong and anti-scientific. To follow its implications would mean neglecting the ‘baby in the bath water’ until it suffers serious damage. To conduct proper research means attending the ‘baby’ and making sure that it is safe and well.

Reiki is a form of energy healing that evidently has been getting so popular that, according to the ‘Shropshire Star’, even stressed hedgehogs are now being treated with this therapy. In case you argue that this publication is not cutting edge when it comes to reporting of scientific advances, you may have a point. So, let us see what evidence we find on this amazing intervention.

A recent systematic review of the therapeutic effects of Reiki concludes that the serious methodological and reporting limitations of limited existing Reiki studies preclude a definitive conclusion on its effectiveness. High-quality randomized controlled trials are needed to address the effectiveness of Reiki over placebo. Considering that this article was published in the JOURNAL OF ALTERNATIVE AND COMPLEMENTARY MEDICINE, this is a fairly damming verdict. The notion that Reiki is but a theatrical placebo recently received more support from a new clinical trial.

This pilot study examined the effects of Reiki therapy and companionship on improvements in quality of life, mood, and symptom distress during chemotherapy. Thirty-six breast cancer patients received usual care, Reiki, or a companion during chemotherapy. Data were collected from patients while they were receiving usual care. Subsequently, patients were randomized to either receive Reiki or a companion during chemotherapy. Questionnaires assessing quality of life, mood, symptom distress, and Reiki acceptability were completed at baseline and chemotherapy sessions 1, 2, and 4. Reiki was rated relaxing and caused no side effects. Both Reiki and companion groups reported improvements in quality of life and mood that were greater than those seen in the usual care group.

The authors of this study conclude that interventions during chemotherapy, such as Reiki or companionship, are feasible, acceptable, and may reduce side effects.

This is an odd conclusion, if there ever was one. Clearly the ‘companionship’ group was included to see whether Reiki has effects beyond simply providing sympathetic attention. The results show that this is not the case. It follows, I think, that Reiki is a placebo; its perceived relaxing effects are the result of non-specific phenomena which have nothing to do with Reiki per se. The fact that the authors fail to spell this out more clearly makes me wonder whether they are researchers or promoters of Reiki.

Some people will feel that it does not matter how Reiki works, the main thing is that it does work. I beg to differ!

If its effects are due to nothing else than attention and companionship, we do not need ‘trained’ Reiki masters to do the treatment; anyone who has time, compassion and sympathy can do it. More importantly, if Reiki is a placebo, we should not mislead people that some super-natural energy is at work. This only promotes irrationality – and, as Voltaire once said: those who make you believe in absurdities can make you commit atrocities.

A special issue of Medical Care has just been published; it was sponsored by the Veterans Health Administration’s Office of Patient Centered Care and Cultural Transformation. A press release made the following statement about it:

Complementary and alternative medicine therapies are increasingly available, used, and appreciated by military patients, according to Drs Taylor and Elwy. They cite statistics showing that CAM programs are now offered at nearly 90 percent of VA medical facilities. Use CAM modalities by veterans and active military personnel is as at least as high as in the general population.

If you smell a bit of the old ad populum fallacy here, you may be right. But let’s look at the actual contents of the special issue. The most interesting article is about a study testing acupuncture for posttraumatic stress disorder (PTSD).

Fifty-five service members meeting research diagnostic criteria for PTSD were randomized to usual PTSD care (UPC) plus eight 60-minute sessions of acupuncture conducted twice weekly or to UPC alone. Outcomes were assessed at baseline and 4, 8, and 12 weeks postrandomization. The primary study outcomes were difference in PTSD symptom improvement on the PTSD Checklist (PCL) and the Clinician-administered PTSD Scale (CAPS) from baseline to 12-week follow-up between the two treatment groups. Secondary outcomes were depression, pain severity, and mental and physical health functioning. Mixed model regression and t test analyses were applied to the data.

The results show that the mean improvement in PTSD severity was significantly greater among those receiving acupuncture than in those receiving UPC. Acupuncture was also associated with significantly greater improvements in depression, pain, and physical and mental health functioning. Pre-post effect-sizes for these outcomes were large and robust.

The authors conclude from these data that acupuncture was effective for reducing PTSD symptoms. Limitations included small sample size and inability to parse specific treatment mechanisms. Larger multisite trials with longer follow-up, comparisons to standard PTSD treatments, and assessments of treatment acceptability are needed. Acupuncture is a novel therapeutic option that may help to improve population reach of PTSD treatment.

What shall we make of this?

I know I must sound like a broken record to some, but I have strong reservations that the interpretation provided here is correct. One does not even need to be a ‘devil’s advocate’ to point out that the observed outcomes may have nothing at all to do with acupuncture per se. A much more rational interpretation of the findings would be that the 8 times 60 minutes of TLC and attention have positive effects on the subjective symptoms of soldiers suffering from PTSD. No needles required for this to happen; and no mystical chi, meridians, life forces etc.

It would, of course, have been quite easy to design the study such that the extra attention is controlled for. But the investigators evidently did not want to do that. They seemed to have the desire to conduct a study where the outcome was clear even before the first patient had been recruited. That some if not most experts would call this poor science or even unethical may not have been their primary concern.

The question I ask myself is, why did the authors of this study fail to express the painfully obvious fact that the results are most likely unrelated to acupuncture? Is it because, in military circles, Occam’s razor is not on the curriculum? Is it because critical thinking has gone out of fashion ( – no, it is not even critical thinking to point out something that is more than obvious)? Is it then because, in the present climate, it is ‘politically’ correct to introduce a bit of ‘holistic touchy feely’ stuff into military medicine?

I would love to hear what my readers think.

One of the problems regularly encountered when evaluating the effectiveness of chiropractic spinal manipulation is that there are numerous chiropractic spinal manipulative techniques and clinical trials rarely provide an exact means of differentiating between them. Faced with a negative studies, chiropractors might therefore argue that the result was negative because the wrong techniques were used; therefore they might insist that it does not reflect chiropractic in a wider sense. Others claim that even a substantial body of negative evidence does not apply to chiropractic as a whole because there is a multitude of techniques that have not yet been properly tested. It seems as though the chiropractic profession wants the cake and eat it.

Amongst the most commonly used is the ‘DIVERSIFIED TECHNIQUE’ (DT) which has been described as follows: Like many chiropractic and osteopathic manipulative techniques, Diversified is characterized by a high velocity low amplitude thrust. Diversified is considered the most generic chiropractic manipulative technique and is differentiated from other techniques in that its objective is to restore proper movement and alignment of spine and joint dysfunction.

Also widely used is a technique called ‘FLEXION DISTRACTION’ (FD) which involves the use of a specialized table that gently distracts or stretches the spine and which allows the chiropractor to isolate the area of disc involvement while slightly flexing the spine in a pumping rhythm.

The ‘ACTIVATOR TECHNIQUE’ (AT) seems a little less popular; it involves having the patient lie in a prone position and comparing the functional leg lengths. Often one leg will seem to be shorter than the other. The chiropractor then carries out a series of muscle tests such as having the patient move their arms in a certain position in order to activate the muscles attached to specific vertebrae. If the leg lengths are not the same, that is taken as a sign that the problem is located at that vertebra. The chiropractor treats problems found in this way moving progressively along the spine in the direction from the feet towards the head. The activator is a small handheld spring-loaded instrument which delivers a small impulse to the spine. It was found to give off no more than 0.3 J of kinetic energy in a 3-millisecond pulse. The aim is to produce enough force to move the vertebrae but not enough to cause injury.

There is limited research comparing the effectiveness of these and the many other techniques used by chiropractors, and the few studies that are available are usually less than rigorous and their findings are thus unreliable. A first step in researching this rather messy area would be to determine which techniques are most frequently employed.

The aim of this new investigation was to do just that, namely to provide insight into which treatment approaches are used most frequently by Australian chiropractors to treat spinal musculoskeletal conditions.

A questionnaire was sent online to the members of the two main Australian chiropractic associations in 2013. The participants were asked to provide information on treatment choices for specific spinal musculoskeletal conditions.

A total of 280 responses were received. DT was the first choice of treatment for most of the included conditions. DT was used significantly less in 4 conditions: cervical disc syndrome with radiculopathy and cervical central stenosis were more likely to be treated with AT. FD was used almost as much as DT in the treatment of lumbar disc syndrome with radiculopathy and lumbar central stenosis. More experienced Australian chiropractors use more AT and soft tissue therapy and less DT compared to their less experienced chiropractors. The majority of the responding chiropractors also used ancillary procedures such as soft tissue techniques and exercise prescription in the treatment of spinal musculoskeletal conditions.

The authors concluded that this survey provides information on commonly used treatment choices to the chiropractic profession. Treatment choices changed based on the region of disorder and whether neurological symptoms were present rather than with specific diagnoses. Diversified technique was the most commonly used spinal manipulative therapy, however, ancillary procedures such as soft tissue techniques and exercise prescription were also commonly utilised. This information may help direct future studies into the efficacy of chiropractic treatment for spinal musculoskeletal disorders.

I am a little less optimistic that this information will help to direct future research. Critical readers might have noticed that the above definitions of two commonly used techniques are rather vague, particularly that of DT.

Why is that so? The answer seems to be that even chiropractors are at a loss coming up with a good definition of their most-used therapeutic techniques. I looked hard for a more precise definition but the best I could find was this: Diversified is characterized by the manual delivery of a high velocity low amplitude thrust to restricted joints of the spine and the extremities. This is known as an adjustment and is performed by hand. Virtually all joints of the body can be adjusted to help restore proper range of motion and function. Initially a functional and manual assessment of each joint’s range and quality of motion will establish the location and degree of joint dysfunction. The patient will then be positioned depending on the region being adjusted when a specific, quick impulse will be delivered through the line of the joint in question. The direction, speed, depth and angles that are used are the product of years of experience, practice and a thorough understanding of spinal mechanics. Often a characteristic ‘crack’ or ‘pop’ may be heard during the process. This is perfectly normal and is nothing to worry about. It is also not a guide as to the value or effectiveness of the adjustment.

This means that the DT is not a single method but a hotchpotch of techniques; this assumption is also confirmed by the following quote: The diversified technique is a technique used by chiropractors that is composed of all other techniques. It is the most commonly used technique and primarily focuses on spinal adjustments to restore function to vertebral and spinal problems.

What does that mean for research into chiropractic spinal manipulation? It means, I think, that even if we manage to define that a study was to test the effectiveness of one named chiropractic technique, such as DT, the chiropractors doing the treatments would most likely do what they believe is required for each individual patient.

There is, of course, nothing wrong with that approach; it is used in many other area of health care as well. In such cases, we need to view the treatment as something like a ‘black box’; we test the effectiveness of the black box without attempting to define its exact contents, and we trust that the clinicians in the trial are well-trained to use the optimal mix of techniques as needed for each individual patient.

I would assume that, in most studies available to date, this is precisely what already has been implemented. It is simply not reasonable to assume that a trial the trialists regularly instructed the chiropractors not to use the optimal treatments.

What does that mean for the interpretation of the existing trial evidence? It means, I think, that we should interpret it on face value. The clinical evidence for chiropractic treatment of most conditions fails to be convincingly positive. Chiropractors often counter that such negative findings fail to take into account that chiropractors use numerous different techniques. This argument is not valid because we must assume that in each trial the optimal techniques were administered.

In other words, the chiropractic attempt to have the cake and eat it has failed.

An international team of researchers wanted to determine the efficacy of laser and needle acupuncture for chronic knee pain. They conducted a Zelen-design clinical trial (randomization occurred before informed consent), in Victoria, Australia (February 2010-December 2012). Community volunteers (282 patients aged ≥50 years with chronic knee pain) were treated by family physician acupuncturists.

The treatments consisted of A) no acupuncture (control group, n = 71), B) needle (n = 70), C) laser (n = 71), and D) sham laser (n = 70) acupuncture. Treatments were delivered for 12 weeks. Participants and acupuncturists were blinded to laser and sham laser acupuncture. Control participants were unaware of the trial.

Primary outcomes were average knee pain (numeric rating scale, 0 [no pain] to 10 [worst pain possible]; minimal clinically important difference [MCID], 1.8 units) and physical function (Western Ontario and McMaster Universities Osteoarthritis Index, 0 [no difficulty] to 68 [extreme difficulty]; MCID, 6 units) at 12 weeks. Secondary outcomes included other pain and function measures, quality of life, global change, and 1-year follow-up. Analyses were by intention-to-treat using multiple imputation for missing outcome data.

At 12 weeks and 1 year, 26 (9%) and 50 (18%) participants were lost to follow-up, respectively. Analyses showed neither needle nor laser acupuncture significantly improved pain (mean difference; -0.4 units; 95% CI, -1.2 to 0.4, and -0.1; 95% CI, -0.9 to 0.7, respectively) or function (-1.7; 95% CI, -6.1 to 2.6, and 0.5; 95% CI, -3.4 to 4.4, respectively) compared with sham at 12 weeks. Compared with control, needle and laser acupuncture resulted in modest improvements in pain (-1.1; 95% CI, -1.8 to -0.4, and -0.8; 95% CI, -1.5 to -0.1, respectively) at 12 weeks, but not at 1 year. Needle acupuncture resulted in modest improvement in function compared with control at 12 weeks (-3.9; 95% CI, -7.7 to -0.2) but was not significantly different from sham (-1.7; 95% CI, -6.1 to 2.6) and was not maintained at 1 year. There were no differences for most secondary outcomes and no serious adverse events.

The authors drew the following conclusions: In patients older than 50 years with moderate or severe chronic knee pain, neither laser nor needle acupuncture conferred benefit over sham for pain or function. Our findings do not support acupuncture for these patients.

This is one of the methodologically best acupuncture studies that I have seen so far.

  • its protocol has been published when the trial started thus allowing maximum transparency
  • it is adequately powered
  • it has a very clever study-design
  • it minimizes bias in all sorts of ways
  • it tests acupuncture for a condition that it is widely used for
  • it even manages to blind acupuncturists by using one treatment arm with laser acupuncture

The results show quite clearly that acupuncture does have mild effects on pain and function that entirely rely on a placebo response.

Will acupuncturists learn from this study and henceforward stop treating knee-patients? Somehow I doubt it! The much more likely scenario is that they will claim the trial was, for this or that reason, not valid. Acupuncture, like most of alternative medicine, seems unable to revise its dogma.

On this blog, I have often pointed out how dismally poor most of the trials of alternative therapies frequently are, particularly those in the realm of chiropractic. A brand-new study seems to prove my point.

The aim of this trial was to determine whether spinal manipulative therapy (SMT) plus home exercise and advice (HEA) compared with HEA alone reduces leg pain in the short and long term in adults with sub-acute and chronic back-related leg-pain (BRLP).

Patients aged 21 years or older with BRLP for least 4 weeks were randomised to receive 12 weeks of SMT plus HEA or HEA alone. Eleven chiropractors with a minimum of 5 years of practice experience delivered SMT in the SMT plus HEA group. The primary outcome was subjective BRLP at 12 and 52 weeks. Secondary outcomes were self-reported low back pain, disability, global improvement, satisfaction, medication use, and general health status at 12 and 52 weeks.

Of the 192 enrolled patients, 191 (99%) provided follow-up data at 12 weeks and 179 (93%) at 52 weeks. For leg pain, SMT plus HEA had a clinically important advantage over HEA (difference, 10 percentage points [95% CI, 2 to 19]; P = 0.008) at 12 weeks but not at 52 weeks (difference, 7 percentage points [CI, -2 to 15]; P = 0.146). Nearly all secondary outcomes improved more with SMT plus HEA at 12 weeks, but only global improvement, satisfaction, and medication use had sustained improvements at 52 weeks. No serious treatment-related adverse events or deaths occurred.

The authors conclude that, for patients with BRLP, SMT plus HEA was more effective than HEA alone after 12 weeks, but the benefit was sustained only for some secondary outcomes at 52 weeks.

This is yet another pragmatic trial following the notorious and increasingly popular A+B versus B design. As pointed out repeatedly on this blog, this study design can hardly ever generate a negative result (A+B is always more than B, unless A has a negative value [which even placebos don’t have]). Thus it is not a true test of the experimental treatment but all an exercise to create a positive finding for a potentially useless treatment. Had the investigators used any mildly pleasant placebo with SMT, the result would have been the same. In this way, they could create results showing that getting a £10 cheque or meeting with pleasant company every other day, together with HEA, is more effective than HEA alone. The conclusion that the SMT, the cheque or the company have specific effects is as implicit in this article as it is potentially wrong.

The authors claim that their study was limited because patient-blinding was not possible. This is not entirely true, I think; it was limited mostly because it failed to point out that the observed outcomes could be and most likely are due to a whole range of factors which are not directly related to SMT and, most crucially, because its write-up, particularly the conclusions, wrongly implied cause and effect between SMT and the outcome. A more accurate conclusion could have been as follows: SMT plus HEA was more effective than HEA alone after 12 weeks, but the benefit was sustained only for some secondary outcomes at 52 weeks. Because the trial design did not control for non-specific effects, the observed outcomes are consistent with SMT being an impressive placebo.

No such critical thought can be found in the article; on the contrary, the authors claim in their discussion section that the current trial adds to the much-needed evidence base about SMT for subacute and chronic BRLP. Such phraseology is designed to mislead decision makers and get SMT accepted as a treatment of conditions for which it is not necessarily useful.

Research where the result is known before the study has even started (studies with a A+B versus B design) is not just useless, it is, in my view, unethical: it fails to answer a real question and is merely a waste of resources as well as an abuse of patients willingness to participate in clinical trials. But the authors of this new trial are in good and numerous company: in the realm of alternative medicine, such pseudo-research is currently being published almost on a daily basis. What is relatively new, however, that even some of the top journals are beginning to fall victim to this incessant stream of nonsense.

Twenty years ago, when I started my Exeter job as a full-time researcher of complementary/alternative medicine (CAM), I defined the aim of my unit as applying science to CAM. At the time, this intention upset quite a few CAM-enthusiasts. One of the most prevalent arguments of CAM-proponents against my plan was that the study of CAM with rigorous science was quite simply an impossibility. They claimed that CAM included mind and body practices, holistic therapies, and other complex interventions which cannot not be put into the ‘straight jacket’ of conventional research, e. g. a controlled clinical trial. I spent the next few years showing that this notion was wrong. Gradually and hesitantly CAM researchers seemed to agree with my view – not all, of course, but first a few and then slowly, often reluctantly the majority of them.

What followed was a period during which several research groups started conducting rigorous tests of the hypotheses underlying CAM. All too often, the results turned out to be disappointing, to say the least: not only did most of the therapies in question fail to show efficacy, they were also by no means free of risks. Worst of all, perhaps, much of CAM was disclosed as being biologically implausible. The realization that rigorous scientific scrutiny often generated findings which were not what proponents had hoped for led to a sharp decline in the willingness of CAM-proponents to conduct rigorous tests of their hypotheses. Consequently, many asked whether science was such a good idea after all.

But that, in turn, created a new problem: once they had (at least nominally) committed themselves to science, how could they turn against it? The answer to this dilemma was easier that anticipated: the solution was to appear dedicated to science but, at the same time, to argue that, because CAM is subtle, holistic, complex etc., a different scientific approach was required. At this stage, I felt we had gone ‘full circle’ and had essentially arrived back where we were 20 years ago – except that CAM-proponents no longer rejected the scientific method outright but merely demanded different tools.

A recent article may serve as an example of this new and revised stance of CAM-proponents on science. Here proponents of alternative medicine argue that a challenge for research methodology in CAM/ICH* is the growing recognition that CAM/IHC practice often involves complex combination of novel interventions that include mind and body practices, holistic therapies, and others. Critics argue that the reductionist placebo controlled randomized control trial (RCT) model that works effectively for determining efficacy for most pharmaceutical or placebo trial RCTs may not be the most appropriate for determining effectiveness in clinical practice for either CAM/IHC or many of the interventions used in primary care, including health promotion practices. Therefore the reductionist methodology inherent in efficacy studies, and in particular in RCTs, may not be appropriate to study the outcomes for much of CAM/IHC, such as Traditional Korean Medicine (TKM) or other complex non-CAM/IHC interventions—especially those addressing comorbidities. In fact it can be argued that reductionist methodology may disrupt the very phenomenon, the whole system, that the research is attempting to capture and evaluate (i.e., the whole system in its naturalistic environment). Key issues that surround selection of the most appropriate methodology to evaluate complex interventions are well described in the Kings Fund report on IHC and also in the UK Medical Research Council (MRC) guidelines for evaluating complex interventions—guidelines which have been largely applied to the complexity of conventional primary care and care for patients with substantial comorbidity. These reports offer several potential solutions to the challenges inherent in studying CAM/IHC. [* IHC = integrated health care]

Let’s be clear and disclose what all of this actually means. The sequence of events, as I see it, can be summarized as follows:

  • We are foremost ALTERNATIVE! Our treatments are far too unique to be subjected to reductionist research; we therefore reject science and insist on an ALTERNATIVE.
  • We (well, some of us) have reconsidered our opposition and are prepared to test our hypotheses scientifically (NOT LEAST BECAUSE WE NEED THE RECOGNITION THAT THIS MIGHT BRING).
  • We are dismayed to see that the results are mostly negative; science, it turns out, works against our interests.
  • We need to reconsider our position.
  • We find it inconceivable that our treatments do not work; all the negative scientific results must therefore be wrong.
  • We always said that our treatments are unique; now we realize that they are far too holistic and complex to be submitted to reductionist scientific methods.
  • We still believe in science (or at least want people to believe that we do) – but we need a different type of science.
  • We insist that RCTs (and all other scientific methods that fail to demonstrate the value of CAM) are not adequate tools for testing complex interventions such as CAM.
  • We have determined that reductionist research methods disturb our subtle treatments.
  • We need pragmatic trials and similarly ‘soft’ methods that capture ‘real life’ situations, do justice to CAM and rarely produce a negative result.

What all of this really means is that, whenever the findings of research fail to disappoint CAM-proponents, the results are by definition false-negative. The obvious solution to this problem is to employ different (weaker) research methods, preferably those that cannot possibly generate a negative finding. Or, to put it bluntly: in CAM, science is acceptable only as long as it produces the desired results.

There must be well over 10 000 clinical trials of acupuncture; Medline lists ~5 000, and many more are hidden in the non-Medline listed literature. That should be good news! Sadly, it isn’t.

It should mean that we now have a pretty good idea for what conditions acupuncture is effective and for which illnesses it does not work. But we don’t! Sceptics say it works for nothing, while acupuncturists claim it is a panacea. The main reason for this continued controversy is that the quality of the vast majority of these 10 000 studies is not just poor, it is lousy.

“Where is the evidence for this outraging statement???” – I hear the acupuncture-enthusiasts shout. Well, how about my own experience as editor-in-chief of FACT? No? Far too anecdotal?

How about looking at Cochrane reviews then; they are considered to be the most independent and reliable evidence in existence? There are many such reviews (most, if not all [co-]authored by acupuncturists) and they all agree that the scientific rigor of the primary studies is fairly awful. Here are the crucial bits of just the last three; feel free to look for more:

All of the studies had a high risk of bias

All included trials had a high risk of bias…

The studies were not judged to be free from bias…

Or how about providing an example? Good idea! Here is a new trial which could stand for numerous others:

This study was performed to compare the efficacy of acupuncture versus corticosteroid injection for the treatment of Quervain’s tendosynovitis (no, you do not need to look up what condition this is for understanding this post). Thirty patients were treated in two groups. The acupuncture group received 5 acupuncture sessions of 30 minutes duration. The injection group received one methylprednisolone acetate injection in the first dorsal compartment of the wrist. The degree of disability and pain was evaluated by using the Quick Disabilities of the Arm, Shoulder, and Hand (Q-DASH) scale and the Visual Analogue Scale (VAS) at baseline and at 2 weeks and 6 weeks after the start of treatment. The baseline means of the Q-DASH and the VAS scores were 62.8 and 6.9, respectively. At the last follow-up, the mean Q-DASH scores were 9.8 versus 6.2 in the acupuncture and injection groups, respectively, and the mean VAS scores were 2 versus 1.2. Thus there were short-term improvements of pain and function in both groups.

The authors drew the following conclusions: Although the success rate was somewhat higher with corticosteroid injection, acupuncture can be considered as an alternative option for treatment of De Quervain’s tenosynovitis.

The flaws of this study are exemplary and numerous:

  • This should have been a study that compares two treatments – the technical term is ‘equivalence trial – and such studies need to be much larger to produce a meaningful result. Small sample sizes in equivalent trials will always make the two treatments look similarly effective, even if one is a pure placebo.
  • There is no gold standard treatment for this condition. This means that a comparative trial makes no sense at all. In such a situation, one ought to conduct a placebo-controlled trial.
  • There was no blinding of patients; therefore their expectation might have distorted the results.
  • The acupuncture group received more treatments than the injection group; therefore the additional attention might have distorted the findings.
  • Even if the results were entirely correct, one cannot conclude from them that acupuncture was effective; the notion that it was similarly ineffective as the injections is just as warranted.

These are just some of the most fatal flaws of this study. The sad thing is that similar criticisms can be made for most of the 10 000 trials of acupuncture. But the point here is not to nit-pick nor to quack-bust. My point is a different and more serious one: fatally flawed research is not just a ‘poor show’, it is unethical because it is a waste of scarce resources and, even more importantly, an abuse of patients for meaningless pseudo-science. All it does is it misleads the public into believing that acupuncture might be good for this or that condition and consequently make wrong therapeutic decisions.

In acupuncture (and indeed in most alternative medicine) research, the problem is so extremely wide-spread that it is high time to do something about it. Journal editors, peer-reviewers, ethics committees, universities, funding agencies and all others concerned with such research have to work together so that such flagrant abuse is stopped once and for all.

Recent Comments

Note that comments can now be edited for up to five minutes after they are first submitted.

Click here for a comprehensive list of recent comments.