research methodology

1 2 3 8

One of the problems regularly encountered when evaluating the effectiveness of chiropractic spinal manipulation is that there are numerous chiropractic spinal manipulative techniques and clinical trials rarely provide an exact means of differentiating between them. Faced with a negative studies, chiropractors might therefore argue that the result was negative because the wrong techniques were used; therefore they might insist that it does not reflect chiropractic in a wider sense. Others claim that even a substantial body of negative evidence does not apply to chiropractic as a whole because there is a multitude of techniques that have not yet been properly tested. It seems as though the chiropractic profession wants the cake and eat it.

Amongst the most commonly used is the ‘DIVERSIFIED TECHNIQUE’ (DT) which has been described as follows: Like many chiropractic and osteopathic manipulative techniques, Diversified is characterized by a high velocity low amplitude thrust. Diversified is considered the most generic chiropractic manipulative technique and is differentiated from other techniques in that its objective is to restore proper movement and alignment of spine and joint dysfunction.

Also widely used is a technique called ‘FLEXION DISTRACTION’ (FD) which involves the use of a specialized table that gently distracts or stretches the spine and which allows the chiropractor to isolate the area of disc involvement while slightly flexing the spine in a pumping rhythm.

The ‘ACTIVATOR TECHNIQUE’ (AT) seems a little less popular; it involves having the patient lie in a prone position and comparing the functional leg lengths. Often one leg will seem to be shorter than the other. The chiropractor then carries out a series of muscle tests such as having the patient move their arms in a certain position in order to activate the muscles attached to specific vertebrae. If the leg lengths are not the same, that is taken as a sign that the problem is located at that vertebra. The chiropractor treats problems found in this way moving progressively along the spine in the direction from the feet towards the head. The activator is a small handheld spring-loaded instrument which delivers a small impulse to the spine. It was found to give off no more than 0.3 J of kinetic energy in a 3-millisecond pulse. The aim is to produce enough force to move the vertebrae but not enough to cause injury.

There is limited research comparing the effectiveness of these and the many other techniques used by chiropractors, and the few studies that are available are usually less than rigorous and their findings are thus unreliable. A first step in researching this rather messy area would be to determine which techniques are most frequently employed.

The aim of this new investigation was to do just that, namely to provide insight into which treatment approaches are used most frequently by Australian chiropractors to treat spinal musculoskeletal conditions.

A questionnaire was sent online to the members of the two main Australian chiropractic associations in 2013. The participants were asked to provide information on treatment choices for specific spinal musculoskeletal conditions.

A total of 280 responses were received. DT was the first choice of treatment for most of the included conditions. DT was used significantly less in 4 conditions: cervical disc syndrome with radiculopathy and cervical central stenosis were more likely to be treated with AT. FD was used almost as much as DT in the treatment of lumbar disc syndrome with radiculopathy and lumbar central stenosis. More experienced Australian chiropractors use more AT and soft tissue therapy and less DT compared to their less experienced chiropractors. The majority of the responding chiropractors also used ancillary procedures such as soft tissue techniques and exercise prescription in the treatment of spinal musculoskeletal conditions.

The authors concluded that this survey provides information on commonly used treatment choices to the chiropractic profession. Treatment choices changed based on the region of disorder and whether neurological symptoms were present rather than with specific diagnoses. Diversified technique was the most commonly used spinal manipulative therapy, however, ancillary procedures such as soft tissue techniques and exercise prescription were also commonly utilised. This information may help direct future studies into the efficacy of chiropractic treatment for spinal musculoskeletal disorders.

I am a little less optimistic that this information will help to direct future research. Critical readers might have noticed that the above definitions of two commonly used techniques are rather vague, particularly that of DT.

Why is that so? The answer seems to be that even chiropractors are at a loss coming up with a good definition of their most-used therapeutic techniques. I looked hard for a more precise definition but the best I could find was this: Diversified is characterized by the manual delivery of a high velocity low amplitude thrust to restricted joints of the spine and the extremities. This is known as an adjustment and is performed by hand. Virtually all joints of the body can be adjusted to help restore proper range of motion and function. Initially a functional and manual assessment of each joint’s range and quality of motion will establish the location and degree of joint dysfunction. The patient will then be positioned depending on the region being adjusted when a specific, quick impulse will be delivered through the line of the joint in question. The direction, speed, depth and angles that are used are the product of years of experience, practice and a thorough understanding of spinal mechanics. Often a characteristic ‘crack’ or ‘pop’ may be heard during the process. This is perfectly normal and is nothing to worry about. It is also not a guide as to the value or effectiveness of the adjustment.

This means that the DT is not a single method but a hotchpotch of techniques; this assumption is also confirmed by the following quote: The diversified technique is a technique used by chiropractors that is composed of all other techniques. It is the most commonly used technique and primarily focuses on spinal adjustments to restore function to vertebral and spinal problems.

What does that mean for research into chiropractic spinal manipulation? It means, I think, that even if we manage to define that a study was to test the effectiveness of one named chiropractic technique, such as DT, the chiropractors doing the treatments would most likely do what they believe is required for each individual patient.

There is, of course, nothing wrong with that approach; it is used in many other area of health care as well. In such cases, we need to view the treatment as something like a ‘black box'; we test the effectiveness of the black box without attempting to define its exact contents, and we trust that the clinicians in the trial are well-trained to use the optimal mix of techniques as needed for each individual patient.

I would assume that, in most studies available to date, this is precisely what already has been implemented. It is simply not reasonable to assume that a trial the trialists regularly instructed the chiropractors not to use the optimal treatments.

What does that mean for the interpretation of the existing trial evidence? It means, I think, that we should interpret it on face value. The clinical evidence for chiropractic treatment of most conditions fails to be convincingly positive. Chiropractors often counter that such negative findings fail to take into account that chiropractors use numerous different techniques. This argument is not valid because we must assume that in each trial the optimal techniques were administered.

In other words, the chiropractic attempt to have the cake and eat it has failed.

A reader of this blog recently sent me the following message: “Looks like this group followed you recent post about how to perform a CAM RCT!” A link directed me to a new trial of ear-acupressure. Today is ‘national acupuncture and oriental medicine day’ in the US, a good occasion perhaps to have a critical look at it.

The aim of this study was to assess the effectiveness of ear acupressure and massage vs. control in the improvement of pain, anxiety and depression in persons diagnosed with dementia.

For this purpose, the researchers recruited a total of 120 elderly dementia patients institutionalized in residential homes. The participants were randomly allocated, to three groups:

  • Control group – they continued with their routine activities;
  • Ear acupressure intervention group – they received ear acupressure treatment (pressure was applied to acupressure points on the ear);
  • Massage therapy intervention group – they received relaxing massage therapy.

Pain, anxiety and depression were assessed with the Doloplus2, Cornell and Campbell scales. The study was carried out during 5 months; three months of experimental treatment and two months with no treatment. The assessments were done at baseline, each month during the treatment and at one and two months of follow-up.

A total of 111 participants completed the study. The ear acupressure intervention group showed better improvements than the two other groups in relation to pain and depression during the treatment period and at one month of follow-up. The best improvement in pain was achieved in the last (3rd) month of ear acupressure treatment. The best results regarding anxiety were also observed in the last month of treatment.

The authors concluded that ear acupressure and massage therapy showed better results than the control group in relation to pain, anxiety and depression. However, ear acupressure achieved more improvements.

The question is: IS THIS A RIGOROUS TRIAL?

My answer would be NO.

Now I better explain why, don’t I?

If we look at them critically, the results of this trial might merely prove that spending some time with a patient, being nice to her, administering a treatment that involves time and touch, etc. yields positive changes in subjective experiences of pain, anxiety and depression. Thus the results of this study might have nothing to do with the therapies per se.

And why would acupressure be more successful than massage therapy? Massage therapy is an ‘old hat’ for many patients; by contrast, acupressure is exotic and relates to mystical life forces etc. Features like that have the potential to maximise the placebo-response. Therefore it is conceivable that they have contributed to the superiority of acupressure over massage.

What I am saying is that the results of this trial can be interpreted in not just one but several ways. The main reason for that is the fact that the control group were not given an acceptable placebo, one that was indistinguishable from the real treatment. Patients were fully aware of what type of intervention they were getting. Therefore their expectations, possibly heightened by the therapists, determined the outcomes. Consequently there were factors at work which were totally beyond the control of the researchers and a clear causal link between the therapy and the outcome cannot be established.

An RCT that is aimed to test the effectiveness of a therapy but fails to establish such a causal link beyond reasonable doubt cannot be characterised as a rigorous study, I am afraid.

Sorry! Did I spoil your ‘national acupuncture and oriental medicine day’?

Whenever I give a public lecture about homeopathy, I explain what it is, briefly go in to its history, explain what its assumptions are, and what the evidence tells us about its efficacy and safety. When I am finished, there usually is a discussion with the audience. This is the part I like best; in fact, it is the main reason why I made the effort to do the lecture in the first place.

The questions vary, of course, but you can bet your last shirt that someone asks: “We know it works for animals; animals cannot experience a placebo-response, and therefore your claim that homeopathy relies on nothing but the placebo-effect must be wrong!” At this stage I often despair a little, I must admit. Not because the question is too daft, but because I did address it during my lecture. Thus I feel that I have failed to get the right message across – I despair with my obviously poor skills of giving an informative lecture!

Yet I need to answer the above question, of course. So I reiterate that the perceived effectiveness of homeopathy relies not just on the placebo-effect but also on phenomena such as regression towards the mean, natural history of the condition etc. I also usually mention that it is erroneous to assume that animals cannot benefit from placebo-effects; they can be conditioned, and pets can react to the expectations of their owners.

Finally, I need to mention the veterinary clinical evidence which – just like in the case of human patients – fails to show that homeopathic remedies are better than placebos for treating animals. Until recently, this was not an easy task because no systematic review of randomised placebo-controlled trials (RCTs) of veterinary homeopathy was available. Now, I am happy to announce, this situation has changed.

Using Cochrane methods, a brand-new review aimed to assess risk of bias and to quantify the effect size of homeopathic interventions compared with placebo for each eligible peer-reviewed trial. Judgement in 7 assessment domains enabled a trial’s risk of bias to be designated as low, unclear or high. A trial was judged to comprise reliable evidence, if its risk of bias was low or was unclear in specified domains. A trial was considered to be free of vested interest, if it was not funded by a homeopathic pharmacy.

The 18 RCTs found by the researchers were disparate in nature, representing 4 species and 11 different medical conditions. Reliable evidence, free from vested interest, was identified in only two trials:

  1. homeopathic Coli had a prophylactic effect on porcine diarrhoea (odds ratio 3.89, 95 per cent confidence interval [CI], 1.19 to 12.68, P=0.02);
  2. individualised homeopathic treatment did not have a more beneficial effect on bovine mastitis than placebo intervention (standardised mean difference -0.31, 95 per cent CI, -0.97 to 0.34, P=0.35).

The authors conclusions are clear: Mixed findings from the only two placebo-controlled RCTs that had suitably reliable evidence precluded generalisable conclusions about the efficacy of any particular homeopathic medicine or the impact of individualised homeopathic intervention on any given medical condition in animals.

My task when lecturing about homeopathy has thus become a great deal easier. But homeopathy-fans are not best pleased with this new article, I guess. They will try to claim that it was a biased piece of research conducted, most likely, by notorious anti-homeopaths who cannot be trusted. So who are the authors of this new publication?

They are RT Mathie from the British Homeopathic Association and J Clausen from one of Germany’s most pro-homeopathic institution, the ‘Karl und Veronica Carstens-Stiftung’.


‘Healing, hype or harm? A critical analysis of complementary or alternative medicine’ is the title of a book that I edited and that was published in 2008. Its publication date coincided with that of ‘Trick or Treatment?’ and therefore the former was almost completely over-shadowed by the latter. Consequently few people know about it. This is a shame, I think, and this post is dedicated to encouraging my readers to have a look at ‘Healing, hype or harm?’

One reviewer commented on Amazon about this book as follows: Vital and informative text that should be read by everyone alongside Ben Goldacre’s ‘Bad Science’ and Singh and Ernt’s ‘Trick or Treatment’. Everyone should be able to made informed choices about the treatments that are peddled to the desperate and gullible. As Tim Minchin famously said ‘What do you call Alternative Medicine that has been proved to work? . . . Medicine!’

This is high praise indeed! But I should not omit the fact that others have commented that they were appalled by our book and found it “disappointing and unsettling”. This does not surprise me in the least; after all, alternative medicine has always been a divisive subject.

The book was written by a total of 17 authors and covers many important aspects of alternative medicine. Some of its most famous contributors are Michael Baum, Gustav Born, David Colquhoun, James Randi and Nick Ross. Some of the most important subjects include:

  • Compassion
  • Quackademia
  • Impartiality
  • Ethics
  • Politics
  • Holism
  • Vitalism
  • Placebo

As already mentioned, our book is already 6 years old; however, this does not mean that it is now out-dated. The subject areas were chosen such that it will be timely for a long time to come. Nor does this book reflect one single point of view; as it was written by over a dozen different experts with vastly different backgrounds, it offers an entire spectrum of views and attitudes. It is, in a word, a book that stimulates critical thinking and thoughtful analysis.

I sincerely think you should have a look at it… and, in case you think I am hoping to maximise my income by telling you all this: all the revenues from this book go to charity.

One of the most commonly ‘accepted’ indications for acupuncture is anxiety. Many trials have suggested that it is effective for that condition. But is this really true? To find out, we need someone to conduct a systematic review or meta-analysis.

Korean researchers have just published such a paper; they wanted to assess the preoperative anxiolytic efficacy of acupuncture therapy and therefore conducted a meta-analysis of all RCTs on the subject. Four electronic databases were searched up to February 2014. Data were included in the meta-analysis from RCTs in which groups receiving preoperative acupuncture treatment were compared with control groups receiving a placebo for anxiety.

Fourteen publications with a total of 1,034 patients were included. Six RCTs, using the State-Trait Anxiety Inventory-State (STAI-S), reported that acupuncture interventions led to greater reductions in preoperative anxiety relative to sham acupuncture. A further eight publications, employing visual analogue scales, also indicated significant differences in preoperative anxiety amelioration between acupuncture and sham acupuncture.

The authors concluded that aacupuncture therapy aiming at reducing preoperative anxiety has a statistically significant effect relative to placebo or nontreatment conditions. Well-designed and rigorous studies that employ large sample sizes are necessary to corroborate this finding.

From these conclusions most casual readers might get the impression that acupuncture is indeed effective. One has to dig a bit deeper to realise that is perhaps not so.

Why? Because the quality of the primary studies was often dismally poor. Most did not even mention adverse effects which, in my view, is a clear breach of publication ethics. What is more, all the studies were wide open to bias. The authors of the meta-analysis include in their results section the following short paragraph:

The 14 included studies exhibited various degrees of bias susceptibility (Figure 2 and Figure 3). The agreement rate, as measured using Cohen’s kappa, was 0.8 [32]. Only six studies reported concealed allocation; the other six described a method of adequate randomization, although the word “randomization” appeared in all of the articles. Thirteen studies prevented blinding of the participants. Participants in these studies had no previous experience of acupuncture. According to STRICTA, two studies enquired after patients’ beliefs as a group: there were no significant differences [20, 24].

There is a saying amongst experts about such meta-analyses: RUBBISH IN, RUBBISH OUT. It describes the fact that several poor studies, pooled meta-analytically, can never give a reliable result.

This does, however, not mean that such meta-analyses are necessarily useless. If the authors prominently (in the abstract) stress that the quality of the primary studies was wanting and that therefore the overall result is unreliable, they might inspire future researchers to conduct more rigorous trials and thus generate progress. Most importantly, by insisting on pointing out these limitations and by not drawing positive conclusions from flawed data, they would avoid misleading those health care professionals – and let’s face it, they are the majority – who merely read the abstract or even just the conclusions of such articles.

The authors of this review have failed to do any of this; they and the journal EBCAM have thus done a disservice to us all by contributing to the constant drip of misleading and false-positive information about the value of acupuncture.

An international team of researchers wanted to determine the efficacy of laser and needle acupuncture for chronic knee pain. They conducted a Zelen-design clinical trial (randomization occurred before informed consent), in Victoria, Australia (February 2010-December 2012). Community volunteers (282 patients aged ≥50 years with chronic knee pain) were treated by family physician acupuncturists.

The treatments consisted of A) no acupuncture (control group, n = 71), B) needle (n = 70), C) laser (n = 71), and D) sham laser (n = 70) acupuncture. Treatments were delivered for 12 weeks. Participants and acupuncturists were blinded to laser and sham laser acupuncture. Control participants were unaware of the trial.

Primary outcomes were average knee pain (numeric rating scale, 0 [no pain] to 10 [worst pain possible]; minimal clinically important difference [MCID], 1.8 units) and physical function (Western Ontario and McMaster Universities Osteoarthritis Index, 0 [no difficulty] to 68 [extreme difficulty]; MCID, 6 units) at 12 weeks. Secondary outcomes included other pain and function measures, quality of life, global change, and 1-year follow-up. Analyses were by intention-to-treat using multiple imputation for missing outcome data.

At 12 weeks and 1 year, 26 (9%) and 50 (18%) participants were lost to follow-up, respectively. Analyses showed neither needle nor laser acupuncture significantly improved pain (mean difference; -0.4 units; 95% CI, -1.2 to 0.4, and -0.1; 95% CI, -0.9 to 0.7, respectively) or function (-1.7; 95% CI, -6.1 to 2.6, and 0.5; 95% CI, -3.4 to 4.4, respectively) compared with sham at 12 weeks. Compared with control, needle and laser acupuncture resulted in modest improvements in pain (-1.1; 95% CI, -1.8 to -0.4, and -0.8; 95% CI, -1.5 to -0.1, respectively) at 12 weeks, but not at 1 year. Needle acupuncture resulted in modest improvement in function compared with control at 12 weeks (-3.9; 95% CI, -7.7 to -0.2) but was not significantly different from sham (-1.7; 95% CI, -6.1 to 2.6) and was not maintained at 1 year. There were no differences for most secondary outcomes and no serious adverse events.

The authors drew the following conclusions: In patients older than 50 years with moderate or severe chronic knee pain, neither laser nor needle acupuncture conferred benefit over sham for pain or function. Our findings do not support acupuncture for these patients.

This is one of the methodologically best acupuncture studies that I have seen so far.

  • its protocol has been published when the trial started thus allowing maximum transparency
  • it is adequately powered
  • it has a very clever study-design
  • it minimizes bias in all sorts of ways
  • it tests acupuncture for a condition that it is widely used for
  • it even manages to blind acupuncturists by using one treatment arm with laser acupuncture

The results show quite clearly that acupuncture does have mild effects on pain and function that entirely rely on a placebo response.

Will acupuncturists learn from this study and henceforward stop treating knee-patients? Somehow I doubt it! The much more likely scenario is that they will claim the trial was, for this or that reason, not valid. Acupuncture, like most of alternative medicine, seems unable to revise its dogma.

Many proponents of alternative medicine seem somewhat suspicious of research; they have obviously understood that it might not produce the positive result they had hoped for; after all, good research tests hypotheses and does not necessarily confirm beliefs. At the same time, they are often tempted to conduct research: this is perceived as being good for the image and, provided the findings are positive, also good for business.

Therefore they seem to be tirelessly looking for a study design that cannot ‘fail’, i.e. one that avoids the risk of negative results but looks respectable enough to be accepted by ‘the establishment’. For these enthusiasts, I have good news: here is the study design that cannot fail.

It is perhaps best outlined as a concrete example; for reasons that will become clear very shortly, I have chosen reflexology as a treatment of diabetic neuropathy, but you can, of course, replace both the treatment and the condition as it suits your needs. Here is the outline:

  • recruit a group of patients suffering from diabetic neuropathy – say 58, that will do nicely,
  • randomly allocate them to two groups,
  • the experimental group receives regular treatments by a motivated reflexologist,
  • the controls get no such therapy,
  • both groups also receive conventional treatments for their neuropathy,
  • the follow-up is 6 months,
  • the following outcome measures are used: pain reduction, glycemic control, nerve conductivity, and thermal and vibration sensitivities,
  • the results show that the reflexology group experience more improvements in all outcome measures than those of control subjects,
  • your conclusion: This study exhibited the efficient utility of reflexology therapy integrated with conventional medicines in managing diabetic neuropathy.

Mission accomplished!

This method is fool-proof, trust me, I have seen it often enough being tested, and never has it generated disappointment. It cannot fail because it follows the notorious A+B versus B design (I know, I have mentioned this several times before on this blog, but it is really important, I think): both patient groups receive the essential mainstream treatment, and the experimental group receives a useless but pleasant alternative treatment in addition. The alternative treatment involves touch, time, compassion, empathy, expectations, etc. All of these elements will inevitably have positive effects, and they can even be used to increase the patients’ compliance with the conventional treatments that is being applied in parallel. Thus all outcome measures will be better in the experimental compared to the control group.

The overall effect is pure magic: even an utterly ineffective treatment will appear as being effective – the perfect method for producing false-positive results.

And now we hopefully all understand why this study design is so very popular in alternative medicine. It looks solid – after all, it’s an RCT!!! – and it thus convinces even mildly critical experts of the notion that the useless treatment is something worth while. Consequently the useless treatment will become accepted as ‘evidence-based’, will be used more widely and perhaps even reimbursed from the public purse. Business will be thriving!

And why did I employ reflexology for diabetic neuropathy? Is that example not a far-fetched? Not a bit! I used it because it describes precisely a study that has just been published. Of course, I could also have taken the chiropractic trial from my last post, or dozens of other studies following the A+B versus B design – it is so brilliantly suited for misleading us all.

On this blog, I have often pointed out how dismally poor most of the trials of alternative therapies frequently are, particularly those in the realm of chiropractic. A brand-new study seems to prove my point.

The aim of this trial was to determine whether spinal manipulative therapy (SMT) plus home exercise and advice (HEA) compared with HEA alone reduces leg pain in the short and long term in adults with sub-acute and chronic back-related leg-pain (BRLP).

Patients aged 21 years or older with BRLP for least 4 weeks were randomised to receive 12 weeks of SMT plus HEA or HEA alone. Eleven chiropractors with a minimum of 5 years of practice experience delivered SMT in the SMT plus HEA group. The primary outcome was subjective BRLP at 12 and 52 weeks. Secondary outcomes were self-reported low back pain, disability, global improvement, satisfaction, medication use, and general health status at 12 and 52 weeks.

Of the 192 enrolled patients, 191 (99%) provided follow-up data at 12 weeks and 179 (93%) at 52 weeks. For leg pain, SMT plus HEA had a clinically important advantage over HEA (difference, 10 percentage points [95% CI, 2 to 19]; P = 0.008) at 12 weeks but not at 52 weeks (difference, 7 percentage points [CI, -2 to 15]; P = 0.146). Nearly all secondary outcomes improved more with SMT plus HEA at 12 weeks, but only global improvement, satisfaction, and medication use had sustained improvements at 52 weeks. No serious treatment-related adverse events or deaths occurred.

The authors conclude that, for patients with BRLP, SMT plus HEA was more effective than HEA alone after 12 weeks, but the benefit was sustained only for some secondary outcomes at 52 weeks.

This is yet another pragmatic trial following the notorious and increasingly popular A+B versus B design. As pointed out repeatedly on this blog, this study design can hardly ever generate a negative result (A+B is always more than B, unless A has a negative value [which even placebos don't have]). Thus it is not a true test of the experimental treatment but all an exercise to create a positive finding for a potentially useless treatment. Had the investigators used any mildly pleasant placebo with SMT, the result would have been the same. In this way, they could create results showing that getting a £10 cheque or meeting with pleasant company every other day, together with HEA, is more effective than HEA alone. The conclusion that the SMT, the cheque or the company have specific effects is as implicit in this article as it is potentially wrong.

The authors claim that their study was limited because patient-blinding was not possible. This is not entirely true, I think; it was limited mostly because it failed to point out that the observed outcomes could be and most likely are due to a whole range of factors which are not directly related to SMT and, most crucially, because its write-up, particularly the conclusions, wrongly implied cause and effect between SMT and the outcome. A more accurate conclusion could have been as follows: SMT plus HEA was more effective than HEA alone after 12 weeks, but the benefit was sustained only for some secondary outcomes at 52 weeks. Because the trial design did not control for non-specific effects, the observed outcomes are consistent with SMT being an impressive placebo.

No such critical thought can be found in the article; on the contrary, the authors claim in their discussion section that the current trial adds to the much-needed evidence base about SMT for subacute and chronic BRLP. Such phraseology is designed to mislead decision makers and get SMT accepted as a treatment of conditions for which it is not necessarily useful.

Research where the result is known before the study has even started (studies with a A+B versus B design) is not just useless, it is, in my view, unethical: it fails to answer a real question and is merely a waste of resources as well as an abuse of patients willingness to participate in clinical trials. But the authors of this new trial are in good and numerous company: in the realm of alternative medicine, such pseudo-research is currently being published almost on a daily basis. What is relatively new, however, that even some of the top journals are beginning to fall victim to this incessant stream of nonsense.

Most of the underlying assumptions of alternative medicine (AM) lack plausibility. Whenever this is the case, so the argument put forward by an international team of researchers in a recent paper, there are difficulties involved in obtaining a valid statistical significance in clinical studies.

Using a mostly statistical approach, they argue that, since the prior probability of a research hypothesis is directly related to its scientific plausibility, the commonly used frequentist statistics, which do not account for this probability, are unsuitable for studies exploring matters in various degree disconnected from science. Any statistical significance obtained in this field should be considered with great caution and may be better applied to more plausible hypotheses (like placebo effect) than the specific efficacy of the intervention.

The researchers conclude that, since achieving meaningful statistical significance is an essential step in the validation of medical interventions, AM practices, producing only outcomes inherently resistant to statistical validation, appear not to belong to modern evidence-based medicine.

To emphasize their arguments, the researchers make the following additional points:

  • It is often forgotten that frequentist statistics, commonly used in clinical trials, provides only indirect evidence in support of the hypothesis examined.
  • The p-value inherently tends to exaggerate the support for the hypothesis tested, especially if the scientific plausibility of the hypothesis is low.
  • When the rationale for a clinical intervention is disconnected from the basic principles of science, as in case of complementary alternative medicines, any positive result obtained in clinical studies is more reasonably ascribable to hypotheses (generally to placebo effect) other than the hypothesis on trial, which commonly is the specific efficacy of the intervention.
  • Since meaningful statistical significance as a rule is an essential step to validation of a medical intervention, complementary alternative medicine cannot be considered evidence-based.

Further explanations can be found in the discussion of the article where the authors argue that the quality of the hypothesis tested should be consistent with sound logic and science and therefore have a reasonable prior probability of being correct. As a rule of thumb, assuming a “neutral” attitude towards the null hypothesis (odds = 1:1), a p-value of 0.01 or, better, 0.001 should suffice to give a satisfactory posterior probability of 0.035 and 0.005 respectively.

In the area of AM, hypotheses often are entirely inconsistent with logic and frequently fly in the face of science. Four examples can demonstrate this instantly and sufficiently, I think:

  • Homeopathic remedies which contain not a single ‘active’ molecule are not likely to generate biological effects.
  • Healing ‘energy’ of Reiki masters has no basis in science.
  • Meridians of acupuncture are pure imagination.
  • Chiropractic subluxation have never been shown to exist.

Positive results from clinical trials of implausible forms of AM are thus either due to chance, bias or must be attributed to more credible causes such as the placebo effect. Since the achievement of meaningful statistical significance is an essential step in the validation of medical interventions, unless some authentic scientific support to AM is provided, one has to conclude that AM cannot be considered as evidence-based.

Such arguments are by no means new; they have been voiced over and over again. Essentially, they amount to the old adage: IF YOU CLAIM THAT YOU HAVE A CAT IN YOUR GARDEN, A SIMPLE PICTURE MAY SUFFICE. IF YOU CLAIM THERE IS A UNICORN IN YOUR GARDEN, YOU NEED SOMETHING MORE CONVINCING. An extraordinary claim requires an extraordinary proof! Put into the context of the current discussion about AM, this means that the usual level of clinical evidence is likely to be very misleading as long as it totally neglects the biological plausibility of the prior hypothesis.

Proponents of AM do not like to hear such arguments. They usually insist on what we might call a ‘level playing field’ and fail to see why their assumptions require not only a higher level of evidence but also a reasonable scientific hypothesis. They forget that the playing field is not even to start with; to understand the situation better, they should read this excellent article. Perhaps its elegant statistical approach will convince them – but I would not hold my breath.

Twenty years ago, when I started my Exeter job as a full-time researcher of complementary/alternative medicine (CAM), I defined the aim of my unit as applying science to CAM. At the time, this intention upset quite a few CAM-enthusiasts. One of the most prevalent arguments of CAM-proponents against my plan was that the study of CAM with rigorous science was quite simply an impossibility. They claimed that CAM included mind and body practices, holistic therapies, and other complex interventions which cannot not be put into the ‘straight jacket’ of conventional research, e. g. a controlled clinical trial. I spent the next few years showing that this notion was wrong. Gradually and hesitantly CAM researchers seemed to agree with my view – not all, of course, but first a few and then slowly, often reluctantly the majority of them.

What followed was a period during which several research groups started conducting rigorous tests of the hypotheses underlying CAM. All too often, the results turned out to be disappointing, to say the least: not only did most of the therapies in question fail to show efficacy, they were also by no means free of risks. Worst of all, perhaps, much of CAM was disclosed as being biologically implausible. The realization that rigorous scientific scrutiny often generated findings which were not what proponents had hoped for led to a sharp decline in the willingness of CAM-proponents to conduct rigorous tests of their hypotheses. Consequently, many asked whether science was such a good idea after all.

But that, in turn, created a new problem: once they had (at least nominally) committed themselves to science, how could they turn against it? The answer to this dilemma was easier that anticipated: the solution was to appear dedicated to science but, at the same time, to argue that, because CAM is subtle, holistic, complex etc., a different scientific approach was required. At this stage, I felt we had gone ‘full circle’ and had essentially arrived back where we were 20 years ago – except that CAM-proponents no longer rejected the scientific method outright but merely demanded different tools.

A recent article may serve as an example of this new and revised stance of CAM-proponents on science. Here proponents of alternative medicine argue that a challenge for research methodology in CAM/ICH* is the growing recognition that CAM/IHC practice often involves complex combination of novel interventions that include mind and body practices, holistic therapies, and others. Critics argue that the reductionist placebo controlled randomized control trial (RCT) model that works effectively for determining efficacy for most pharmaceutical or placebo trial RCTs may not be the most appropriate for determining effectiveness in clinical practice for either CAM/IHC or many of the interventions used in primary care, including health promotion practices. Therefore the reductionist methodology inherent in efficacy studies, and in particular in RCTs, may not be appropriate to study the outcomes for much of CAM/IHC, such as Traditional Korean Medicine (TKM) or other complex non-CAM/IHC interventions—especially those addressing comorbidities. In fact it can be argued that reductionist methodology may disrupt the very phenomenon, the whole system, that the research is attempting to capture and evaluate (i.e., the whole system in its naturalistic environment). Key issues that surround selection of the most appropriate methodology to evaluate complex interventions are well described in the Kings Fund report on IHC and also in the UK Medical Research Council (MRC) guidelines for evaluating complex interventions—guidelines which have been largely applied to the complexity of conventional primary care and care for patients with substantial comorbidity. These reports offer several potential solutions to the challenges inherent in studying CAM/IHC. [* IHC = integrated health care]

Let’s be clear and disclose what all of this actually means. The sequence of events, as I see it, can be summarized as follows:

  • We are foremost ALTERNATIVE! Our treatments are far too unique to be subjected to reductionist research; we therefore reject science and insist on an ALTERNATIVE.
  • We (well, some of us) have reconsidered our opposition and are prepared to test our hypotheses scientifically (NOT LEAST BECAUSE WE NEED THE RECOGNITION THAT THIS MIGHT BRING).
  • We are dismayed to see that the results are mostly negative; science, it turns out, works against our interests.
  • We need to reconsider our position.
  • We find it inconceivable that our treatments do not work; all the negative scientific results must therefore be wrong.
  • We always said that our treatments are unique; now we realize that they are far too holistic and complex to be submitted to reductionist scientific methods.
  • We still believe in science (or at least want people to believe that we do) – but we need a different type of science.
  • We insist that RCTs (and all other scientific methods that fail to demonstrate the value of CAM) are not adequate tools for testing complex interventions such as CAM.
  • We have determined that reductionist research methods disturb our subtle treatments.
  • We need pragmatic trials and similarly ‘soft’ methods that capture ‘real life’ situations, do justice to CAM and rarely produce a negative result.

What all of this really means is that, whenever the findings of research fail to disappoint CAM-proponents, the results are by definition false-negative. The obvious solution to this problem is to employ different (weaker) research methods, preferably those that cannot possibly generate a negative finding. Or, to put it bluntly: in CAM, science is acceptable only as long as it produces the desired results.

1 2 3 8
Recent Comments
Click here for a comprehensive list of recent comments.