MD, PhD, FMedSci, FSB, FRCP, FRCPEd

scientific misconduct

Yesterday, BBC NEWS published the following interesting text about a BBC4 broadcast entitled ‘THE ROYAL ACTIVIST’ aired on the same day:

Prince Charles has been a well-known supporter of complementary medicine. According to a… former Labour cabinet minister, Peter Hain, it was a topic they shared an interest in.

“He had been constantly frustrated at his inability to persuade any health ministers anywhere that that was a good idea, and so he, as he once described it to me, found me unique from this point of view, in being somebody that actually agreed with him on this, and might want to deliver it.”
Mr Hain added: “When I was Secretary of State for Northern Ireland in 2005-7, he was delighted when I told him that since I was running the place I could more or less do what I wanted to do.***
“I was able to introduce a trial for complementary medicine on the NHS, and it had spectacularly good results, that people’s well-being and health was vastly improved.

“And when he learnt about this he was really enthusiastic and tried to persuade the Welsh government to do the same thing and the government in Whitehall to do the same thing for England, but not successfully,” added Mr Hain.

*** obviously there is no homeopathic remedy for megalomania (but that’s a different story)

Oh really?

A TRIAL?

SPECTACULARLY GOOD RESULTS?

NO KIDDING?

Let’s have a look at the ‘trial’ and its results. An easily accessible report provides the following details about it:

From February 2007 to February 2008, Get Well UK ran the UK’s first government-backed complementary therapy pilot. Sixteen practitioners provided treatments including acupuncture, osteopathy and aromatherapy, to more than 700 patients at two GP practices in Belfast and Derry.   

The BBC made an hour long documentary following our trials and tribulations, which was broadcast on BBC1 NI on 5 May 2008.

Following the successful completion of the pilot, the results were analysed by Social and Market Research and recommendations were made to the Health Minister

Aims and Objectives 

The aim of the project was to pilot services integrating complementary medicine into existing primary care services in Northern Ireland. Get Well UK provided this pilot project for the Department for Health, Social Services and Public Safety (DHSSPS) during 2007.

The objectives were:

  • To measure the health outcomes of the service and monitor health improvements.
  • To redress inequalities in access to complementary medicine by providing therapies through the NHS, allowing access regardless of income.
  • To contribute to best practise in the field of delivering complementary therapies through primary care.
  • To provide work for suitably skilled and qualified practitioners.
  • To increase patient satisfaction with quick access to expert care.
  • To help patients learn skills to improve and retain their health.
  • To free up GP time to work with other patients.
  • To deliver the programme for 700 patients.

Results 

The results of the pilot were analysed by Social and Market Research, who produced this report.

The findings can be summarised as follows: 

Following the pilot, 80% of patients reported an improvement in their symptoms, 64% took less time off work and 55% reduced their use of painkillers.

In the pilot, 713 patients with a range of ages and demographic backgrounds and either physical or mental health conditions were referred to various complementary and alternative medicine (CAM) therapies via nine GP practices in Belfast and Londonderry. Patients assessed their own health and wellbeing pre and post therapy and GPs and CAM practitioners also rated patients’ responses to treatment and the overall effectiveness of the scheme.

Health improvement
• 81% of patients reported an improvement in their physical health
• 79% reported an improvement in their mental health
• 84% of patients linked an improvement in their health and wellbeing directly to their CAM treatment
• In 65% of patient cases, GPs documented a health improvement, correlating closely to patient-reported improvements
• 94% of patients said they would recommend CAM to another patient with their condition
• 87% of patient indicated a desire to continue with their CAM treatment

Painkillers and medication
• Half of GPs reported prescribing less medication and all reported that patients had indicated to them that they needed less
• 62% of patients reported suffering from less pain
• 55% reported using less painkillers following treatment
• Patients using medication reduced from 75% before treatment to 61% after treatment
• 44% of those taking medication before treatment had reduced their use afterwards

Health service and social benefits
• 24% of patients who used health services prior to treatment (i.e. primary and secondary care, accident and emergency) reported using the services less after treatment
• 65% of GPs reported seeing the patient less following the CAM referral
• Half of GPs said the scheme had reduced their workload and 17% reported a financial saving for their practice
• Half of GPs said their patients were using secondary care services less.

Impressed? Well, in case you are, please consider this:

  • there was no control group
  • therefore it is not possible to attribute any of the outcomes to the alternative therapies offered
  • they could have been due to placebo-effects
  • or to the natural history of the disease
  • or to regression towards the mean
  • or to social desirability
  • or to many other factors which are unrelated to the alternative treatments provided
  • most outcome measures were not objectively verified
  • the patients were self-selected
  • they would all have had conventional treatments in parallel
  • this ‘trial’ was of such poor quality that its findings were never published in a peer-reviewed journal
  • this was not a ‘trial’ but a ‘pilot study’
  • pilot studies are not normally for measuring outcomes but for testing the feasibility of a proper trial
  • the research expertise of the investigators was close to zero
  • the scientific community merely had pitiful smiles for this ‘trial’ when it was published
  • neither Northern Ireland nor any other region implemented the programme despite its “spectacularly good results”.

So, is the whole ‘trial’ story an utterly irrelevant old hat?

Certainly not! Its true significance does not lie in the fact that a few amateurs are trying to push bogus treatments into the NHS via the flimsiest pseudo-research of the century. The true significance, I think, is that it shows how Prince Charles, once again, oversteps the boundaries of his constitutional role.

Dr. Oz, famous through his TV show promoting all types of quackery, recently testified before a US Senate subcommittee hearing on protecting consumers from false and deceptive advertising of weight loss products. This event turned out to be less than flattering for Dr Oz. One journalist commented that he “might as well be a cowardly lion — sent home with his tail between his legs after being accused at a congressional hearing of lying on his show about weight-loss claims.”

“I don’t get why you need to say this stuff, because you know it’s not true,” said Senator Claire McCaskill, who led the commerce subcommittee hearing. “The scientific community is almost monolithically against you in terms of the efficacy of the products you called ‘miracles,’ ” the Democratic senator from Missouri told Oz. “It’s a major problem when people are spending more and more money and they’re gaining more and more weight,” said Senator Amy Klobuchar.“Either you don’t talk about these things at all, or you’re going to have to be more specific because right now . . . this is not working.”

A source close to Dr Oz said he was perplexed: “We were invited down to Washington to testify at a hearing about scams and instead it became all about how much we hate your show.” Oz himself testified that he “heard the message…I do personally believe in the items that I talk about.”

“I intensively study them. I have given my family these products. . . . If you can lose a pound a week more than you would have lost by using them, it jump-starts you and gets you going. I think it makes sense.” “I’m surprised you’re defending this,” McCaskill replied. “It’s something that gives people false hope. I don’t see why you need to go there.”

Another journalist commented that the Senators repeatedly placed him on the defense over his weight loss products: “I know you know how much power you have. I know you know that. You are very powerful and [with] power comes a great deal of responsibility,” Senator Claire McCaskill , who led the Senate’s consumer protection hearing titled “Protecting Consumers from False and Deceptive Advertising of Weight-Loss Products…You are being made an example of today because of the power you have in this space…We didn’t call this hearing to beat up on you but we did call this hearing to talk about a real crisis in consumer protection. You can either be part of the police here or you can be part of the problem.”

Oz insisted he was no huckster but admitted the products promoted on his show don’t always have “the scientific muster” to present their benefits as “fact…I actually do personally believe in the items that I talk about in the show. I passionately studied them. I recognize that oftentimes they don’t have the scientific muster to present as fact but nevertheless I would give my audience the advice I give my family all the time. And I have given my family these products,” he said.

Dr Oz also said that some alternative treatments, such as prayer, cannot be tested scientifically. “I don’t think this ought to be a referendum on the use of alternative medical therapies. Because if that’s the case, listen, I’ve been criticized for having folks coming on my show talking about the power of prayer,” he said. “I can’t prove that prayer helps people survive an illness.”

No, Dr Oz! I know you are mistaken! I have done the research – both on alternative slimming aids and on spiritual healing. The results quite clearly show that these methods are not more effective than a placebo.

Auricular acupuncture (AA), according to the ‘COLLEGE OF AURICULAR ACUPUNCTURE’, has its origins in Modern Europe. In 1957 Dr. Paul Nogier, a neurologist from Lyons in France, observed a locum doctor treating sciatica by cauterizing an area of the ear. This prompted extensive research culminating in the development of the somatopic correspondence of specific parts of the body to the ear based upon the concept of an inverted foetus. Dr. Nogier believed that pain and other symptoms in the body could be alleviated by needling, massaging or electronically stimulating the corresponding region of the ear. Auricular Acupuncture is a specialized complementary therapy where acupuncture points on the outer ear are treated, using either needles or acupunctoscopes (electrical location and stimulation machines) to help relieve many chronic complaints. There are over 200 acupuncture points on the ear, each point named after an area of our anatomy. The outer ear acts like a switchboard to the brain. Each acupuncture point being treated, triggers electrical impulses from the ear via the brain, to the specific part of the body being treated.

Sounds odd? Well, that’s because it is odd!

But just because something is odd does not mean it is ineffective – so, what does the reliable evidence tell us? Here are some conclusions from systematic reviews:

The evidence that auricular acupuncture reduces postoperative pain is promising but not compelling.

The evidence for the effectiveness of AA for the symptomatic treatment of insomnia is limited.

The benefit of ear-acupressure for symptomatic relief of allergic rhinitis is unknown…

All of these analyses point out that the quality of the studies is usually very poor, and stress that more and better research is required. It is therefore interesting to note that a new study has just been published. Perhaps it could settle the question about the effectiveness of AA?

The aim of this study was 1) to evaluate whether auricular acupuncture effective for reducing health care provider stress and anxiety and 2) to determine, if auricular acupuncture impacts provider capacity for developing caring relationships with patients. Pre-intervention and post-intervention surveys were evaluated to see, if auricular acupuncture was associated with changes in State-Trait Anxiety Inventory (STAI), Professional Quality of Life, and Caring Ability Inventory scores. The results indicate that, compared with baseline, participants had a significant reduction in state anxiety (STAI), trait anxiety (STAI), burnout, and secondary traumatic stress scores (Professional Quality of Life). Significant increases were noted in courage and patience, two dimensions of the Caring Ability Inventory.

From these findings, the authors conclude that auricular acupuncture is an effective intervention for the relief of stress/anxiety in providers and supports heightened capacity for caring.

Sounds odd again? Yes, because it is odd!

I would argue that a study of any controversial therapy that has already been tested repeatedly in poor quality trials must have sufficient scientific rigor to advance the field of inquiry. If it does not fulfil this criterion, it is quite simply not ethical. The new study does not even have a control group; we can therefore not begin to tell whether the observed outcomes were due to non-specific effects, the natural history of the condition or regression towards the mean (to mention but a few of the possible sources of bias). To conclude that AA is ‘an effective intervention’ is therefore utterly barmy.

All of this could be entirely trivial and inconsequential. I am afraid, however, that it is not. Alternative medicine is littered with such unethically flawed research conducted by naïve and clueless pseudo-scientists who arrive at outrageous conclusions. This relentless flow of false-positive findings misleads consumers, health care professionals, decision makers and everyone else to draw the wrong conclusions about bogus therapies. And, in the end, this sort of thing even does a grave disfavour to any branch of alternative medicine that might have some degree of respectability.

IT IS HIGH TIME THAT THIS NONSENSE STOPS! IT BORDERS ON SCIENTIFIC MISCONDUCT.

Today, there are several dozens of journals publishing articles on alternative medicine. ‘The Journal of Alternative and Complementary Medicine’ is one of the best known, and it has one of the highest impact factors of them all. The current issue holds a few ‘gems’ which might be worthy of a comment or two. Here I have selected three articles reporting clinical studies, and I reproduce their abstracts (almost) in full (in italics) and add my comments (for clarity in bold). All the articles are available electronically, and I have provided the links for those who want to investigate beyond the abstracts.

STUDY No 1

The first ‘pilot study‘ was aimed to demonstrate the potential of auricular acupuncture (AAT) for insomnia in maintenance haemodialysis (MHD) patients and to prepare for a future randomized controlled trial.

Eligible patients were enrolled into this descriptive pilot study and received AAT designed to manage insomnia for 4 weeks. Questionnaires that used the Pittsburgh sleep quality index (PSQI) were completed at baseline, after a 4-week intervention, and 1 month after completion of treatment. Sleep quality and other clinical characteristics, including sleeping pills taken, were statistically compared between different time points.

A total of 22 patients were selected as eligible participants and completed the treatment and questionnaires. The mean global PSQI score was significantly decreased after AAT intervention (p<0.05). Participants reported improved sleep quality (p<0.01), shorter sleep latency (p<0.05), less sleep disturbance (p<0.01), and less daytime dysfunction (p=0.01). They also exhibited less dependency on sleep medications, indicated by the reduction in weekly estazolam consumption from 6.98±4.44 pills to 4.23±2.66 pills (p<0.01). However, these improvements were not preserved 1 month after treatment.

Conclusions: In this single-center pilot study, complementary AAT for MHD patients with severe insomnia was feasible and well tolerated and showed encouraging results for sleep quality.

My comments:

In alternative medicine research, it has become far too common (almost generally accepted) to call a flimsy trial a ‘pilot study’. The authors give their game away by stating that, by conducting this trial, they want to ‘demonstrate the potential of AAT’. This is not a legitimate aim of research; science is for TESTING hypotheses, not for PROVING them!

The results of this trial show that patients experienced improvements after receiving AAT which, however, did not last. As there was no placebo control group, the most likely explanation for these outcomes would be that AAT generated a short-lasting placebo effect.

A sample size of 22 is, of course, far to small to allow any conclusions about the safety of the intervention. Despite these obvious facts, the authors seem convinced that AAT is both safe and effective.

STUDY No 2

The aim of the second study was to compare the therapeutic effect of Yamamoto new scalp acupuncture (YNSA), a recently developed microcupuncture system, with traditional acupuncture (TCA) for the prophylaxis and treatment of migraine headache.

In a randomized clinical trial, 80 patients with migraine headache were assigned to receive YNSA or TCA. A pain visual analogue scale (VAS) and migraine therapy assessment questionnaire (MTAQ) were completed before treatment, after 6 and 18 sections of treatment, and 1 month after completion of therapy.

All the recruited patients completed the study. Baseline characteristics were similar between the two groups. Frequency and severity of migraine attacks, nausea, the need for rescue treatment, and work absence rate decreased similarly in both groups. Recovery from headache and ability to continue daily activities 2 hours after medical treatment showed similar improvement in both groups (p>0.05).

Conclusions: Classic acupuncture and YNSA are similarly effective in the prophylaxis and treatment of migraine headache and may be considered as alternatives to pharmacotherapy.

My comments:

This is what is technically called an ‘equivalence trial’, i.e. a study that compares an experimental treatment (YNSA) to one that is (assumed to be) effective. To demonstrate equivalence, such trials need to have large sample sizes, and this study is woefully underpowered. As it stands, the results show nothing meaningful at all; if anything, they suggest that both interventions were similarly useless.

STUDY No 3

The third study determined whether injection with hypertonic dextrose and morrhuate sodium (prolotherapy) using a pragmatic, clinically determined injection schedule for knee osteoarthritis (KOA) results in improved knee pain, function, and stiffness compared to baseline status.

The participants were 38 adults who had at least 3 months of symptomatic KOA and who were in the control groups of a prior prolotherapy randomized controlled trial (RCT) (Prior-Control), were ineligible for the RCT (Prior-Ineligible), or were eligible but declined the RCT (Prior-Declined).

The injection sessions at occurred at 1, 5, and 9 weeks with as-needed treatment at weeks 13 and 17. Extra-articular injections of 15% dextrose and 5% morrhuate sodium were done at peri-articular tendon and ligament insertions. A single intra-articular injection of 6 mL 25% dextrose was performed through an inferomedial approach.

The primary outcome measure was the validated Western Ontario McMaster University Osteoarthritis Index (WOMAC). The secondary outcome measure was the Knee Pain Scale and postprocedure opioid medication use and participant satisfaction.

The Prior-Declined group reported the most severe baseline WOMAC score (p=0.02). Compared to baseline status, participants in the Prior-Control group reported a score change of 12.4±3.5 points (19.5%, p=0.002). Prior-Decline and Prior-Ineligible groups improved by 19.4±7.0 (42.9%, p=0.05) and 17.8±3.9 (28.4%, p=0.008) points, respectively; 55.6% of Prior-Control, 75% of Prior-Decline, and 50% of Prior-Ineligible participants reported score improvement in excess of the 12-point minimal clinical important difference on the WOMAC measure. Postprocedure opioid medication resulted in rapid diminution of prolotherapy injection pain. Satisfaction was high and there were no adverse events.

Conclusions: Prolotherapy using dextrose and morrhuate sodium injections for participants with mild-to-severe KOA resulted in safe, significant, sustained improvement of WOMAC-based knee pain, function, and stiffness scores compared to baseline status.

My Comments:

This study had nothing that one might call a proper control group: all the three groups mentioned were treated with the experimental treatment. No attempt was made to control for even the most obvious biases: the observed effects could have been due to placebo or any other non-specific effects. The authors conclusions imply a causal relationship between the treatment and the outcome which is wrong. The notion that the experimental treatment is ‘safe’ is based on just 38 patients and therefore not reasonable.

IMPLICATION

All of this might seem rather trivial, and my comments could be viewed as a deliberate and vicious attempt to discredit one of the most respected journals of alternative medicine. Yet, considering that articles of this nature are more the rule than the exception in alternative medicine, I do think that this flagrant lack of scientific rigour is a relevant issue and has important implications.

As long as research in this area continues to be deeply flawed, as long as reviewers turn a blind eye to (or are not smart enough to detect) even the most obvious mistakes, as long as journal editors accept any rubbish in order to fill their pages, there is a great danger that we are being continuously being mislead about the supposed therapeutic value of alternative therapies.

Many who read this blog will, of course, have the capacity to think critically and might therefore not fall into the trap of accepting the conclusions of fatally flawed research. But many other people, including politicians, journalists and consumers, might not have the necessary appraisal skills and will thus not be able to tell that such studies can serve only one purpose: to popularise bogus treatments and thereby render health care less effective and more dangerous. Enthusiasts of alternative medicine are usually fully convinced that such studies amount to evidence and ram this pseudo-information down the throat of health care decision makers – the effects of such lobbying on public health can be disastrous.

And there is another downside to the publication of such dismal drivel: assuming (as I do) that not all of alternative medicine is completely useless, such embarrassingly poor research will inevitably have detrimental effects on the discipline of alternative medicine. After being exposed to a seemingly endless stream of pseudo-research, critics will eventually give up taking any of it seriously and might claim that none of it is worth the bother. In other words, those who conduct, accept or publish such nonsensical papers are not only endangering medical progress in general, they are also harming the very cause they try so desperately hard to advance.

A recent meta-analysis evaluated the efficacy of acupuncture for treatment of irritable bowel syndrome (IBS) and arrived at bizarrely positive conclusions.

The authors state that they searched 4 electronic databases for double-blind, placebo-controlled trials investigating the efficacy of acupuncture in the management of IBS. Studies were screened for inclusion based on randomization, controls, and measurable outcomes reported.

Six RCTs were included in the meta-analysis, and 5 articles were of high quality.  The pooled relative risk for clinical improvement with acupuncture was 1.75 (95%CI: 1.24-2.46, P = 0.001). Using two different statistical approaches, the authors confirmed the efficacy of acupuncture for treating IBS and concluded that acupuncture exhibits clinically and statistically significant control of IBS symptoms.

As IBS is a common and often difficult to treat condition, this would be great news! But is it true? We do not need to look far to find the embarrassing mistakes and – dare I say it? – lies on which this result was constructed.

The largest RCT included in this meta-analysis was neither placebo-controlled nor double blind; it was a pragmatic trial with the infamous ‘A+B versus B’ design. Here is the key part of its methods section: 116 patients were offered 10 weekly individualised acupuncture sessions plus usual care, 117 patients continued with usual care alone. Intriguingly, this was the ONLY one of the 6 RCTs with a significantly positive result!

The second largest study (as well as all the other trials) showed that acupuncture was no better than sham treatments. Here is the key quote from this trial: there was no statistically significant difference between acupuncture and sham acupuncture.

So, let me re-write the conclusions of this meta-analysis without spin, lies or hype: These results of this meta-analysis seem to indicate that:

  1. currently there are several RCTs testing whether acupuncture is an effective therapy for IBS,
  2. all the RCTs that adequately control for placebo-effects show no effectiveness of acupuncture,
  3. the only RCT that yields a positive result does not make any attempt to control for placebo-effects,
  4. this suggests that acupuncture is a placebo,
  5. it also demonstrates how misleading studies with the infamous ‘A+B versus B’ design can be,
  6. finally, this meta-analysis seems to be a prime example of scientific misconduct with the aim of creating a positive result out of data which are, in fact, negative.

It is almost 10 years ago that Prof Kathy Sykes’ BBC series entitled ALTERNATIVE MEDICINE was aired. I had been hired by the BBC as their advisor for the programme and had tried my best to iron out the many mistakes that were about to be broadcast. But the scope for corrections turned out to be narrow and, at one stage, the errors seemed too serious and too far beyond repair to continue with my task. I had thus offered my resignation from this post. Fortunately this move led to some of my concerns being addressed after all, and they convinced me to remain in post.

The first part of the series was on acupuncture, and Kathy presented the opening scene of a young women undergoing open heart surgery with the aid of acupuncture. All the BBC had ever shown me and asked me to advise on was the text – I had never seen the images. Kathy’s text included the statement that the patient was having the surgery “with only needles to control the pain.”  I had not objected to this statement in the firm belief that the images of the film would back up this extraordinary claim. As it turned out, it did not; the patient clearly had all sorts of other treatments given through intra-venous lines and, in the film, these were openly in the view of Kathy Sykes.

This overt contradiction annoyed not just me but several other people as well. One of them was Simon Singh who filed an official complaint against the BBC for misleading the public, and eventually won his case.

The notion that acupuncture can serve as an alternative to anaesthesia or other surgical conditions crops up with amazing regularity. It is important not least because is often used as a promotional tool with the implication that, IF ACUPUNCTURE CAN ACHIVE SUCH DRAMATIC EFFECTS, IT MUST BE AN INCREDIBLY USEFUL TREATMENT! It is therefore relevant to ask what the scientific evidence tells us about this issue.

This was the question we wanted to address in a recent publication. Specifically, our aim was to summarise recent systematic reviews of acupuncture for surgical conditions.

Thirteen electronic databases were searched for relevant reviews published since 2000. Data were extracted by two independent reviewers according to predefined criteria. Twelve systematic reviews met our inclusion criteria. They related to the prevention or treatment of post-operative nausea and vomiting as well as to surgical or post-operative pain. The reviews drew conclusions which were far from uniform; specifically for surgical pain the evidence was not convincing. We concluded that “the evidence is insufficient to suggest that acupuncture is an effective intervention in surgical settings.”

So, Kathy Sykes’ comment was misguided in more than just one way: firstly, the scene she described in the film did not support what she was saying; secondly, the scientific evidence fails to support the notion that acupuncture can be used as an alternative to analgesia during surgery.

This story has several positive outcomes all the same. After seeing the BBC programme, Simon Singh contacted me to learn my views on the matter. This prompted me to support his complaint against the BBC and helped him to win this case. Furthermore, it led to a co-operation and friendship which produced our book TRICK OR TREATMENT.

Systematic reviews are widely considered to be the most reliable type of evidence for judging the effectiveness of therapeutic interventions. Such reviews should be focused on a well-defined research question and identify, critically appraise and synthesize the totality of the high quality research evidence relevant to that question. Often it is possible to pool the data from individual studies and thus create a new numerical result of the existing evidence; in this case, we speak of a meta-analysis, a sub-category of systematic reviews.

One strength of systematic review is that they minimise selection and random biases by considering at the totality of the evidence of a pre-defined nature and quality. A crucial precondition, however, is that the quality of the primary studies is critically assessed. If this is done well, the researchers will usually be able to determine how robust any given result is, and whether high quality trials generate similar findings as those of lower quality. If there is a discrepancy between findings from rigorous and flimsy studies, it is obviously advisable to trust the former and discard the latter.

And this is where systematic reviews of alternative treatments can run into difficulties. For any given research question in this area we usually have a paucity of primary studies. Equally important is the fact that many of the available trials tend to be of low quality. Consequently, there often is a lack of high quality studies, and this makes it all the more important to include a robust critical evaluation of the primary data. Not doing so would render the overall result of the review less than reliable – in fact, such a paper would not qualify as a systematic review at all; it would be a pseudo-systematic review, i.e. a review which pretends to be systematic but, in fact, is not. Such papers are a menace in that they can seriously mislead us, particularly if we are not familiar with the essential requirements for a reliable review.

This is precisely where some promoters of bogus treatments seem to see their opportunity of making their unproven therapy look as though it was evidence-based. Pseudo-systematic reviews can be manipulated to yield a desired outcome. In my last post, I have shown that this can be done by including treatments which are effective so that an ineffective therapy appears effective (“chiropractic is so much more than just spinal manipulation”). An even simpler method is to exclude some of the studies that contradict one’s belief from the review. Obviously, the review would then not comprise the totality of the available evidence. But, unless the reader bothers to do a considerable amount of research, he/she would be highly unlikely to notice. All one needs to do is to smuggle the paper past the peer-review process – hardly a difficult task, given the plethora of alternative medicine journals that bend over backwards to publish any rubbish as long as it promotes alternative medicine.

Alternatively (or in addition) one can save oneself a lot of work and omit the process of critically evaluating the primary studies. This method is increasingly popular in alternative medicine. It is a fool-proof method of generating a false-positive overall result. As poor quality trials have a tendency to deliver false-positive results, it is obvious that a predominance of flimsy studies must create a false-positive result.

A particularly notorious example of a pseudo-systematic review that used this as well as most of the other tricks for misleading the reader is the famous ‘systematic’ review by Bronfort et al. It was commissioned by the UK GENERAL CHIROPRACTIC COUNCIL after the chiropractic profession got into trouble and was keen to defend those bogus treatments disclosed by Simon Singh. Bronfort and his colleagues thus swiftly published (of course, in a chiro-journal) an all-encompassing review attempting to show that, at least for some conditions, chiropractic was effective. Its lengthy conclusions seemed encouraging: Spinal manipulation/mobilization is effective in adults for: acute, subacute, and chronic low back pain; migraine and cervicogenic headache; cervicogenic dizziness; manipulation/mobilization is effective for several extremity joint conditions; and thoracic manipulation/mobilization is effective for acute/subacute neck pain. The evidence is inconclusive for cervical manipulation/mobilization alone for neck pain of any duration, and for manipulation/mobilization for mid back pain, sciatica, tension-type headache, coccydynia, temporomandibular joint disorders, fibromyalgia, premenstrual syndrome, and pneumonia in older adults. Spinal manipulation is not effective for asthma and dysmenorrhea when compared to sham manipulation, or for Stage 1 hypertension when added to an antihypertensive diet. In children, the evidence is inconclusive regarding the effectiveness for otitis media and enuresis, and it is not effective for infantile colic and asthma when compared to sham manipulation. Massage is effective in adults for chronic low back pain and chronic neck pain. The evidence is inconclusive for knee osteoarthritis, fibromyalgia, myofascial pain syndrome, migraine headache, and premenstrual syndrome. In children, the evidence is inconclusive for asthma and infantile colic. 

Chiropractors across the world cite this paper as evidence that chiropractic has at least some evidence base. What they omit to tell us (perhaps because they do not appreciate it themselves) is the fact that Bronfort et al

  • failed to formulate a focussed research question,
  • invented his own categories of inconclusive findings,
  • included all sorts of studies which had nothing to do with chiropractic,
  • and did not to make an assessment of the quality of the included primary studies they included in their review.

If, for a certain condition, three trials were included, for instance, two of which were positive but of poor quality and one was negative but of good quality, the authors would conclude that, overall, there is sound evidence.

Bronfort himself is, of course, more than likely to know all that (he has learnt his trade with an excellent Dutch research team and published several high quality reviews) – but his readers mostly don’t. And for chiropractors, this ‘systematic’ review is now considered to be the most reliable evidence in their field.

The efficacy or effectiveness of medical interventions is, of course, best tested in clinical trials. The principle of a clinical trial is fairly simple: typically, a group of patients is divided (preferably at random) into two subgroups, one (the ‘verum’ group) is treated with the experimental treatment and the other (the ‘control’ group) with another option (often a placebo), and the eventual outcomes of the two groups is compared. If done well, such studies are able to exclude biases and confounding factors such that their findings allow causal inference. In other words, they can tell us whether an outcome was caused by the intervention per se or by some other factor such as the natural history of the disease, regression towards the mean etc.

A clinical trial is a research tool for testing hypotheses; strictly speaking, it tests the ‘null-hypothesis’: “the experimental treatment generates the same outcomes as the treatment of the control group”. If the trial shows no difference between the outcomes of the two groups, the null-hypothesis is confirmed. In this case, we commonly speak of a negative result. If the experimental treatment was better than the control treatment, the null-hypothesis is rejected, and we commonly speak of a positive result. In other words, clinical trials can only generate positive or negative results, because the null-hypothesis must either be confirmed or rejected – there are no grey tones between the black of a negative and the white of a positive study.

For enthusiasts of alternative medicine, this can create a dilemma, particularly if there are lots of published studies with negative results. In this case, the totality of the available trial evidence is negative which means the treatment in question cannot be characterised as effective. It goes without saying that such an overall conclusion rubs the proponents of that therapy the wrong way. Consequently, they might look for ways to avoid this scenario.

One fairly obvious way of achieving this aim is to simply re-categorise the results. What, if we invented a new category? What, if we called some of the negative studies by a different name? What about NON-CONCLUSIVE?

That would be brilliant, wouldn’t it. We might end up with a simple statistic where the majority of the evidence is, after all, positive. And this, of course, would give the impression that the ineffective treatment in question is effective!

How exactly do we do this? We continue to call positive studies POSITIVE; we then call studies where the experimental treatment generated worst results than the control treatment (usually a placebo) NEGATIVE; and finally we call those studies where the experimental treatment created outcomes which were not different from placebo NON-CONCLUSIVE.

In the realm of alternative medicine, this ‘non-conclusive result’ method has recently become incredibly popular . Take homeopathy, for instance. The Faculty of Homeopathy proudly claim the following about clinical trials of homeopathy: Up to the end of 2011, there have been 164 peer-reviewed papers reporting randomised controlled trials (RCTs) in homeopathy. This represents research in 89 different medical conditions. Of those 164 RCT papers, 71 (43%) were positive, 9 (6%) negative and 80 (49%) non-conclusive.

This misleading nonsense was, of course, warmly received by homeopaths. The British Homeopathic Association, like many other organisations and individuals with an axe to grind lapped up the message and promptly repeated it: The body of evidence that exists shows that much more investigation is required – 43% of all the randomised controlled trials carried out have been positive, 6% negative and 49% inconclusive.

Let’s be clear what has happened here: the true percentage figures seem to show that 43% of studies (mostly of poor quality) suggest a positive result for homeopathy, while 57% of them (on average the ones of better quality) were negative. In other words, the majority of this evidence is negative. If we conducted a proper systematic review of this body of evidence, we would, of course, have to account for the quality of each study, and in this case we would have to conclude that homeopathy is not supported by sound evidence of effectiveness.

The little trick of applying the ‘NON-CONCLUSIVE’ method has thus turned this overall result upside down: black has become white! No wonder that it is so popular with proponents of all sorts of bogus treatments.

Whenever a new trial of an alternative intervention emerges which fails to confirm the wishful thinking of the proponents of that therapy, the world of alternative medicine is in turmoil. What can be done about yet another piece of unfavourable evidence? The easiest solution would be to ignore it, of course – and this is precisely what is often tried. But this tactic usually proves to be unsatisfactory; it does not neutralise the new evidence, and each time someone brings it up, one has to stick one’s head back into the sand. Rather than denying its existence, it would be preferable to have a tool which invalidates the study in question once and for all.

The ‘fatal flaw’ solution is simpler than anticipated! Alternative treatments are ‘very special’, and this notion must be emphasised, blown up beyond all proportions and used cleverly to discredit studies with unfavourable outcomes: the trick is simply to claim that studies with unfavourable results have a ‘fatal flaw’ in the way the alternative treatment was applied. As only the experts in the ‘very special’ treatment in question are able to judge the adequacy of their therapy, nobody is allowed to doubt their verdict.

Take acupuncture, for instance; it is an ancient ‘art’ which only the very best will ever master – at least that is what we are being told. So, all the proponents need to do in order to invalidate a trial, is read the methods section of the paper in full detail and state ‘ex cathedra’ that the way acupuncture was done in this particular study is completely ridiculous. The wrong points were stimulated, or the right points were stimulated but not long enough [or too long], or the needling was too deep [or too shallow], or the type of stimulus employed was not as recommended by TCM experts, or the contra-indications were not observed etc. etc.

As nobody can tell a correct acupuncture from an incorrect one, this ‘fatal flaw’ method is fairly fool-proof. It is also ever so simple: acupuncture-fans do not necessarily study hard to find the ‘fatal flaw’, they only have to look at the result of a study – if it was favourable, the treatment was obviously done perfectly by highly experienced experts; if it was unfavourable, the therapists clearly must have been morons who picked up their acupuncture skills in a single weekend course. The reasons for this judgement can always be found or, if all else fails, invented.

And the end-result of the ‘fatal flaw’ method is most satisfactory; what is more, it can be applied to all alternative therapies – homeopathy, herbal medicine, reflexology, Reiki healing, colonic irrigation…the method works for all of them! What is even more, the ‘fatal flaw’ method is adaptable to other aspects of scientific investigations such that it fits every conceivable circumstance.

An article documenting the ‘fatal flaw’ has to be published, of course – but this is no problem! There are dozens of dodgy alternative medicine journals which are only too keen to print even the most far-fetched nonsense as long as it promotes alternative medicine in some way. Once this paper is published, the proponents of the therapy in question have a comfortable default position to rely on each time someone cites the unfavourable study: “WHAT NOT THAT STUDY AGAIN! THE TREATMENT HAS BEEN SHOWN TO BE ALL WRONG. NOBODY CAN EXPECT GOOD RESULTS FROM A THERAPY THAT WAS NOT CORRECTLY ADMINISTERED. IF YOU DON’T HAVE BETTER STUDIES TO SUPPORT YOUR ARGUMENTS, YOU BETTER SHUT UP.”

There might, in fact, be better studies – but chances are that the ‘other side’ has already documented a ‘fatal flaw’ in them too.

Cancer patients are bombarded with information about supplements which allegedly are effective for their condition. I estimate that 99.99% of this information is unreliable and much of it is outright dangerous. So, there is an urgent need for trustworthy, objective information. But which source can we trust?

The authors of a recent article in ‘INTEGRATIVE CANCER THARAPIES’ (the first journal to spearhead and focus on a new and growing movement in cancer treatment. The journal emphasizes scientific understanding of alternative medicine and traditional medicine therapies, and their responsible integration with conventional health care. Integrative care includes therapeutic interventions in diet, lifestyle, exercise, stress care, and nutritional supplements, as well as experimental vaccines, chrono-chemotherapy, and other advanced treatments) review the issue of dietary supplements in the treatment of cancer patients. They claim that the optimal approach is to discuss both the facts and the uncertainty with the patient, in order to reach a mutually informed decision. This sounds promising, and we might thus trust them to deliver something reliable.

In order to enable doctors and other health care professionals to have such discussion, the authors then report on the work of the ‘Clinical Practice Committee’ of ‘The Society of Integrative Oncology’. This panel undertook the challenge of providing basic information to physicians who wish to discuss these issues with their patients. A list of supplements that have the best suggestions of benefit was constructed by leading researchers and clinicians who have experience in using these supplements:

  1. curcumin,
  2. glutamine,
  3. vitamin D,
  4. maitake mushrooms,
  5. fish oil,
  6. green tea,
  7. milk thistle,
  8. astragalus,
  9. melatonin,
  10. probiotics.

The authors claim that their review includes basic information on each supplement, such as evidence on effectiveness and clinical trials, adverse effects, and interactions with medications. The information was constructed to provide an up-to-date base of knowledge, so that physicians and other health care providers would be aware of the supplements and be able to discuss realistic expectations and potential benefits and risks (my emphasis).

At first glance, this task looks ambitious but laudable; however, after studying the paper in some detail, I must admit that I have considerable problems taking it seriously – and here is why.

The first question I ask myself when reading the abstract is: Who are these “leading researchers and clinicians”? Surely such a consensus exercise crucially depends on who is being consulted. The article itself does not reveal who these experts are, merely that they are all members of the ‘Society of Integrative Oncology’. A little research reveals this organisation to be devoted to integrating all sorts of alternative therapies into cancer care. If we assume that the experts are identical with the authors of the review; one should point out that most of them are proponents of alternative medicine. This lack of critical input seems more than a little disconcerting.

My next questions are: How did they identify the 10 supplements and how did they evaluate the evidence for or against them? The article informs us that a 5-step procedure was employed:

1. Each clinician in this project was requested to construct a list of supplements that they tend to use frequently in their practice.

2. An initial list of close to 25 supplements was constructed. This list included supplements that have suggestions of some possible benefit and likely to carry minimal risk in cancer care.

3. From that long list, the group agreed on the 10 leading supplements that have the best suggestions of benefit.

4. Each participant selected 1 to 2 supplements that they have interest and experience in their use and wrote a manuscript related to the selected supplement in a uniformed and agreed format. The agreed format was constructed to provide a base of knowledge, so physicians and other health care providers would be able to discuss realistic expectations and potential benefits and risks with patients and families that seek that kind of information.

5. The revised document was circulated among participants for revisions and comments.

This method might look fine to proponents of alternative medicine, but from a scientific point of view, it is seriously wanting. Essentially, they asked those experts who are in favour of a given supplement to write a report to justify his/her preference. This method is not just open bias, it formally invites bias.

Predictably then, the reviews of the 10 chosen supplements are woefully inadequate. These is no evidence of a systematic approach; the cited evidence is demonstrably cherry-picked; there is a complete lack of critical analysis; for several supplements, clinical data are virtually absent without the authors finding this embarrassing void a reason for concern; dosage recommendations are often vague and naïve, to say the least (for instance, for milk thistle: 200 to 400 mg per day – without indication of what the named weight range refers to, the fresh plant, dried powder, extract…?); safety data are incomplete and nobody seems to mind that supplements are not subject to systematic post-marketing surveillance; the text is full of naïve thinking and contradictions (e.g.”There are no reported side effects of the mushroom extracts or the Maitake D-fraction. As Maitake may lower blood sugar, it should be used with caution in patients with diabetes“); evidence suggesting that a given supplement might reduce the risk of cancer is presented as though this means it is an effective treatment for an existing cancer; cancer is usually treated as though it is one disease entity without any differentiation of different cancer types.

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. But I do wonder, isn’t being in favour of integrating half-baked nonsense into cancer care and being selected for one’s favourable attitude towards certain supplements already a conflict of interest?

In any case, the review is in my view not of sufficient rigor to form the basis for well-informed discussions with patients. The authors of the review cite a guideline by the ‘Society of Integrative Oncology’ for the use of supplements in cancer care which states: For cancer patients who wish to use nutritional supplements, including botanicals for purported antitumor effects, it is recommended that they consult a trained professional. During the consultation, the professional should provide support, discuss realistic expectations, and explore potential benefits and risks. It is recommended that use of those agents occur only in the context of clinical trials, recognized nutritional guidelines, clinical evaluation of the risk/benefit ratio based on available evidence, and close monitoring of adverse effects. It seems to me that, with this review, the authors have not adhered to their own guideline.

Criticising the work of others is perhaps not very difficult, however, doing a better job usually is. So, can I offer anything that is better than the above criticised review? The answer is YES. Our initiative ‘CAM cancer’ provides up-to-date, concise and evidence-based systematic reviews of many supplements and other alternative treatments that cancer patients are likely to hear about. Their conclusions are not nearly as uncritically positive as those of the article in ‘INTEGRATIVE CANCER THERAPIES’.

I happen to believe that it is important for cancer patients to have access to reliable information and that it is unethical to mislead them with biased accounts about the value of any treatment.

Gravityscan Badge

Recent Comments

Note that comments can be edited for up to five minutes after they are first submitted.


Click here for a comprehensive list of recent comments.

Categories