reviewer bias

Musculoskeletal and rheumatic conditions, often just called “arthritis” by lay people, bring more patients to alternative practitioners than any other type of disease. It is therefore particularly important to know whether alternative medicines (AMs) demonstrably generate more good than harm for such patients. Most alternative practitioners, of course, firmly believe in what they are doing. But what does the reliable evidence show?

To find out, ‘Arthritis Research UK’ has sponsored a massive project  lasting several years to review the literature and critically evaluate the trial data. They convened a panel of experts (I was one of them) to evaluate all the clinical trials that are available in 4 specific clinical areas. The results for those forms of AM that are to be taken by mouth or applied topically have been published some time ago, now the report, especially written for lay people, on those treatments that are practitioner-based has been published. It covers the following 25 modalities: 


Alexander technique


Autogenic training


Chiropractic (spinal manipulation)

Copper bracelets

Craniosacral therapy

Crystal healing


Kinesiology (applied kinesiology)

Healing therapies



Magnet therapy (static magnets)



Music therapy

Osteopathy (spinal manipulation)

Qigong (internal qigong)


Relaxation therapy


Tai chi


Our findings are somewhat disappointing: only very few treatments were shown to be effective.

In the case of rheumatoid arthritis, 24 trials were included with a total of 1,500 patients. The totality of this evidence failed to provide convincing evidence that any form of AM is effective for this particular condition.

For osteoarthritis, 53 trials with a total of ~6,000 patients were available. They showed reasonably sound evidence only for two treatments: Tai chi and acupuncture.

Fifty trials were included with a total of ~3,000 patients suffering from fibromyalgia. The results provided weak evidence for Tai chi and relaxation-therapies, as well as more conclusive evidence for acupuncture and massage therapy.

Low back pain had attracted more research than any of the other diseases: 75 trials with ~11,600 patients. The evidence for Alexander Technique, osteopathy and relaxation therapies was promising by not ultimately convincing, and reasonably good evidence in support of yoga and acupuncture was also found.

The majority of the experts felt that the therapies in question did not frequently cause harm, but there were two important exceptions: osteopathy and chiropractic. For both, the report noted the existence of frequent yet mild, as well as serious but rare adverse effects.

As virtually all osteopaths and chiropractors earn their living by treating patients with musculoskeletal problems, the report comes as an embarrassment for these two professions. In particular, our conclusions about chiropractic were quite clear:

There are serious doubts as to whether chiropractic works for the conditions considered here: the trial evidence suggests that it’s not effective in the treatment of fibromyalgia and there’s only little evidence that it’s effective in osteoarthritis or chronic low back pain. There’s currently no evidence for rheumatoid arthritis.

Our point that chiropractic is not demonstrably effective for chronic back pain deserves some further comment, I think. It seems to be in contradiction to the guideline by NICE, as chiropractors will surely be quick to point out. How can this be?

One explanation is that, since the NICE-guidelines were drawn up, new evidence has emerged which was not positive. The recent Cochrane review, for instance, concludes that spinal manipulation “is no more effective for acute low-back pain than inert interventions, sham SMT or as adjunct therapy”

Another explanation could be that the experts on the panel writing the NICE-guideline were less than impartial towards chiropractic and thus arrived at false-positive or over-optimistic conclusions.

Chiropractors might say that my presence on the ‘Arthritis Research’-panel suggests that we were biased against chiropractic. If anything, the opposite is true: firstly, I am not even aware of having a bias against chiropractic, and no chiropractor has ever demonstrated otherwise; all I ever aim at( in my scientific publications) is to produce fair, unbiased but critical assessments of the existing evidence. Secondly, I was only one of a total of 9 panel members. As the following list shows, the panel included three experts in AM, and most sceptics would probably categorise two of them (Lewith and MacPherson) as being clearly pro-AM:

Professor Michael Doherty – professor of rheumatology, University of Nottingham

Professor Edzard Ernst – emeritus professor of complementary medicine, Peninsula Medical School

Margaret Fisken – patient representative, Aberdeenshire

Dr Gareth Jones (project lead) – senior lecturer in epidemiology, University of Aberdeen

Professor George Lewith – professor of health research, University of Southampton

Dr Hugh MacPherson – senior research fellow in health sciences, University of York

Professor Gary Macfarlane (chair of committee) professor of epidemiology, University of Aberdeen

Professor Julius Sim – professor of health care research, Keele University

Jane Tadman – representative from Arthritis Research UK, Chesterfield

What can we conclude from all that? I think it is safe to say that the evidence for practitioner-based AMs as a treatment of the 4 named conditions is disappointing. In particular, chiropractic is not a demonstrably effective therapy for any of them. This, of course begs the question, for what condition is chiropractic proven to work! I am not aware of any, are you?

The question whether spinal manipulation is an effective treatment for infant colic has attracted much attention in recent years. The main reason for this is, of course, that a few years ago Simon Singh had disclosed in a comment that the British Chiropractic Association (BCA) was promoting chiropractic treatment for this and several other childhood condition on their website. Simon famously wrote “they (the BCA) happily promote bogus treatments” and was subsequently sued for libel by the BCA. Eventually, the BCA lost the libel action as well as lots of money, and the entire chiropractic profession ended up with enough egg on their faces to cook omelets for all their patients.

At the time, the BCA had taken advice from several medical and legal experts; one of their medical advisers, I was told, was Prof George Lewith. Intriguingly, he and several others have just published a Cochrane review of manipulative therapies for infant colic. Here are the unabbreviated conclusions from their article:

The studies included in this meta-analysis were generally small and methodologically prone to bias, which makes it impossible to arrive at a definitive conclusion about the effectiveness of manipulative therapies for infantile colic. The majority of the included trials appeared to indicate that the parents of infants receiving manipulative therapies reported fewer hours crying per day than parents whose infants did not, based on contemporaneous crying diaries, and this difference was statistically significant. The trials also indicate that a greater proportion of those parents reported improvements that were clinically significant. However, most studies had a high risk of performance bias due to the fact that the assessors (parents) were not blind to who had received the intervention. When combining only those trials with a low risk of such performance bias, the results did not reach statistical significance. Further research is required where those assessing the treatment outcomes do not know whether or not the infant has received a manipulative therapy. There are inadequate data to reach any definitive conclusions about the safety of these interventions”

Cochrane reviews also carry a “plain language” summary which might be easier to understand for lay people. And here are the conclusions from this section of the review:

The studies involved too few participants and were of insufficient quality to draw confident conclusions about the usefulness and safety of manipulative therapies. Although five of the six trials suggested crying is reduced by treatment with manipulative therapies, there was no evidence of manipulative therapies improving infant colic when we only included studies where the parents did not know if their child had received the treatment or not. No adverse effects were found, but they were only evaluated in one of the six studies.

If we read it carefully, this article seems to confirm that there is no reliable evidence to suggest that manipulative therapies are effective for infant colic. In the analyses, the positive effect disappears, if the parents are properly blinded;  thus it is due to expectation or placebo. The studies that seem to show a positive effect are false positive, and spinal manipulation is, in fact, not effective.

The analyses disclose another intriguing aspect: most trials failed to mention adverse effects. This confirms the findings of our own investigation and amounts to a remarkable breach of publication ethics (nobody seems to be astonished by this fact; is it normal that chiropractic researchers ignore generally accepted rules of ethics?). It also reflects badly on the ability of the investigators of the primary studies to be objective. They seem to aim at demonstrating only the positive effects of their intervention; science is, however, not about confirming the researchers’ prejudices, it is about testing hypotheses.

The most remarkable thing about the new Cochrane review  is, I think, the in-congruence of the actual results and the authors’ conclusion. To a critical observer, the former are clearly negative but  the latter sound almost positive. I think this begs the question about the possibility of reviewer bias.

We have recently discussed on this blog whether reviews by one single author are necessarily biased. The new Cochrane review has 6 authors, and it seems to me that its conclusions are considerably more biased than my single-author review of chiropractic spinal manipulation for infant colic; in 2009, I concluded simply that “the claim [of effectiveness] is not based on convincing data from rigorous clinical trials”.

Which of the two conclusions describe the facts more helpfully and more accurately?

I think, I rest my case.