‘THE HINKLEY TIMES’ is not a paper that I read often, I have to admit – but maybe I should! It was there that I found the following remarkable article:
Bosworth MP David Tredinnick has asked questions in the House of Commons about the growing problem of antibiotic resistance within hospitals, suggesting herbal remedies could be answer.
The Tory MP, who has a keen interest in alternative medicine particularly herbal curatives, asked Jeremy Hunt, Secretary of State for Health, whether the problem was being discussed at the very top level.
He said: “Does my right honourable friend agree that a critical problem that A and E units will face in the future is antibiotic resistance? Is he aware that the science and technology committee, of which I am a member, has been looking at this issue and it also interests the health committee, of which I am also a member? Can he assure me that he is talking to the Prime Minister about how to stimulate new antibiotic research, and will he also remember that nature has its own remedies, such as tea tree oil?”
In reply Mr Hunt said: “My honourable friend is right about the seriousness of the issue of antimicrobial resistance. Some 25,000 people die in Europe every year as a result of the failure of antibiotics – more than die in road traffic accidents. I raised the issue at the World Health Assembly and I have discussed it closely with the Prime Minister.”
David Tredinnick is no stranger to strange ideas. Wikipedia (yes I know, many people do not like it as a source) sums it up quite succinctly:
He is a supporter of complementary and alternative medicine (CAM). He has made supportive comments in Parliament on homeopathy, despite continued lack of evidence of its effectiveness. He has supported chiropractic and mentioned the influence of the Moon on blood clotting. In this same debate he characterised scientists as “racially prejudiced”. He has tabled several early day motions in support of homeopathy’s continued funding on the National Health Service.Tredinnick’s views continue to cause amused disbelief in some quarters and a spokesman for the Royal College of Surgeons of England said they would “laugh their heads off” at the suggestion they could not operate at the full moon.
At the 2010 general election, in addition to candidates from the two main parties, Tredinnick was opposed by New Scientist journalist Dr. Michael Brooks who objected to “Tredinnick’s outspoken promotion of complementary and alternative medicine.”During a hustings debate called by Brooks to “highlight the scientific literacy of the UK’s elected representatives” Brooks claimed that Tredinnick regarded homeopathy as a suitable treatment for Malaria and HIV, which Tredinnick did not deny. Tredinnick in turn argued that “alternative treatments are incredibly good value for money” and stated his belief that randomised controlled trials are not effective at evaluating very dilute preparations.
In March 2013 Tredinnick was ridiculed as “nonsensical” by the government’s outgoing chief scientist, Sir John Beddington, who said the MP had fallen for the “Galileo fallacy” (Galileo was laughed at but was right, therefore since I am laughed at I must be right).
In July 2013 Tredinnick sponsored an EDM congratulating a farmer on his decision to use homeopathy with what were claimed to be positive results.The motion was supported by one other MP but the British veterinary association says there is no evidence of any benefit.
Tredinnick is a supporter of astrology especially the use of it in medical practice.In November 2009, he spoke at a meeting organised by the Astrological Association of Great Britain, where he related his personal experience of astrology and illness, advocating that astrology be integrated into the NHS.
Tredinnick’s appointment to the Health committee in June 2010 was criticised in two science reports in the Guardian. Martin Robbins said his appointment was “an extremely disturbing development” even though “Tredinnick is a figure unlikely to be taken seriously by policymakers” whilst Nature‘s Adam Rutherford described Tredinnick as “misinformed about a great many things” and said that “giving [him] influence on medical policy ..is a bad move.”The Telegraph writer Ian Douglas also described it as “a problem.”
His appointment to the Science and technology committee also drew criticism. Andy McSmith in the Independent, cited his views that homeopathy could cure HIV, TB, malaria, urinary infections, diarrhoea, skin eruptions, diabetes, epilepsy, eye infections, intestinal parasites, cancer, and gangrene amongst others and quoted Imran Khan, head of the Campaign for Science and Engineering, as saying that “someone with such incredibly odd views is not helpful”. Tom Whipple in the Times said his appointment caused despair,whilst Elizabeth Gibney in the Times Higher Education quoted the Skeptical Voter website as saying that Tredinnick is “perhaps the worst example of scientific illiteracy in government”…
In 2009 Tredinnick attempted to claim the £125 cost of attending a course on “intimate relationships” through his Parliamentary expenses. He was also found to have used expenses to purchase astrology software, claiming it was for a debate on alternative medicine.
Compared to some of theses bizarre activities, the notion that herbal remedies might provide the solution for antibiotic resistance seems almost reasonable and clever.
Tredinnick does not seem to know that:
- many antibiotics originate from plants or other natural substances,
- several large pharmaceutical companies are feverishly looking for more such substances from plants,
- most plants do actually contain substances which have antibiotic activity,
- however, most cannot be used as medicines, for instance, because they are far too toxic (tea tree oil is a good example for this),
- once a pure compound has been isolated from a plant and is used therapeutically, it ceases to be herbal medicine (which is defined as the use of full plant extracts),
- it is thus unlikely that full plant extracts, i. e. herbal medicine, will ever provide a solution to antibiotic resistance.
I have little doubt that Tredinnick will continue to mislead parliament and the public with his nonsensical views about alternative medicine. And even if it might have no effect whatsoever, I will continue to point out just how nonsensical they are.
Some time ago, I published a post entitled HOW TO BECOME A CHARLATAN. This prompted ‘THE NORWEGIAN ACADEMY OF SCIENCE AND LETTERS’ to invite me to give a lecture on the subject, a great honour, I am sure. Consequently, I have thought about this somewhat unusual subject quite a lot.
Obviously, my thoughts come from the perspective of someone who has researched alternative medicine for many years. Pseudoscientists seem to love alternative medicine and proponents of alternative medicine love pseudoscience. As a result, alternative medicine is densely populated by pseudoscientists.
But what is the characteristic of pseudoscience? Reflecting on this question, I found not one but several hallmarks (and for each of them, there are many posts on this blog which provide further explanations):
- A bizarre theory elevated to the status of a dogma
- Conspiracy theories
- The abuse of science (not for testing but) for confirming assumptions.
- Jumping to conclusions
- The torture of data (until they ‘confess’)
- The use of fallacies
- The absence of rigorous investigations into the risks of their treatments
- The disregard of the methodological quality of data supporting their treatments.
- Telling and promoting untruths
- Ad hominem attacks
Based on these 12 hallmarks, one could create a simple score which indicates the likelihood of the presence of pseudoscience. In other words, it might be useful to consider pseudoscience in terms of a sliding scale. Some things in alternative medicine can be just a bit pseudoscientific, others quite a lot, while others again are hopelessly so.
The issue of pseudoscience is by no means just academic; it is very real problem and has many important, practical implications. The most important one probably is that, in health care (and other areas as well), pseudoscience can be harmful, even to the point that it costs lives of vulnerable patients who believe that everything masquerading as science can be relied upon.
I find it always nice to see that people appreciate my work. Yet sometimes I am a little surprised to realise what some commercially interested firms make of it. Recently I came across a website that proudly used my research for advertising the use of magnetic bracelets against pain. Here is the text in question:
The extra strong magnets make this magnetic bracelet the fastest acting pain reliever. While wearing this magnetic bracelet customers with wrist and hand pain report significant pain relief….
What is a magnetic bracelet and what are the benefits? Magnetic bracelets are a piece of jewelry, worn for the therapeutic benefits of the magnetic field. Magnetic bracelets has been used successfully by many people for pain relief of inflammatory conditions such as arthritis, tendinitis and bursitis.
A randomized, placebo controlled trial with three parallel groups, came to the conclusion : Pain from osteoarthritis of the hip and knee decreases when wearing magnetic bracelets. It is uncertain whether this response is due to specific or non-specific (placebo) effects. Tim Harlow, general practitioner, Colin Greaves, research fellow, Adrian White, senior research fellow, Liz Brown, research assistant, Anna Hart, statistician, Edzard Ernst, professor of complementary medicine.
The entrepreneurs seem to have forgotten a few things which we tried to make clear in our paper:
- this article was published in the Christmas issue of the BMJ which specialises in publishing unusual and odd findings with a high entertainment value,
- in our paper, we point out that “the contamination of group B with stronger magnets prevented a more objective estimation of any-placebo effect”,
- and stressed that “there were problems with the weak magnets”,
- and that “a per-specification analysis suggested (but could not confirm) a specific effect of magnetic bracelets over and above placebo”.
Most importantly, this was just one trial, and surely one swallow does not make a summer! We should always consider the totality of the reliable evidence. Being conscientious researchers, at the time, we did exactly that and conducted a systematic review. Here is the abstract in its full beauty:
Static magnets are marketed with claims of effectiveness for reducing pain, although evidence of scientific principles or biological mechanisms to support such claims is limited. We performed a systematic review and meta-analysis to assess the clinical evidence from randomized trials of static magnets for treating pain.
Systematic literature searches were conducted from inception to March 2007 for the following data sources: MEDLINE, EMBASE, AMED (Allied and Complementary Medicine Database), CINAHL, Scopus, the Cochrane Library and the UK National Research Register. All randomized clinical trials of static magnets for treating pain from any cause were considered. Trials were included only if they involved a placebo control or a weak magnet as the control, with pain as an outcome measure. The mean change in pain, as measured on a 100-mm visual analogue scale, was defined as the primary outcome and was used to assess the difference between static magnets and placebo.
Twenty-nine potentially relevant trials were identified. Nine randomized placebo-controlled trials assessing pain with a visual analogue scale were included in the main meta-analysis; analysis of these trials suggested no significant difference in pain reduction (weighted mean difference [on a 100-mm visual analogue scale] 2.1 mm, 95% confidence interval -1.8 to 5.9 mm, p = 0.29). This result was corroborated by sensitivity analyses excluding trials of acute effects and conditions other than musculoskeletal conditions. Analysis of trials that assessed pain with different scales suggested significant heterogeneity among the trials, which means that pooling these data is unreliable.
The evidence does not support the use of static magnets for pain relief, and therefore magnets cannot be recommended as an effective treatment. For osteoarthritis, the evidence is insufficient to exclude a clinically important benefit, which creates an opportunity for further investigation.
So, would I, on the basis of the current best evidence, recommend magnetic bracelets to people who suffer from pain? No! In my view, only charlatans would do such a thing.
When I first read about agrohomeopathy (i.e. the use of potentised preparations for the health of plants and soils) I thought that it must be a hoax. Then I realised that it was entirely serious (a Google search returns ~28 000 hits for ‘agrohomeopathy’) – serious but nevertheless too weird for words. Because it is so utterly unbelievable, I cite (in italics) the key parts of an article on the subject.
What’s better than ORGANIC or BIODYNAMIC farming? AGROHOMEOPATHY! What is Agrohomeopathy? It’s the specialized area of homeopathy used to treat your garden and crops. Agrohomeopathy is the most chemical free, non-toxic method of growing food and other crops that you can get. Agrohomeopathy makes your plants resistant to disease and pests by strengthening them from the inside out. In nature, it is the weakest of organisms that are attacked and destroyed. Agrohomeopathy helps build up the plant’s basic structure and gives it optimum health, thus reducing and sometimes even eliminating it’s susceptibility. And the skeptics can’t blame THESE effects on placebo, can they?!….
Homeopathic treatment for your crops is a win-win situation. It is backed by decades of research and practice. Try it for yourself and see. And if you have problems or need help, there are experts in the field who are eager to help, who want to get your feedback & experience…
If you think this is far-fetched, rest assured that other sources go even further. Look at this statement, for instance:
Agro-Homeopathy not only treats the disease symptoms of the plant and performs preventive actions, but can also treat traumas retained in the biological memory of the plant, which resulted from conditions such as forced hybridization, moving to places outside their natural habitats , or exaggerated fertilization that maximizes production to the extreme.
So, plants have a ‘biological memory’ that is able to retain information of a past trauma! Fascinating, this gets more fantastic by the minute.
And there is plenty of practical advice too; just consider this helpful hint, if you are a keen gardener: the effects and benefits of homeopathic Silicea are so numerous that an entire article has been devoted to them at: Homeopathic Silica – The Gardener’s Friend. Needless to say, Silicea is one remedy no gardener or farmer should be without…
According to this website, homeopathic silica is a miracle cure; it
- Aids germination of seeds
- Reduces transplant shock
- Strengthens weak and spindly plants
- Increases vigour and resistance of plants to pests, moulds, and mildew
- Aids water retention in plants growing on arid soils
- Stimulates flower growth, both in number and size
- Assists seed generation and development
- Improves fruit-setting when applied after flowering
- Stimulates premature flowering and prevents seed formation when applied in overdose to weeds
- Changes the ionisation of soil particles so that water-repellent soil readily absorbs moisture … and more!
The sceptics can indeed not blame ‘THESE effects’ on placebo. Nobody needs to do that because they do not exist! I could not find a single piece of reliable evidence to demonstrate that highly diluted homeopathic remedies can cure diseases of plants.
I hope that a few agrohomeopathic readers of these lines will correct me by showing me solid data – but somehow I doubt it.
A recent meta-analysis evaluated the efficacy of acupuncture for treatment of irritable bowel syndrome (IBS) and arrived at bizarrely positive conclusions.
The authors state that they searched 4 electronic databases for double-blind, placebo-controlled trials investigating the efficacy of acupuncture in the management of IBS. Studies were screened for inclusion based on randomization, controls, and measurable outcomes reported.
Six RCTs were included in the meta-analysis, and 5 articles were of high quality. The pooled relative risk for clinical improvement with acupuncture was 1.75 (95%CI: 1.24-2.46, P = 0.001). Using two different statistical approaches, the authors confirmed the efficacy of acupuncture for treating IBS and concluded that acupuncture exhibits clinically and statistically significant control of IBS symptoms.
As IBS is a common and often difficult to treat condition, this would be great news! But is it true? We do not need to look far to find the embarrassing mistakes and – dare I say it? – lies on which this result was constructed.
The largest RCT included in this meta-analysis was neither placebo-controlled nor double blind; it was a pragmatic trial with the infamous ‘A+B versus B’ design. Here is the key part of its methods section: 116 patients were offered 10 weekly individualised acupuncture sessions plus usual care, 117 patients continued with usual care alone. Intriguingly, this was the ONLY one of the 6 RCTs with a significantly positive result!
The second largest study (as well as all the other trials) showed that acupuncture was no better than sham treatments. Here is the key quote from this trial: there was no statistically significant difference between acupuncture and sham acupuncture.
So, let me re-write the conclusions of this meta-analysis without spin, lies or hype: These results of this meta-analysis seem to indicate that:
- currently there are several RCTs testing whether acupuncture is an effective therapy for IBS,
- all the RCTs that adequately control for placebo-effects show no effectiveness of acupuncture,
- the only RCT that yields a positive result does not make any attempt to control for placebo-effects,
- this suggests that acupuncture is a placebo,
- it also demonstrates how misleading studies with the infamous ‘A+B versus B’ design can be,
- finally, this meta-analysis seems to be a prime example of scientific misconduct with the aim of creating a positive result out of data which are, in fact, negative.
There is much debate about the usefulness of chiropractic. Specifically, many people doubt that their chiropractic spinal manipulations generate more good than harm, particularly for conditions which are not related to the spine. But do chiropractors treat such conditions frequently and, if yes, what techniques do they employ?
This investigation was aimed at describing the clinical practices of chiropractors in Victoria, Australia. It was a cross-sectional survey of 180 chiropractors in active clinical practice in Victoria who had been randomly selected from the list of 1298 chiropractors registered on Chiropractors Registration Board of Victoria. Twenty-four chiropractors were ineligible, 72 agreed to participate, and 52 completed the study.
Each participating chiropractor documented encounters with up to 100 consecutive patients. For each chiropractor-patient encounter, information collected included patient health profile, patient reasons for encounter, problems and diagnoses, and chiropractic care.
Data were collected on 4464 chiropractor-patient encounters between 11 December 2010 and 28 September 2012. In most (71%) cases, patients were aged 25-64 years; 1% of encounters were with infants. Musculoskeletal reasons for the consultation were described by patients at a rate of 60 per 100 encounters, while maintenance and wellness or check-up reasons were described at a rate of 39 per 100 encounters. Back problems were managed at a rate of 62 per 100 encounters.
The most frequent care provided by the chiropractors was spinal manipulative therapy and massage. The table shows the precise conditions treated
|Problem group||No. (%) of recorded diagnoses* (n = 5985)||Rate per 100 encounters (n = 4417)||95% CI||ICC|
|Back problem||2757 (46.07%)||62.42||(55.24–70.53)||0.312|
|Neck problem||683 (11.41%)||15.46||(11.23–21.30)||0.233|
|Muscle problem||434 (7.25%)||9.83||(6.64–14.55)||0.207|
|Health maintenance or preventive care||254 (4.24%)||5.75||(3.24–10.22)||0.251|
|Back syndrome with radiating pain||215 (3.59%)||4.87||(2.91–8.14)||0.165|
|Musculoskeletal symptom or complaint, or other||219 (3.66%)||4.96||(2.39–10.28)||0.350|
|Sprain or strain of joint||167 (2.79%)||3.78||(2.30–6.22)||0.115|
|Shoulder problem||87 (1.45%)||1.97||(1.37–2.83)||0.022|
|Nerve-related problem||62 (1.04%)||1.40||(0.72–2.75)||0.072|
|General symptom or complaint, other||51 (0.85%)||1.15||(0.22–6.06)||0.407|
|Bursitis, tendinitis or synovitis||47 (0.79%)||1.06||(0.71–1.60)||0.011|
|Kyphosis and scoliosis||47 (0.79%)||1.06||(0.65–1.75)||0.023|
|Foot or toe symptom or complaint||48 (0.80%)||1.09||(0.41–2.87)||0.123|
|Ankle problem||46 (0.77%)||1.04||(0.40–2.69)||0.112|
|Osteoarthrosis, other (not spine)||39 (0.65%)||0.88||(0.51–1.53)||0.023|
|Hip symptom or complaint||35 (0.58%)||0.79||(0.53–1.19)||0.006|
|Leg or thigh symptom or complaint||35 (0.58%)||0.79||(0.49–1.28)||0.012|
|Musculoskeletal injury||33 (0.55%)||0.75||(0.45–1.24)||0.013|
These findings are impressive in that they suggest that most Australian chiropractors treat non-spinal conditions for which there is no evidence that the most frequently used interventions are effective. The treatments employed are depicted in this graph:
Distribution of techniques and care provided by chiropractors, with 95% CI
[Activator = hand-held spring-loaded device that delivers an impulse to the spine. Drop piece = chiropractic treatment table with a segmented drop system which quickly lowers the section of the patient’s body corresponding with the spinal region being treated. Blocks = wedge-shaped blocks placed under the pelvis.
Chiro system = chiropractic system of care, eg, Applied Kinesiology, Sacro-Occipital Technique, Neuroemotional Technique. Flexion distraction = chiropractic treatment table that flexes in the middle to provide traction and mobilisation to the lumbar spine.]
There is no good evidence I know of demonstrating these techniques to be effective for the majority of the conditions listed in the above table.
A similar bone of contention is the frequent use of ‘maintenance’ and ‘wellness’ care. The authors of the article comment: The common use of maintenance and wellness-related terms reflects current debate in the chiropractic profession. “Chiropractic wellness care” is considered by an indeterminate proportion of the profession as an integral part of chiropractic practice, with the belief that regular chiropractic care may have value in maintaining and promoting health, as well as preventing disease. The definition of wellness chiropractic care is controversial, with some chiropractors promoting only spine care as a form of wellness, and others promoting evidence-based health promotion, eg, smoking cessation and weight reduction, alongside spine care. A 2011 consensus process in the chiropractic profession in the United States emphasised that wellness practice must include health promotion and education, and active strategies to foster positive changes in health behaviours. My own systematic review of regular chiropractic care, however, shows that the claimed effects are totally unproven.
One does not need to be overly critical to conclude from all this that the chiropractors surveyed in this investigation earn their daily bread mostly by being economical with the truth regarding the lack of evidence for their actions.
When we talk about conflicts of interest, we usually think of financial concerns. But conflicts of interests also extend to non-financial matters, such as strong beliefs. These are important in alternative medicine – I would even go as far as to claim that they dominate this field.
My detractors have often claimed that this is where my problem lies. They are convinced that, in 1993, I came into the job as PROFESSOR OF COMPLEMENTARY MEDICINE with an axe to grind; I was determined or perhaps even paid to show that all alternative medicine is utter hocus-pocus, they say. The truth is that, if anything, I was on the side of alternative medicine – and I can prove it. Using the example of homeopathy, I have dedicated an entire article to demonstrate that the myth is untrue – I was not closed-minded or out to ditch homeopathy (or any other form of alternative medicine for that matter).
What then could constitute my ‘conflict of interest’? Surely, he was bribed, I hear them say. Just look at the funds he took from industry. Some of those people have even gone to the trouble of running freedom of information requests to obtain the precise figures for my research-funding. Subsequently they triumphantly publish them and say: Look he got £x from this company and £y from that firm. And they are, of course, correct: I did receive support from commercially interested parties on several occasions. But what my detractors forget is that these were all pro-alternative medicine institutions. More importantly, I always made very sure that no strings were attached with any funds we accepted.
Our core funds came from ‘The Laing Foundation’ which endowed Exeter University with £ 1.5 million. This was done with the understanding that Exeter would put the same amount again into the kitty (which they never did). Anyone who can do simple arithmetic can tell that, to sustain up to 20 staff for almost 20 years, £1.5 million is not nearly enough. There must have been other sources. Who exactly gave money?
Despite utterly useless fundraising by the University, we did manage to obtain additional funds. I managed to receive support in the form of multiple research fellowships, for instance. It came from various sources; for instance, manufacturers of herbal medicines, Boots, the Pilkington Family Trust (yes, the glass manufacturers).
A hugely helpful contributor to our work was the sizable number (I estimate around 30) of visitors from abroad who came on their own money simply because they wanted to learn from and with us. They stayed between 3 months and 4 years, and importantly contributed to our research, knowledge and fun.
In addition, we soon devised ways to generate our own money. For instance, we started an annual conference for researchers in our field which ran for 14 successful years. As we managed everything on a shoestring and did all the organisation ourselves, we made a tidy profit each year which, of course, went straight back into our research. We also published several books which generated some revenue for the same purpose.
And then we received research funding for specific projects, for instance, from THE PRINCE OF WALES’ FOUNDATION FOR INTEGRATED HEALTH, a Japanese organisation supporting Jorhei Healing, THE WELCOME TRUST, the NHS, and even a homeopathic company.
So, do I have a conflict of interest? Did I take money from anyone who might have wanted to ditch alternative medicine? I don’t think so! And if I tell you that, when I came to Exeter in 1993, I donated ~£120 000 of my own funds towards the research of my unit, even my detractors might, for once, be embarrassed to have thought otherwise.
Do you think that chiropractic is effective for asthma? I don’t – in fact, I know it isn’t because, in 2009, I have published a systematic review of the available RCTs which showed quite clearly that the best evidence suggested chiropractic was ineffective for that condition.
But this is clearly not true, might some enthusiasts reply. What is more, they can even refer to a 2010 systematic review which indicates that chiropractic is effective; its conclusions speak a very clear language: …the eight retrieved studies indicated that chiropractic care showed improvements in subjective measures and, to a lesser degree objective measures… How on earth can this be?
I would not be surprised, if chiropractors claimed the discrepancy is due to the fact that Prof Ernst is biased. Others might point out that the more recent review includes more studies and thus ought to be more reliable. The newer review does, in fact, have about twice the number of studies than mine.
How come? Were plenty of new RCTs published during the 12 months that lay between the two publications? The answer is NO. But why then the discrepant conclusions?
The answer is much less puzzling than you might think. The ‘alchemists of alternative medicine’ regularly succeed in smuggling non-evidence into such reviews in order to beautify the overall picture and confirm their wishful thinking. The case of chiropractic for asthma does by no means stand alone, but it is a classic example of how we are being misled by charlatans.
Anyone who reads the full text of the two reviews mentioned above will find that they do, in fact, include exactly the same amount of RCTs. The reason why they arrive at different conclusions is simple: the enthusiasts’ review added NON-EVIDENCE to the existing RCTs. To be precise, the authors included one case series, one case study, one survey, two randomized controlled trials (RCTs), one randomized patient and observer blinded cross-over trial, one single blind cross study design, and one self-reported impairment questionnaire.
Now, there is nothing wrong with case reports, case series, or surveys – except THEY TELL US NOTHING ABOUT EFFECTIVENESS. I would bet my last shirt that the authors know all of that; yet they make fairly firm and positive conclusions about effectiveness. As the RCT-results collectively happen to be negative, they even pretend that case reports etc. outweigh the findings of RCTs.
And why do they do that? Because they are interested in the truth, or because they don’t mind using alchemy in order to mislead us? Your guess is as good as mine.
Systematic reviews are widely considered to be the most reliable type of evidence for judging the effectiveness of therapeutic interventions. Such reviews should be focused on a well-defined research question and identify, critically appraise and synthesize the totality of the high quality research evidence relevant to that question. Often it is possible to pool the data from individual studies and thus create a new numerical result of the existing evidence; in this case, we speak of a meta-analysis, a sub-category of systematic reviews.
One strength of systematic review is that they minimise selection and random biases by considering at the totality of the evidence of a pre-defined nature and quality. A crucial precondition, however, is that the quality of the primary studies is critically assessed. If this is done well, the researchers will usually be able to determine how robust any given result is, and whether high quality trials generate similar findings as those of lower quality. If there is a discrepancy between findings from rigorous and flimsy studies, it is obviously advisable to trust the former and discard the latter.
And this is where systematic reviews of alternative treatments can run into difficulties. For any given research question in this area we usually have a paucity of primary studies. Equally important is the fact that many of the available trials tend to be of low quality. Consequently, there often is a lack of high quality studies, and this makes it all the more important to include a robust critical evaluation of the primary data. Not doing so would render the overall result of the review less than reliable – in fact, such a paper would not qualify as a systematic review at all; it would be a pseudo-systematic review, i.e. a review which pretends to be systematic but, in fact, is not. Such papers are a menace in that they can seriously mislead us, particularly if we are not familiar with the essential requirements for a reliable review.
This is precisely where some promoters of bogus treatments seem to see their opportunity of making their unproven therapy look as though it was evidence-based. Pseudo-systematic reviews can be manipulated to yield a desired outcome. In my last post, I have shown that this can be done by including treatments which are effective so that an ineffective therapy appears effective (“chiropractic is so much more than just spinal manipulation”). An even simpler method is to exclude some of the studies that contradict one’s belief from the review. Obviously, the review would then not comprise the totality of the available evidence. But, unless the reader bothers to do a considerable amount of research, he/she would be highly unlikely to notice. All one needs to do is to smuggle the paper past the peer-review process – hardly a difficult task, given the plethora of alternative medicine journals that bend over backwards to publish any rubbish as long as it promotes alternative medicine.
Alternatively (or in addition) one can save oneself a lot of work and omit the process of critically evaluating the primary studies. This method is increasingly popular in alternative medicine. It is a fool-proof method of generating a false-positive overall result. As poor quality trials have a tendency to deliver false-positive results, it is obvious that a predominance of flimsy studies must create a false-positive result.
A particularly notorious example of a pseudo-systematic review that used this as well as most of the other tricks for misleading the reader is the famous ‘systematic’ review by Bronfort et al. It was commissioned by the UK GENERAL CHIROPRACTIC COUNCIL after the chiropractic profession got into trouble and was keen to defend those bogus treatments disclosed by Simon Singh. Bronfort and his colleagues thus swiftly published (of course, in a chiro-journal) an all-encompassing review attempting to show that, at least for some conditions, chiropractic was effective. Its lengthy conclusions seemed encouraging: Spinal manipulation/mobilization is effective in adults for: acute, subacute, and chronic low back pain; migraine and cervicogenic headache; cervicogenic dizziness; manipulation/mobilization is effective for several extremity joint conditions; and thoracic manipulation/mobilization is effective for acute/subacute neck pain. The evidence is inconclusive for cervical manipulation/mobilization alone for neck pain of any duration, and for manipulation/mobilization for mid back pain, sciatica, tension-type headache, coccydynia, temporomandibular joint disorders, fibromyalgia, premenstrual syndrome, and pneumonia in older adults. Spinal manipulation is not effective for asthma and dysmenorrhea when compared to sham manipulation, or for Stage 1 hypertension when added to an antihypertensive diet. In children, the evidence is inconclusive regarding the effectiveness for otitis media and enuresis, and it is not effective for infantile colic and asthma when compared to sham manipulation. Massage is effective in adults for chronic low back pain and chronic neck pain. The evidence is inconclusive for knee osteoarthritis, fibromyalgia, myofascial pain syndrome, migraine headache, and premenstrual syndrome. In children, the evidence is inconclusive for asthma and infantile colic.
Chiropractors across the world cite this paper as evidence that chiropractic has at least some evidence base. What they omit to tell us (perhaps because they do not appreciate it themselves) is the fact that Bronfort et al
- failed to formulate a focussed research question,
- invented his own categories of inconclusive findings,
- included all sorts of studies which had nothing to do with chiropractic,
- and did not to make an assessment of the quality of the included primary studies they included in their review.
If, for a certain condition, three trials were included, for instance, two of which were positive but of poor quality and one was negative but of good quality, the authors would conclude that, overall, there is sound evidence.
Bronfort himself is, of course, more than likely to know all that (he has learnt his trade with an excellent Dutch research team and published several high quality reviews) – but his readers mostly don’t. And for chiropractors, this ‘systematic’ review is now considered to be the most reliable evidence in their field.
Whenever a new trial of an alternative intervention emerges which fails to confirm the wishful thinking of the proponents of that therapy, the world of alternative medicine is in turmoil. What can be done about yet another piece of unfavourable evidence? The easiest solution would be to ignore it, of course – and this is precisely what is often tried. But this tactic usually proves to be unsatisfactory; it does not neutralise the new evidence, and each time someone brings it up, one has to stick one’s head back into the sand. Rather than denying its existence, it would be preferable to have a tool which invalidates the study in question once and for all.
The ‘fatal flaw’ solution is simpler than anticipated! Alternative treatments are ‘very special’, and this notion must be emphasised, blown up beyond all proportions and used cleverly to discredit studies with unfavourable outcomes: the trick is simply to claim that studies with unfavourable results have a ‘fatal flaw’ in the way the alternative treatment was applied. As only the experts in the ‘very special’ treatment in question are able to judge the adequacy of their therapy, nobody is allowed to doubt their verdict.
Take acupuncture, for instance; it is an ancient ‘art’ which only the very best will ever master – at least that is what we are being told. So, all the proponents need to do in order to invalidate a trial, is read the methods section of the paper in full detail and state ‘ex cathedra’ that the way acupuncture was done in this particular study is completely ridiculous. The wrong points were stimulated, or the right points were stimulated but not long enough [or too long], or the needling was too deep [or too shallow], or the type of stimulus employed was not as recommended by TCM experts, or the contra-indications were not observed etc. etc.
As nobody can tell a correct acupuncture from an incorrect one, this ‘fatal flaw’ method is fairly fool-proof. It is also ever so simple: acupuncture-fans do not necessarily study hard to find the ‘fatal flaw’, they only have to look at the result of a study – if it was favourable, the treatment was obviously done perfectly by highly experienced experts; if it was unfavourable, the therapists clearly must have been morons who picked up their acupuncture skills in a single weekend course. The reasons for this judgement can always be found or, if all else fails, invented.
And the end-result of the ‘fatal flaw’ method is most satisfactory; what is more, it can be applied to all alternative therapies – homeopathy, herbal medicine, reflexology, Reiki healing, colonic irrigation…the method works for all of them! What is even more, the ‘fatal flaw’ method is adaptable to other aspects of scientific investigations such that it fits every conceivable circumstance.
An article documenting the ‘fatal flaw’ has to be published, of course – but this is no problem! There are dozens of dodgy alternative medicine journals which are only too keen to print even the most far-fetched nonsense as long as it promotes alternative medicine in some way. Once this paper is published, the proponents of the therapy in question have a comfortable default position to rely on each time someone cites the unfavourable study: “WHAT NOT THAT STUDY AGAIN! THE TREATMENT HAS BEEN SHOWN TO BE ALL WRONG. NOBODY CAN EXPECT GOOD RESULTS FROM A THERAPY THAT WAS NOT CORRECTLY ADMINISTERED. IF YOU DON’T HAVE BETTER STUDIES TO SUPPORT YOUR ARGUMENTS, YOU BETTER SHUT UP.”
There might, in fact, be better studies – but chances are that the ‘other side’ has already documented a ‘fatal flaw’ in them too.