MD, PhD, FMedSci, FRSB, FRCP, FRCPEd.

bias

1 2 3 40

I have often cautioned my readers about the ‘evidence’ supporting acupuncture (and other alternative therapies). Rightly so, I think. Here is yet another warning.

This systematic review assessed the clinical effectiveness of acupuncture in the treatment of postpartum depression (PPD). Nine trials involving 653 women were selected. A meta-analysis demonstrated that the acupuncture group had a significantly greater overall effective rate compared with the control group. Moreover, acupuncture significantly increased oestradiol levels compared with the control group. Regarding the HAMD and EPDS scores, no difference was found between the two groups. The Chinese authors concluded that acupuncture appears to be effective for postpartum depression with respect to certain outcomes. However, the evidence thus far is inconclusive. Further high-quality RCTs following standardised guidelines with a low risk of bias are needed to confirm the effectiveness of acupuncture for postpartum depression.

What a conclusion!

What a review!

What a journal!

What evidence!

Let’s start with the conclusion: if the authors feel that the evidence is ‘inconclusive’, why do they state that ‘acupuncture appears to be effective for postpartum depression‘. To me this does simply not make sense!

Such oddities are abundant in the review. The abstract does not mention the fact that all trials were from China (published in Chinese which means that people who cannot read Chinese are unable to check any of the reported findings), and their majority was of very poor quality – two good reasons to discard the lot without further ado and conclude that there is no reliable evidence at all.

The authors also tell us very little about the treatments used in the control groups. In the paper, they state that “the control group needed to have received a placebo or any type of herb, drug and psychological intervention”. But was acupuncture better than all or any of these treatments? I could not find sufficient data in the paper to answer this question.

Moreover, only three trials seem to have bothered to mention adverse effects. Thus the majority of the studies were in breach of research ethics. No mention is made of this in the discussion.

In the paper, the authors re-state that “this meta-analysis showed that the acupuncture group had a significantly greater overall effective rate compared with the control group. Moreover, acupuncture significantly increased oestradiol levels compared with the control group.” This is, I think, highly misleading (see above).

Finally, let’s have a quick look at the journal ‘Acupuncture in Medicine’ (AiM). Even though it is published by the BMJ group (the reason for this phenomenon can be found here: “AiM is owned by the British Medical Acupuncture Society and published by BMJ“; this means that all BMAS-members automatically receive the journal which thus is a resounding commercial success), it is little more than a cult-newsletter. The editorial board is full of acupuncture enthusiasts, and the journal hardly ever publishes anything that is remotely critical of the wonderous myths of acupuncture.

My conclusion considering all this is as follows: we ought to be very careful before accepting any ‘evidence’ that is currently being published about the benefits of acupuncture, even if it superficially looks ok. More often than not, it turns out to be profoundly misleading, utterly useless and potentially harmful pseudo-evidence.


Reference

Acupunct Med. 2018 Jun 15. pii: acupmed-2017-011530. doi: 10.1136/acupmed-2017-011530. [Epub ahead of print]

Effectiveness of acupuncture in postpartum depression: a systematic review and meta-analysis.

Li S, Zhong W, Peng W, Jiang G.

How often do we hear this sentence: “I know, because I have done my research!” I don’t doubt that most people who make this claim believe it to be true.

But is it?

What many mean by saying, “I know, because I have done my research”, is that they went on the internet and looked at a few websites. Others might have been more thorough and read books and perhaps even some original papers. But does that justify their claim, “I know, because I have done my research”?

The thing is, there is research and there is research.

The dictionary defines research as “The systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions.” This definition is helpful because it mentions several issues which, I believe, are important.

Research should be:

  • systematic,
  • an investigation,
  • establish facts,
  • reach new conclusions.

To me, this indicates that none of the following can be truly called research:

  • looking at a few randomly chosen papers,
  • merely reading material published by others,
  • uncritically adopting the views of others,
  • repeating the conclusions of others.

Obviously, I am being very harsh and uncompromising here. Not many people could, according to these principles, truthfully claim to have done research in alternative medicine. Most people in this realm do not fulfil any of those criteria.

As I said, there is research and research – research that meets the above criteria, and the type of research most people mean when they claim: “I know, because I have done my research.”

Personally, I don’t mind that the term ‘research’ is used in more than one way:

  • there is research meeting the criteria of the strict definition
  • and there is a common usage of the word.

But what I do mind, however, is when the real research is claimed to be as relevant and reliable as the common usage of the term. This would be a classical false equivalence, akin to putting experts on a par with pseudo-experts, to believing that facts are no different from fantasy, or to assume that truth is akin to post-truth.

Sadly, in the realm of alternative medicine (and alarmingly, in other areas as well), this is exactly what has happened since quite some time. No doubt, this might be one reason why many consumers are so confused and often make wrong, sometimes dangerous therapeutic decisions. And this is why I think it is important to point out the difference between research and research.

They say that minds are like parachutes – they function only when open. Having an open mind means being receptive to new and different ideas or the opinions of others.

I am regularly accused of lacking this quality. Most recently, an acupuncturist questioned whether acupuncture-sceptics, and I in particular, have an open mind. Subsequently, an interesting dialogue ensued:

___________________________________________________________

Tom Kennedy on Wednesday 01 August 2018 at 19:27

@Rich It sounds to me as if you are at least partly open-minded, and take a more genuinely scientific approach than most here – i.e. rather than dismissing something with a lot of intriguing evidence behind it (even if much of this evidence is still hotly debated) mainly on the grounds that it ‘sounds a bit silly’, you understand that it’s possible to look at something like acupuncture objectively without being put off by the strange terminology associated with it. I strongly urge you to consult various other outlets as well as this one before coming to any final judgement. http://www.evidencebasedacupuncture.org/ for example is run by intelligent people genuinely trying to present the facts as they see them. Yes, they have an ‘agenda’ in that they are acupuncturists, but I can assure you, having had detailed discussions with some of them, that they are motivated by the urge to see acupuncture help more people rather than anything sinister, and they are trying to present an honest appraisal of the evidence. No doubt virtually everyone here will dismiss everything there with (or without) a cursory glance, but perhaps you won’t fall into that category. I hope you find something of interest there, and come to a balanced opinion.

the EVIDENCEBASEDACUPUNCTURE site you recommend quotes the Vickers meta-analysis thus:
“A meta-analysis of 17,922 patients from randomized trials concluded, “Acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option. Significant differences between true and sham acupuncture indicate that acupuncture is more than a placebo.”
Pity that they forgot a bit. The full conclusion reads:
“Acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option. Significant differences between true and sham acupuncture indicate that acupuncture is more than a placebo. However, these differences are relatively modest, suggesting that factors in addition to the specific effects of needling are important contributors to the therapeutic effects of acupuncture.”
AND YOU TRY TO LECTURE US ABOUT AN OPEN MIND?

@Edzard I’m not sure I understand your point. ‘However, these differences are relatively modest, suggesting that factors in addition to the specific effects of needling are important contributors to the therapeutic effects of acupuncture.’ Perhaps the full conclusion should always be quoted, but I don’t think that addendum changes the context significantly. Acupuncture has been shown to be more than a placebo in a large meta-analysis (when compared to arguably active sham controls). The authors put it well I think, in the ‘Interpretation’ section:

‘Our finding that acupuncture has effects over and above sham acupuncture is therefore of major importance for clinical practice. Even though on average these effects are small, the clinical decision made by doctors and patients is not between true and sham acupuncture, but between a referral to an acupuncturist or avoiding such a referral. The total effects of acupuncture, as experienced by the patient in routine practice, include both the specific effects associated with correct needle insertion according to acupuncture theory, non-specific physiologic effects of needling, and non-specific psychological (placebo) effects related to the patient’s belief that treatment will be effective.’

Compare this to Richard’s comment here, for example: ‘Of course the effects of ‘acupuncture’ (if any) are due to placebo responses (and perhaps nocebo responses in some cases). What else?’. And your post tile includes the line ‘the effects of acupuncture are due to placebo’. These are the kinds of comment that to me seem closed-minded in the face of some significant evidence.

edzard on Thursday 02 August 2018 at 12:46

“Perhaps the full conclusion should always be quoted…”

YES, IF NOT, IT’S CALLED ‘BEING ECONOMICAL WITH THE TRUTH’

“…I don’t think that addendum changes the context significantly.”
IT’S NOT AN ADDENDUM, BUT PART OF THE CONCLUSION; AND YOU ARE WRONG, FOR ME, IT CHANGES A LOT.
“…your post tile includes the line ‘the effects of acupuncture are due to placebo’.”
BECAUSE THIS IS WHAT THE PAPER DISCUSSED IN THAT PARTICULAR POST IMPLIED.

I think you need a new keyboard – the caps key seems to be stuck.

‘IT’S CALLED ‘BEING ECONOMICAL WITH THE TRUTH’’

The title of this post is: ‘Yet another confirmation: the effects of acupuncture are due to placebo’. That’s also being economical with the truth I think. You argue ‘BECAUSE THIS IS WHAT THE PAPER DISCUSSED IN THAT PARTICULAR POST IMPLIED’, but is it? The authors state ‘Future studies are needed to confirm or refute any effects of acupuncture in acute stroke’, and that would have been a much more balanced headline. You clearly imply here that it has been CONFIRMED that the effects of acupuncture are due to placebo, and that this trial is further confirmation. This is misleading at best. Yes, you add in brackets ‘(for acute stroke)’ at the end of the post, but why not in the title, unless you want to give the impression this is true for acupuncture in general

Edzard on Thursday 02 August 2018 at 14:09

my post is about critical evaluation of the published literature.
and this is what follows from a critical evaluation of this particular article.
I am not surprised that you cannot follow this line of reasoning.
could it be that the lack of an open mind is not my but your problem

Tom Kennedy on Thursday 02 August 2018 at 14:43

‘could it be that the lack of an open mind is not my but your problem?’

Who knows, maybe the problem is both of ours? I’m open to all possibilities!

VERY GOOD!
ok, let’s have a look.
you 1st: learnt acupuncture [a therapy that relies on a 2000 year old dogma], never published anything negative about it, never used any other therapeutic modality, even treated my own daughter with acupuncture when she suffered from infant colic, earn my livelihood by doing acupuncture.
[I MIGHT BE WRONG HERE, AS I DON’T KNOW ALL THAT MUCH ABOUT YOU, SO PLEASE CORRECT ME] me next: studied acupuncture during my time in med school, used it occasionally, learnt to use dozens of other therapeutic modalities, published lots about acupuncture based on the current evidence [this means that some conclusions – even of my Cochrane reviews – were positive but have since changed], worked with acupuncturists from across the globe, published one book about acupuncture together with several acupuncture fans, now dedicate my time to the critical analysis of the literature and bogus claims, have no conflicts of interest.
[IN CASE YOU KNOW MORE RELEVANT THINGS ABOUT ME, PLEASE ADD]

@Edzard Your summaries seem to be more or less accurate. However, a) I wouldn’t agree with your use of the term ‘dogma’; b) I haven’t published any scientific papers, but I’ve acknowledged various problems in the acupuncture field through informal pieces; c) I’ve used other CAM modalities, and I’ve directly or indirectly experienced many conventional modalities; d) I only earn part of my livelihood by doing acupuncture. Yes, my background makes it more likely that I’ll be biased in favour of acupuncture. But your credentials in no way guarantee open-mindedness on the subject, and personally I don’t see that displayed often on this blog. It still makes for interesting and stimulating reading though.

what problems in the acupuncture field have you acknowledged through informal pieces?
can you provide links?
I want to get a feel for the openness of your mind.
“…your credentials in no way guarantee open-mindedness on the subject, and personally I don’t see that displayed often on this blog.”
1) you seem to forget that blog-posts are not scientific papers, not even close.
2) you also forget that my blog is dedicated to the CRITICAL assessment of alt med.
finally, what would make you think that someone has an open mind towards acupuncture, if not the fact that someone has a track record of publishing positive conclusions about it when the evidence allows?
remember: an open mind should not be so open that your brain falls out!

Tom Kennedy on Friday 03 August 2018 at 11:20

Here’s one example: https://www.tomtheacupuncturist.com/blog/2017/2/24/does-acupuncture-really-work

‘what would make you think that someone has an open mind towards acupuncture, if not the fact that someone has a track record of publishing positive conclusions about it when the evidence allows?’

I think there’s plenty of evidence that allows for positive conclusions about acupuncture, but you don’t report these. I understand the slant of this blog, but I’d say it comes across as ‘negative assessment’ rather than ‘critical assessment’. Perhaps you’ll argue that your critical assessment has led you to a negative assessment? I’ll just have to disagree that that’s a fair and open-minded summary of the evidence.

Out of interest, can I ask what your acupuncture training involved (hours, theory, clinic time etc.)?

I am sorry to say that I see no critical evaluation in the post you linked to.
” I’d say it comes across as ‘negative assessment’ rather than ‘critical assessment’.
have you noticed that criticism is often experienced as negative to the person(s) it is aimed at?

Tom Kennedy on Friday 03 August 2018 at 12:55

‘I am sorry to say that I see no critical evaluation in the post you linked to’

I’ll just have to live with that. I feel as though it acknowledges some of the problems in the acupuncture world, in an attempt at balance. I don’t feel your posts aim for balance, but as you said, a blog isn’t a scientific paper so it’s your prerogative to skew things as you see fit

Edzard on Friday 03 August 2018 at 13:18

it seems to me that the ‘screwing things as you see fit’ is your game.

____________________________________________________________________

This exchange shows how easily I can be provoked to get stroppy and even impolite – I do apologise.

But it also made me wonder: how can anyone be sure to have an open mind?

And how can we decide that a person has a closed mind?

We probably all think we are open minded, but are we correct?

I am not at all sure that I know the answer. It obviously depends a lot on the subject. There are subjects where one hardly needs to keep an open mind and some where it might be advisable to have a closed mind:

  • the notion that the earth is flat,
  • flying carpets,
  • iridology,
  • reflexology,
  • chiropractic subluxation,
  • the vital force,
  • detox,
  • homeopathy.

No doubt, there will be people who even disagree with this short list.

Something that intrigues me – and I am here main ly talking about alternative medicine – is the fact that I often get praised by people who say, “I do appreciate your critical stance on therapy X, but on my treatment Y you are clearly biased and unfairly negative!” To me, it is an indication of a closed mind, if criticism is applauded as long as it does not tackle someone’s own belief system.

On the subject of homeopathy, Prof M Baum and I once published a paper entitled ‘Should we maintain an open mind about homeopathy?’ Its introduction explains the problem quite well, I think:

Once upon a time, doctors had little patience with the claims made for alternative medicines. In recent years the climate has changed dramatically. It is now politically correct to have an open mind about such matters; “the patient knows best” and “it worked for me” seem to be the new mantras. Although this may be a reasonable approach to some of the more plausible aspects of alternative medicine, such as herbal medicine or physical therapies that require manipulation, we believe it cannot apply across the board. Some of these alternatives are based on obsolete or metaphysical concepts of human biology and physiology that have to be described as absurd with proponents who will not subject their interventions to scientific scrutiny or if they do, and are found wanting, suggest that the mere fact of critical evaluation is sufficient to chase the healing process away. These individuals have a conflict of interest more powerful than the requirement for scientific integrity and yet defend themselves by claiming that those wanting to carry out the trials are in the pocket of the pharmaceutical industry and are part of a conspiracy to deny their patients tried and tested palliatives….

END OF QUOTE

And this leads me to try to define 10 criteria indicative for an open mind.

  1. to be free of conflicts of interest,
  2. integrity,
  3. honesty,
  4. to resist the temptation of applying double standards,
  5. to have a track record of having changed one’s views in line with the evidence,
  6. to not cling to overt absurdities,
  7. to reject conspiracy theories,
  8. to be able to engage in a meaningful dialogue with people who have different views,
  9. to avoid fallacious thinking,
  10. to be willing to learn more on the subject in question.

I would be truly interested to hear, if you have further criteria, or indeed any other thoughts on the subject.

The Royal College of Chiropractors (RCC), a Company Limited by guarantee, was given a royal charter in 2013. It has following objectives:

  • to promote the art, science and practice of chiropractic;
  • to improve and maintain standards in the practice of chiropractic for the benefit of the public;
  • to promote awareness and understanding of chiropractic amongst medical practitioners and other healthcare professionals and the public;
  • to educate and train practitioners in the art, science and practice of chiropractic;
  • to advance the study of and research in chiropractic.

In a previous post, I pointed out that the RCC may not currently have the expertise and know-how to meet all these aims. To support the RCC in their praiseworthy endeavours, I therefore offered to give one or more evidence-based lectures on these subjects free of charge.

And what was the reaction?

Nothing!

This might be disappointing, but it is not really surprising. Following the loss of almost all chiropractic credibility after the BCA/Simon Singh libel case, the RCC must now be busy focussing on re-inventing the chiropractic profession. A recent article published by RCC seems to confirm this suspicion. It starts by defining chiropractic:

“Chiropractic, as practised in the UK, is not a treatment but a statutorily-regulated healthcare profession.”

Obviously, this definition reflects the wish of this profession to re-invent themselves. D. D. Palmer, who invented chiropractic 120 years ago, would probably not agree with this definition. He wrote in 1897 “CHIROPRACTIC IS A SCIENCE OF HEALING WITHOUT DRUGS”. This is woolly to the extreme, but it makes one thing fairly clear: chiropractic is a therapy and not a profession.

So, why do chiropractors wish to alter this dictum by their founding father? The answer is, I think, clear from the rest of the above RCC-quote: “Chiropractors offer a wide range of interventions including, but not limited to, manual therapy (soft-tissue techniques, mobilisation and spinal manipulation), exercise rehabilitation and self-management advice, and utilise psychologically-informed programmes of care. Chiropractic, like other healthcare professions, is informed by the evidence base and develops accordingly.”

Many chiropractors have finally understood that spinal manipulation, the undisputed hallmark intervention of chiropractors, is not quite what Palmer made it out to be. Thus, they try their utmost to style themselves as back specialists who use all sorts of (mostly physiotherapeutic) therapies in addition to spinal manipulation. This strategy has obvious advantages: as soon as someone points out that spinal manipulations might not do more good than harm, they can claim that manipulations are by no means their only tool. This clever trick renders them immune to such criticism, they hope.

The RCC-document has another section that I find revealing, as it harps back to what we just discussed. It is entitled ‘The evidence base for musculoskeletal care‘. Let me quote it in its entirety:

The evidence base for the care chiropractors provide (Clar et al, 2014) is common to that for physiotherapists and osteopaths in respect of musculoskeletal (MSK) conditions. Thus, like physiotherapists and osteopaths, chiropractors provide care for a wide range of MSK problems, and may advertise that they do so [as determined by the UK Advertising Standards Authority (ASA)].

Chiropractors are most closely associated with management of low back pain, and the NICE Low Back Pain and Sciatica Guideline ‘NG59’ provides clear recommendations for managing low back pain with or without sciatica, which always includes exercise and may include manual therapy (spinal manipulation, mobilisation or soft tissue techniques such as massage) as part of a treatment package, with or without psychological therapy. Note that NG59 does not specify chiropractic care, physiotherapy care nor osteopathy care for the non-invasive management of low back pain, but explains that: ‘mobilisation and soft tissue techniques are performed by a wide variety of practitioners; whereas spinal manipulation is usually performed by chiropractors or osteopaths, and by doctors or physiotherapists who have undergone additional training in manipulation’ (See NICE NG59, p806).

The Manipulative Association of Chartered Physiotherapists (MACP), recently renamed the Musculoskeletal Association of Chartered Physiotherapists, is recognised as the UK’s specialist manipulative therapy group by the International Federation of Orthopaedic Manipulative Physical Therapists, and has approximately 1100 members. The UK statutory Osteopathic Register lists approximately 5300 osteopaths. Thus, collectively, there are approximately twice as many osteopaths and manipulating physiotherapists as there are chiropractors currently practising spinal manipulation in the UK.

END OF QUOTE

To me this sounds almost as though the RCC is saying something like this:

  1. We are very much like physiotherapists and therefore all the positive evidence for physiotherapy is really also our evidence. So, critics of chiropractic’s lack of sound evidence-base, get lost!
  2. The new NICE guidelines were a real blow to us, but we now try to spin them such that consumers don’t realise that chiropractic is no longer recommended as a first-line therapy.
  3. In any case, other professions also occasionally use those questionable spinal manipulations (and they are even more numerous). So, any criticism  of spinal manipulation  should not be directed at us but at physios and osteopaths.
  4. We know, of course, that chiropractors treat lots of non-spinal conditions (asthma, bed-wetting, infant colic etc.). Yet we try our very best to hide this fact and pretend that we are all focussed on back pain. This avoids admitting that, for all such conditions, the evidence suggests our manipulations to be worst than useless.

Personally, I find the RCC-strategy very understandable; after all, the RCC has to try to save the bacon for UK chiropractors. Yet, it is nevertheless an attempt at misleading the public about what is really going on. And even, if someone is sufficiently naïve to swallow this spin, one question emerges loud and clear: if chiropractic is just a limited version of physiotherapy, why don’t we simply use physiotherapists for back problems and forget about chiropractors?

(In case the RCC change their mind and want to listen to me elaborating on these themes, my offer for a free lecture still stands!)

Needle acupuncture in small children is controversial, not least because the evidence that it works is negative or weak, and because small children are unable to consent to the treatment. Yet it is recommended by some acupuncturists for infant colic. This, of course, begs the questions:

  1. Does the best evidence tell us that acupuncture is effective for infant colic?
  2. Are acupuncturists who recommend acupuncture for this condition responsible and ethical?

This systematic review and a blinding-test validation based on individual patient data from randomised controlled trials was aimed to assess its efficacy for treating infantile colic. Primary end-points were crying time at mid-treatment, at the end of treatment and at a 1-month follow-up. A 30-min mean difference (MD) in crying time between acupuncture and control was predefined as a clinically important difference. Pearson’s chi-squared test and the James and Bang indices were used to test the success of blinding of the outcome assessors [parents].

The investigators included three randomised controlled trials with data from 307 participants. Only one of the included trials obtained a successful blinding of the outcome assessors in both the acupuncture and control groups. The MD in crying time between acupuncture intervention and no acupuncture control was -24.9 min at mid-treatment, -11.4 min at the end of treatment and -11.8 min at the 4-week follow-up. The heterogeneity was negligible in all analyses. The statistically significant result at mid-treatment was lost when excluding the apparently unblinded study in a sensitivity analysis: MD -13.8 min. The registration of crying during treatment suggested more crying during acupuncture.

The authors concluded that percutaneous needle acupuncture treatments should not be recommended for infantile colic on a general basis.

The authors also provide this further comment: “Our blinding test validated IPD meta-analysis of minimal acupuncture treatments of infantile colic did not show clinically relevant effects in pain reduction as estimated by differences in crying time between needle acupuncture intervention and no acupuncture control. Analyses indicated that acupuncture treatment induced crying in many of the children. Caution should therefore be exercised in recommending potentially painful treatments with uncertain efficacy in infants. The studies are few, the analysis is made on small samples of individuals, and conclusions should be considered in this context. With this limitation in mind, our findings do not support the idea that percutaneous needle acupuncture should be recommended for treatment of infantile colic on a general basis.”

So, returning to the two questions that I listed above – what are the answers?

I think they must be:

  1. No.
  2. No.

Can I tempt you to run a little (hopefully instructive) thought-experiment with you? It is quite simple: I will tell you about the design of a clinical trial, and you will tell me what the likely outcome of this study would be.

Are you game?

Here we go:

_____________________________________________________________________________

Imagine we conduct a trial of acupuncture for persistent pain (any type of pain really). We want to find out whether acupuncture is more than a placebo when it comes to pain-control. Of course, we want our trial to look as rigorous as possible. So, we design it as a randomised, sham-controlled, partially-blinded study. To be really ‘cutting edge’, our study will not have two but three parallel groups:

1. Standard needle acupuncture administered according to a protocol recommended by a team of expert acupuncturists.

2. Minimally invasive sham-acupuncture employing shallow needle insertion using short needles at non-acupuncture points. Patients in groups 1 and 2 are blinded, i. e. they are not supposed to know whether they receive the sham or real acupuncture.

3. No treatment at all.

We apply the treatments for a sufficiently long time, say 12 weeks. Before we start, after 6 and 12 weeks, we measure our patients’ pain with a validated method. We use sound statistical methods to compare the outcomes between the three groups.

WHAT DO YOU THINK THE RESULT WOULD BE?

You are not sure?

Well, let me give you some hints:

Group 3 is not going to do very well; not only do they receive no therapy at all, but they are also disappointed to have ended up in this group as they joined the study in the hope to get acupuncture. Therefore, they will (claim to) feel a lot of pain.

Group 2 will be pleased to receive some treatment. However, during the course of the 6 weeks, they will get more and more suspicious. As they were told during the process of obtaining informed consent that the trial entails treating some patients with a sham/placebo, they are bound to ask themselves whether they ended up in this group. They will see the short needles and the shallow needling, and a percentage of patients from this group will doubtlessly suspect that they are getting the sham treatment. The doubters will not show a powerful placebo response. Therefore, the average pain scores in this group will decrease – but only a little.

Group 1 will also be pleased to receive some treatment. As the therapists cannot be blinded, they will do their best to meet the high expectations of their patients. Consequently, they will benefit fully from the placebo effect of the intervention and the pain score of this group will decrease significantly.

So, now we can surely predict the most likely result of this trial without even conducting it. Assuming that acupuncture is a placebo-therapy, as many people do, we now see that group 3 will suffer the most pain. In comparison, groups 1 and 2 will show better outcomes.

Of course, the main question is, how do groups 1 and 2 compare to each other? After all, we designed our sham-controlled trial in order to answer exactly this issue: is acupuncture more than a placebo? As pointed out above, some patients in group 2 would have become suspicious and therefore would not have experienced the full placebo-response. This means that, provided the sample sizes are sufficiently large, there should be a significant difference between these two groups favouring real acupuncture over sham. In other words, our trial will conclude that acupuncture is better than placebo, even if acupuncture is a placebo.

THANK YOU FOR DOING THIS THOUGHT EXPERIMENT WITH ME.

Now I can tell you that it has a very real basis. The leading medical journal, JAMA, just published such a study and, to make matters worse, the trial was even sponsored by one of the most prestigious funding agencies: the NIH.

Here is the abstract:

___________________________________________________________________________

Musculoskeletal symptoms are the most common adverse effects of aromatase inhibitors and often result in therapy discontinuation. Small studies suggest that acupuncture may decrease aromatase inhibitor-related joint symptoms.

Objective:

To determine the effect of acupuncture in reducing aromatase inhibitor-related joint pain.

Design, Setting, and Patients:

Randomized clinical trial conducted at 11 academic centers and clinical sites in the United States from March 2012 to February 2017 (final date of follow-up, September 5, 2017). Eligible patients were postmenopausal women with early-stage breast cancer who were taking an aromatase inhibitor and scored at least 3 on the Brief Pain Inventory Worst Pain (BPI-WP) item (score range, 0-10; higher scores indicate greater pain).

Interventions:

Patients were randomized 2:1:1 to the true acupuncture (n = 110), sham acupuncture (n = 59), or waitlist control (n = 57) group. True acupuncture and sham acupuncture protocols consisted of 12 acupuncture sessions over 6 weeks (2 sessions per week), followed by 1 session per week for 6 weeks. The waitlist control group did not receive any intervention. All participants were offered 10 acupuncture sessions to be used between weeks 24 and 52.

Main Outcomes and Measures:

The primary end point was the 6-week BPI-WP score. Mean 6-week BPI-WP scores were compared by study group using linear regression, adjusted for baseline pain and stratification factors (clinically meaningful difference specified as 2 points).

Results:

Among 226 randomized patients (mean [SD] age, 60.7 [8.6] years; 88% white; mean [SD] baseline BPI-WP score, 6.6 [1.5]), 206 (91.1%) completed the trial. From baseline to 6 weeks, the mean observed BPI-WP score decreased by 2.05 points (reduced pain) in the true acupuncture group, by 1.07 points in the sham acupuncture group, and by 0.99 points in the waitlist control group. The adjusted difference for true acupuncture vs sham acupuncture was 0.92 points (95% CI, 0.20-1.65; P = .01) and for true acupuncture vs waitlist control was 0.96 points (95% CI, 0.24-1.67; P = .01). Patients in the true acupuncture group experienced more grade 1 bruising compared with patients in the sham acupuncture group (47% vs 25%; P = .01).

Conclusions and Relevance:

Among postmenopausal women with early-stage breast cancer and aromatase inhibitor-related arthralgias, true acupuncture compared with sham acupuncture or with waitlist control resulted in a statistically significant reduction in joint pain at 6 weeks, although the observed improvement was of uncertain clinical importance.

__________________________________________________________________________

Do you see how easy it is to deceive (almost) everyone with a trial that looks rigorous to (almost) everyone?

My lesson from all this is as follows: whether consciously or unconsciously, SCAM-researchers often build into their trials more or less well-hidden little loopholes that ensure they generate a positive outcome. Thus even a placebo can appear to be effective. They are true masters of producing false-positive findings which later become part of a meta-analysis which is, of course, equally false-positive. It is a great shame, in my view, that even top journals (in the above case JAMA) and prestigious funders (in the above case the NIH) cannot (or want not to?) see behind this type of trickery.

Having yesterday been to a ‘Skeptics in the Pub’ event on MEDITATION in Cambridge (my home town since last year) I had to think about the subject quite a bit. As I have hardly covered this topic on my blog, I am today trying to briefly summarise my view on it.

The first thing that strikes me when looking at the evidence on meditation is that it is highly confusing. There seem to be:

  • a lack of clear definitions,
  • hundreds of studies, most of which are of poor or even very poor quality,
  • lots of people with ’emotional baggage’,
  • plenty of strange links to cults and religions,
  • dozens of different meditation methods and regimen,
  • unbelievable claims by enthusiasts,
  • lots of weirdly enthusiastic followers.

What was confirmed yesterday is the fact that, once we look at the reliable medical evidence, we are bound to find that the health claims of various meditation techniques are hugely exaggerated. There is almost no strong evidence to suggest that meditation does affect any condition. The small effects that do emerge from some meta-analyses could easily be due to residual bias and confounding; it is not possible to rigorously control for placebo effects in clinical trials of meditation.

Another thing that came out clearly yesterday is the fact that meditation might not be as risk-free as it is usually presented. Several cases of psychoses after meditation are on record; some of these are both severe and log-lasting. How often do they happen? Nobody knows! Like with most alternative therapies, there is no reporting system in place that could possibly give us anything like a reliable answer.

For me, however, the biggest danger with (certain forms of) meditation is not the risk of psychosis. It is the risk of getting sucked into a cult that then takes over the victim and more or less destroys his or her personality. I have seen this several times, and it is a truly frightening phenomenon.

In our now 10-year-old book THE DESKTOP GUIDE TO COMPLEMENTARY AND ALTERNATIVE MEDICINE, we included a chapter on meditation. It concluded that “meditation appears to be safe for most people and those with sufficient motivation to practise regularly will probably find a relaxing experience. Evidence for effectiveness in any indication is week.” Even today, this is not far off the mark, I think. If I had to re-write it now, I would perhaps mention the potential for harm and also add that, as a therapy, the risk/benefit balance of meditation fails to be convincingly positive. 

PS

I highly recommend ‘Skeptics in the Pub’ events to anyone who likes stimulating talks and critical thinking.

On this blog, we constantly discuss the shortcomings of clinical trials of (and other research into) alternative medicine. Yet, there can be no question that research into conventional medicine is often unreliable as well.

What might be the main reasons for this lamentable fact?

A recent BMJ article discussed 5 prominent reasons:

Firstly, much research fails to address questions that matter. For example, new drugs are tested against placebo rather than against usual treatments. Or the question may already have been answered, but the researchers haven’t undertaken a systematic review that would have told them the research was not needed. Or the research may use outcomes, perhaps surrogate measures, that are not useful.

Secondly, the methods of the studies may be inadequate. Many studies are too small, and more than half fail to deal adequately with bias. Studies are not replicated, and when people have tried to replicate studies they find that most do not have reproducible results.

Thirdly, research is not efficiently regulated and managed. Quality assurance systems fail to pick up the flaws in the research proposals. Or the bureaucracy involved in having research funded and approved may encourage researchers to conduct studies that are too small or too short term.

Fourthly, the research that is completed is not made fully accessible. Half of studies are never published at all, and there is a bias in what is published, meaning that treatments may seem to be more effective and safer than they actually are. Then not all outcome measures are reported, again with a bias towards those are positive.

Fifthly, published reports of research are often biased and unusable. In trials about a third of interventions are inadequately described meaning they cannot be implemented. Half of study outcomes are not reported.

END OF QUOTE

Apparently, these 5 issues are the reason why 85% of biomedical research is being wasted.

That is in CONVENTIONAL medicine, of course.

What about alternative medicine?

There is no question in my mind that the percentage figure must be even higher here. But do the same reasons apply? Let’s go through them again:

  1. Much research fails to address questions that matter. That is certainly true for alternative medicine – just think of the plethora of utterly useless surveys that are being published.
  2. The methods of the studies may be inadequate. Also true, as we have seen hundreds of time on this blog. Some of the most prevalent flaws include in my experience small sample sizes, lack of adequate controls (e.g. A+B vs B design) and misleading conclusions.
  3. Research is not efficiently regulated and managed. True, but probably not a specific feature of alternative medicine research.
  4. Research that is completed is not made fully accessible. most likely true but, due to lack of information and transparency, impossible to judge.
  5. Published reports of research are often biased and unusable. This is unquestionably a prominent feature of alternative medicine research.

All of this seems to indicate that the problems are very similar – similar but much more profound in the realm of alternative medicine, I’d say based on many years of experience (yes, what follows is opinion and not evidence because the latter is hardly available).

The thing is that, like almost any other job, research needs knowledge, skills, training, experience, integrity and impartiality to do it properly. It simply cannot be done well without such qualities. In alternative medicine, we do not have many individuals who have all or even most of these qualities. Instead, we have people who often are evangelic believers in alternative medicine, want to further their field by doing some research and therefore acquire a thin veneer of scientific expertise.

In my 25 years of experience in this area, I have not often seen researchers who knew that research is for testing hypotheses and not for trying to prove one’s hunches to be correct. In my own team, those who were the most enthusiastic about a particular therapy (and were thus seen as experts in its clinical application), were often the lousiest researchers who had the most difficulties coping with the scientific approach.

For me, this continues to be THE problem in alternative medicine research. The investigators – and some of them are now sufficiently skilled to bluff us to believe they are serious scientists – essentially start on the wrong foot. Because they never were properly trained and educated, they fail to appreciate how research proceeds. They hardly know how to properly establish a hypothesis, and – most crucially – they don’t know that, once that is done, you ought to conduct investigation after investigation to show that your hypothesis is incorrect. Only once all reasonable attempts to disprove it have failed, can your hypothesis be considered correct. These multiple attempts of disproving go entirely against the grain of an enthusiast who has plenty of emotional baggage and therefore cannot bring him/herself to honestly attempt to disprove his/her beloved hypothesis.

The plainly visible result of this situation is the fact that we have dozens of alternative medicine researchers who never publish a negative finding related to their pet therapy (some of them were admitted to what I call my HALL OF FAME on this blog, in case you want to verify this statement). And the lamentable consequence of all this is the fast-growing mountain of dangerously misleading (but often seemingly robust) articles about alternative treatments polluting Medline and other databases.

An article entitled “Homeopathy in the Age of Antimicrobial Resistance: Is It a Viable Treatment for Upper Respiratory Tract Infections?” cannot possibly be ignored on this blog, particularly if published in the amazing journal ‘Homeopathy‘. The title does not bode well, in my view – but let’s see. Below, I copy the abstract of the paper without any changes; all I have done is include a few numbers in brackets; they refer to my comments that follow.

START OF ABSTRACT

Acute upper respiratory tract infections (URTIs) and their complications are the most frequent cause of antibiotic prescribing in primary care. With multi-resistant organisms proliferating, appropriate alternative treatments to these conditions are urgently required. Homeopathy presents one solution (1); however, there are many methods of homeopathic prescribing. This review of the literature considers firstly whether homeopathy offers a viable alternative therapeutic solution for acute URTIs (2) and their complications, and secondly how such homeopathic intervention might take place.

METHOD:

Critical review of post 1994 (3) clinical studies featuring homeopathic treatment of acute URTIs and their complications. Study design, treatment intervention, cohort group, measurement and outcome were considered. Discussion focused on the extent to which homeopathy is used to treat URTIs, rate of improvement and tolerability of the treatment, complications of URTIs, prophylactic and long-term effects, and the use of combination versus single homeopathic remedies.

RESULTS:

Multiple peer-reviewed (4) studies were found in which homeopathy had been used to treat URTIs and associated symptoms (cough, pharyngitis, tonsillitis, otitis media, acute sinusitis, etc.). Nine randomised controlled trials (RCTs) and 8 observational/cohort studies were analysed, 7 of which were paediatric studies. Seven RCTs used combination remedies with multiple constituents. Results for homeopathy treatment were positive overall (5), with faster resolution, reduced use of antibiotics and possible prophylactic and longer-term benefits.

CONCLUSIONS:

Variations in size, location, cohort and outcome measures make comparisons and generalisations concerning homeopathic clinical trials for URTIs problematic (6). Nevertheless, study findings suggest at least equivalence between homeopathy and conventional treatment for uncomplicated URTI cases (7), with fewer adverse events and potentially broader therapeutic outcomes. The use of non-individualised homeopathic compounds tailored for the paediatric population merits further investigation, including through cohort studies (8). In the light of antimicrobial resistance, homeopathy offers alternative strategies for minor infections and possible prevention of recurring URTIs (9).

END OF ABSTRACT

And here are my comments:

  1. This sounds as though the author already knew the conclusion of her review before she even started.
  2. Did she not just state that homeopathy is a solution?
  3. This is most unusual; why were pre-1994 articles not considered?
  4. This too is unusual; that a study is peer-reviewed is not really possible to affirm, one must take the journal’s word for it. Yet we know that peer-review is farcical in the realm of alternative medicine (see also below). Therefore, this is an odd inclusion criterion to mention in an abstract.
  5. An overall positive result obtained by including uncontrolled observational and cohort studies lacking a control group is meaningless. There is also no assessment of the quality of the RCTs; after a quick glance, I get the impression that the methodologically sound studies do not show homeopathy to be superior to placebo.
  6. Reviewers need to disentangle these complicating factors and arrive at a conclusion. This is almost invariably problematic, but it is the reviewer’s job.
  7. What might be the conventional treatment of uncomplicated URTI?
  8. Why on earth cohort studies? They tell us nothing about equivalence, efficacy etc.
  9. To reach that conclusion seems to have been the aim of this review (see my point number 1). If I am not mistaken, antibiotics are not indicated in the vast majority of cases of uncomplicated URTI. This means that the true alternative in the light of antimicrobial resistance is to not prescribe antibiotics and treat the patient symptomatically. No need at all for homeopathic placebos, and no need for wishful thinking reviews!

In the paper, the author explains her liking of uncontrolled studies: Non-RCTs and patient reported surveys are considered by some to be inferior forms of research evidence, but are important adjuncts to RCTs that can measure key markers such as patient satisfaction, quality of life and functional health. Observational studies such as clinical outcome studies and case reports, monitoring the effects of homeopathy in real-life clinical settings, are a helpful adjunct to RCTs and more closely reflect real-life experiences of patients and physicians than RCTs, and are therefore considered in this study. I would counter that this is not an issue of inferiority but one that depends on the research question; if the research question relates to efficacy/effectiveness, uncontrolled data are next to useless.

She also makes fairly categorical statements in the conclusion section of the paper about the effectiveness of homeopathy: [the] combined evidence from these and other studies suggests that homeopathic treatment can exert biological effects with fewer adverse events and broader therapeutic opportunities than conventional medicine in the treatment of URTIs. Given the cost implications of treating URTIs and their complications in children, and the relative absence of effective alternatives without potential side effects, the use of non-individualised homeopathic compounds tailored for the paediatric population merits further investigation, including through large-scale cohort studies…  the most important evidence still arises from practical clinical experience and from the successful treatment of millions of patients. I would counter that none of these conclusions are warranted by the data presented.

From reading the paper, I get the impression that the author (the paper provides no information about her conflicts of interest, nor funding source) is a novice to conducting reviews (even though the author is a senior lecturer, the paper reads more like a poorly organised essay than a scientific review). I am therefore hesitant to criticise her – but I do nevertheless find the fact deplorable that her article passed the peer-review process of ‘Homeopathy‘ and was published in a seemingly respectable journal. If anything, articles of this nature are counter-productive for everyone concerned; they certainly do not further effective patient care, and they give homeopathy-research a worse name than it already has.

Why  do most alternative practitioners  show such dogged determination not to change their view of the efficacy of their therapy, even if  good evidence shows that it is a placebo? This is the question that I have been pondering for some time. I have seen many doctors change their mind about this or that treatment in the light of new evidence. In fact, I have not seen one who has not done so at some stage. Yet I have never seen an alternative therapist change his/her mind about his/her alternative therapy. Why is that?

You might say that the answers are obvious:

  • because they have heavily invested in their therapy, both emotionally and financially;
  • because their therapy has ‘stood the test of time’;
  • because they believe what they were taught;
  • because they are deluded, not very bright, etc.;
  • because they need to earn a living.

All of these reasons may apply. But do they really tell the whole story? While contemplating about this question, I thought of something that had previously not been entirely clear to me: they simply KNOW that the evidence MUST be wrong.

Let me try to explain.

Consider an acupuncturist (I could have chosen almost any other type of alternative practitioner) who has many years of experience. He has grown to be a well-respected expert in the world of acupuncture. He sits on various committees and has advised important institutions. He knows the literature and has treated thousands of patients.

This experience has taught him one thing for sure: his patients do benefit from his treatment. He has seen it happening too many times; it cannot be a coincidence. Acupuncture works, no question about it.

And this is also what the studies tell him. Even the most sceptical scientist cannot deny the fact that patients do get better after acupuncture. So, what is the problem?

The problem is that sceptics say that this is due to a placebo effect, and many studies seem to confirm this to be true. Yet, our acupuncturist completely dismisses the placebo explanation.

Why?

  • Because he has heavily invested in their therapy? Perhaps.
  • Because acupuncture has ‘stood the test of time’? Perhaps.
  • Because he believes what he has been taught? Perhaps.
  • Because he is deluded, not very bright, etc.? Perhaps.
  • Because he needs to earn a living? Perhaps.

But there is something else.

He has only ever treated his patients with acupuncture. He has therefore no experience of real medicine, or other therapeutic options. He has no perspective. Therefore, he does not know that patients often get better, even if they receive an ineffective treatment, even if they receive no treatment, and even if they receive a harmful treatment. Every improvement he notes in his patients, he relates to his acupuncture. Our acupuncturist never had the opportunity to learn to doubt cause and effect in his clinical routine. He never had to question the benefits of acupuncture. He never had to select from a pool of therapies the optimal one, because he only ever used acupuncture.

It is this lack of experience that never led him to think critically about acupuncture. He is in a similar situation as physicians were 200 years ago; they only (mainly) had blood-letting, and because some patients improved with it, they had no reason to doubt it. He only ever saw his successes (not that all his patients improved, but those who did not, did not return). He simply KNOWS that acupuncture works, because his own, very limited experience never forced him to consider anything else. And because he KNOWS, the evidence that does not agree with his knowledge MUST be wrong.

I am of course exaggerating and simplifying in order to make a point. And please don’t get me wrong.

I am not saying that doctors cannot be stubborn. And I am not saying that all alternative practitioners have such limited experience and are unable to change their mind in the light of new evidence. However, I am trying to say that many alternative practitioners have a limited perspective and therefore find it impossible to be critical about their own practice.

If I am right, there would be an easy (and entirely alternative) cure to remedy this situation. We should sent our acupuncturist to a homeopath (or any other alternative practitioner whose practice he assumes to be entirely bogus) and ask him to watch what kind of therapeutic success the homeopath is generating. The acupuncturist would soon see that it is very similar to his own. He would then have the choice to agree that highly diluted homeopathic remedies are effective in curing illness, or that the homeopath relies on the same phenomenon as his own practice: placebo.

Sadly, this is not going to happen, is it?

 

1 2 3 40
Recent Comments

Note that comments can be edited for up to five minutes after they are first submitted.


Click here for a comprehensive list of recent comments.

Categories