MD, PhD, FMedSci, FRSB, FRCP, FRCPEd.

meta-analysis

1 2 3 5

I have often cautioned my readers about the ‘evidence’ supporting acupuncture (and other alternative therapies). Rightly so, I think. Here is yet another warning.

This systematic review assessed the clinical effectiveness of acupuncture in the treatment of postpartum depression (PPD). Nine trials involving 653 women were selected. A meta-analysis demonstrated that the acupuncture group had a significantly greater overall effective rate compared with the control group. Moreover, acupuncture significantly increased oestradiol levels compared with the control group. Regarding the HAMD and EPDS scores, no difference was found between the two groups. The Chinese authors concluded that acupuncture appears to be effective for postpartum depression with respect to certain outcomes. However, the evidence thus far is inconclusive. Further high-quality RCTs following standardised guidelines with a low risk of bias are needed to confirm the effectiveness of acupuncture for postpartum depression.

What a conclusion!

What a review!

What a journal!

What evidence!

Let’s start with the conclusion: if the authors feel that the evidence is ‘inconclusive’, why do they state that ‘acupuncture appears to be effective for postpartum depression‘. To me this does simply not make sense!

Such oddities are abundant in the review. The abstract does not mention the fact that all trials were from China (published in Chinese which means that people who cannot read Chinese are unable to check any of the reported findings), and their majority was of very poor quality – two good reasons to discard the lot without further ado and conclude that there is no reliable evidence at all.

The authors also tell us very little about the treatments used in the control groups. In the paper, they state that “the control group needed to have received a placebo or any type of herb, drug and psychological intervention”. But was acupuncture better than all or any of these treatments? I could not find sufficient data in the paper to answer this question.

Moreover, only three trials seem to have bothered to mention adverse effects. Thus the majority of the studies were in breach of research ethics. No mention is made of this in the discussion.

In the paper, the authors re-state that “this meta-analysis showed that the acupuncture group had a significantly greater overall effective rate compared with the control group. Moreover, acupuncture significantly increased oestradiol levels compared with the control group.” This is, I think, highly misleading (see above).

Finally, let’s have a quick look at the journal ‘Acupuncture in Medicine’ (AiM). Even though it is published by the BMJ group (the reason for this phenomenon can be found here: “AiM is owned by the British Medical Acupuncture Society and published by BMJ“; this means that all BMAS-members automatically receive the journal which thus is a resounding commercial success), it is little more than a cult-newsletter. The editorial board is full of acupuncture enthusiasts, and the journal hardly ever publishes anything that is remotely critical of the wonderous myths of acupuncture.

My conclusion considering all this is as follows: we ought to be very careful before accepting any ‘evidence’ that is currently being published about the benefits of acupuncture, even if it superficially looks ok. More often than not, it turns out to be profoundly misleading, utterly useless and potentially harmful pseudo-evidence.


Reference

Acupunct Med. 2018 Jun 15. pii: acupmed-2017-011530. doi: 10.1136/acupmed-2017-011530. [Epub ahead of print]

Effectiveness of acupuncture in postpartum depression: a systematic review and meta-analysis.

Li S, Zhong W, Peng W, Jiang G.

Most people probably think of acupuncture as being used mainly as a therapy for pain control. But acupuncture is currently being promoted (and has traditionally been used) for all sorts of conditions. One of them is stroke. It is said to speed up recovery and even improve survival rates after such an event. There are plenty of studies on this subject, but their results are far from uniform. What is needed in this situation, is a rigorous summary of the evidence.

The authors of this Cochrane review wanted to assess whether acupuncture could reduce the proportion of people suffering death or dependency after acute ischaemic or haemorrhagic stroke. They included all randomized clinical trials (RCTs) of acupuncture started within 30 days after stroke onset. Acupuncture had to be compared with placebo or sham acupuncture or open control (no placebo) in people with acute ischaemic or haemorrhagic stroke, or both. Comparisons were made versus (1) all controls (open control or sham acupuncture), and (2) sham acupuncture controls.

The investigators included 33 RCTs with 3946 participants. Outcome data were available for up to 22 trials (2865 participants) that compared acupuncture with any control (open control or sham acupuncture) but for only 6 trials (668 participants) comparing acupuncture with sham acupuncture.

When compared with any control (11 trials with 1582 participants), findings of lower odds of death or dependency at the end of follow-up and over the long term (≥ three months) in the acupuncture group were uncertain and were not confirmed by trials comparing acupuncture with sham acupuncture. In trials comparing acupuncture with any control, findings that acupuncture was associated with increases in the global neurological deficit score and in the motor function score were uncertain. These findings were not confirmed in trials comparing acupuncture with sham acupuncture.Trials comparing acupuncture with any control showed little or no difference in death or institutional care or death at the end of follow-up.The incidence of adverse events (eg, pain, dizziness, faint) in the acupuncture arms of open and sham control trials was 6.2% (64/1037 participants), and 1.4% of these patients (14/1037 participants) discontinued acupuncture. When acupuncture was compared with sham acupuncture, findings for adverse events were uncertain.

The authors concluded that this updated review indicates that apparently improved outcomes with acupuncture in acute stroke are confounded by the risk of bias related to use of open controls. Adverse events related to acupuncture were reported to be minor and usually did not result in stopping treatment. Future studies are needed to confirm or refute any effects of acupuncture in acute stroke. Trials should clearly report the method of randomization, concealment of allocation, and whether blinding of participants, personnel, and outcome assessors was achieved, while paying close attention to the effects of acupuncture on long-term functional outcomes.

This Cochrane review seems to be thorough, but it is badly written (Cochrane reviewers: please don’t let this become the norm!). It contains some interesting facts. The majority of the studies came from China. This review confirmed the often very poor methodological quality of acupuncture trials which I have frequently mentioned before.

In particular, the RCTs originating from China were amongst those that most overtly lacked rigor, also a fact that has been discussed regularly on this blog.

For me, by far the most important finding of this review is that studies which at least partly control for placebo effects fail to show positive results. Depending on where you stand in the never-ending debate about acupuncture, this could lead to two dramatically different conclusions:

  1. If you are a believer in or earn your living from acupuncture, you might say that these results suggest that the trials were in some way insufficient and therefore they produced false-negative results.
  2. If you are a more reasonable observer, you might feel that these results show that acupuncture (for acute stroke) is a placebo therapy.

Regardless to which camp you belong, one thing seems to be certain: acupuncture for stroke (and other indications) is not supported by sound evidence. And that means, I think, that it is not responsible to use it in routine care.

My previous post praised the validity and trustworthiness of Cochrane reviews. This post continues in the same line.

Like osteoarthritis, acute stroke has been a condition for which acupuncture-fans prided themselves of being able to produce fairly good evidence. A Cochrane review of 2005, however, was inconclusive and concluded that the number of patients is too small to be certain whether acupuncture is effective for treatment of acute ischaemic or haemorrhagic stroke. Larger, methodologically-sound trials are required.

So, 13 years later, we do have more evidence, and it would be interesting to know what the best evidence tells us today. This new review will tell us because it is the update of the previous Cochrane Review originally published in 2005.

The authors sought randomized clinical trials (RCTs) of acupuncture started within 30 days from stroke onset compared with placebo or sham acupuncture or open control (no placebo) in people with acute ischemic or haemorrhagic stroke, or both. Needling into the skin was required for acupuncture. Comparisons were made versus (1) all controls (open control or sham acupuncture), and (2) sham acupuncture controls.

Two review authors applied the inclusion criteria, assessed trial quality and risk of bias, and extracted data independently. They contacted study authors to ask for missing data and assessed the quality of the evidence by using the GRADE approach. The primary outcome was defined as death or dependency at the end of follow-up.

In total, 33 RCTs with 3946 participants were included. Twenty new trials with 2780 participants had been completed since the previous review. Outcome data were available for up to 22 trials (2865 participants) that compared acupuncture with any control (open control or sham acupuncture) but for only six trials (668 participants) that compared acupuncture with sham acupuncture control. The authors downgraded the evidence to low or very low quality because of risk of bias in included studies, inconsistency in the acupuncture intervention and outcome measures, and imprecision in effect estimates.

When compared with any control (11 trials with 1582 participants), findings of lower odds of death or dependency at the end of follow-up and over the long term (≥ three months) in the acupuncture group were uncertain (odds ratio [OR] 0.61, 95% confidence interval [CI] 0.46 to 0.79; very low-quality evidence; and OR 0.67, 95% CI 0.53 to 0.85; eight trials with 1436 participants; very low-quality evidence, respectively) and were not confirmed by trials comparing acupuncture with sham acupuncture (OR 0.71, 95% CI 0.43 to 1.18; low-quality evidence; and OR 0.67, 95% CI 0.40 to 1.12; low-quality evidence, respectively).In trials comparing acupuncture with any control, findings that acupuncture was associated with increases in the global neurological deficit score and in the motor function score were uncertain (standardized mean difference [SMD] 0.84, 95% CI 0.36 to 1.32; 12 trials with 1086 participants; very low-quality evidence; and SMD 1.08, 95% CI 0.45 to 1.71; 11 trials with 895 participants; very low-quality evidence).

These findings were not confirmed in trials comparing acupuncture with sham acupuncture (SMD 0.01, 95% CI -0.55 to 0.57; low-quality evidence; and SMD 0.10, 95% CI -0.38 to 0.17; low-quality evidence, respectively).Trials comparing acupuncture with any control reported little or no difference in death or institutional care at the end of follow-up (OR 0.78, 95% CI 0.54 to 1.12; five trials with 1120 participants; low-quality evidence), death within the first two weeks (OR 0.91, 95% CI 0.33 to 2.55; 18 trials with 1612 participants; low-quality evidence), or death at the end of follow-up (OR 1.08, 95% CI 0.74 to 1.58; 22 trials with 2865 participants; low-quality evidence).

The incidence of adverse events (eg, pain, dizziness, fainting) in the acupuncture arms of open and sham control trials was 6.2% (64/1037 participants), and 1.4% of these (14/1037 participants) discontinued acupuncture. When acupuncture was compared with sham acupuncture, findings for adverse events were uncertain (OR 0.58, 95% CI 0.29 to 1.16; five trials with 576 participants; low-quality evidence).

The authors concluded that this updated review indicates that apparently improved outcomes with acupuncture in acute stroke are confounded by the risk of bias related to use of open controls. Adverse events related to acupuncture were reported to be minor and usually did not result in stopping treatment. Future studies are needed to confirm or refute any effects of acupuncture in acute stroke. Trials should clearly report the method of randomization, concealment of allocation, and whether blinding of participants, personnel, and outcome assessors was achieved, while paying close attention to the effects of acupuncture on long-term functional outcomes.

These cautious conclusions might be explained by the fact that Chinese researchers are reluctant to state anything overtly negative about any TCM therapy. Recently, one expert who spoke out was even imprisoned for criticising a TCM product! But in truth, this review really shows that acupuncture has no convincing effect in acute stroke.

And for me, this conclusion is fascinating. I have been involved in acupuncture/stroke research since the early 1990s.

Our RCT produced a resounding negative result concluding that acupuncture is not superior to sham treatment for recovery in activities of daily living and health-related quality of life after stroke, although there may be a limited effect on leg function in more severely affected patients.

Our 1996 systematic review concluded that the evidence that acupuncture is a useful adjunct for stroke rehabilitation is encouraging but not compelling.

By 2001, more data had become available but the conclusion became even less encouraging: there is no compelling evidence to show that acupuncture is effective in stroke rehabilitation.

Finally, by 2010, there were 10 RCT and we were able to do a meta-analysis of the data. We concluded that our meta-analyses of data from rigorous randomized sham-controlled trials did not show a positive effect of acupuncture as a treatment for functional recovery after stroke.

Yes, my reviews were on slightly different research questions. Yet, they do reveal how a critical assessment of the slowly emerging evidence had to arrive at more and more negative conclusions about the role of acupuncture in the management of stroke patients. For a long time, this message was in stark contrast to what acupuncture-fans were claiming. I wonder whether they will now finally change their mind.

An announcement (it’s in German, I’m afraid) proudly declaring that ‘homeopathy fulfils the criteria of evidence-based medicine‘ caught my attention.

Here is the story:

In 2016, Dr. Melanie Wölk, did a ‘Master of Science’* at the ‘Donau University’ in Krems, Austria investigating the question whether homeopathy follows the rules of evidence-based medicine (EBM). She arrived at the conclusion that YES, IT DOES! This pleased the leading Austrian manufacturer of homeopathics (Dr Peithner) so much and so durably that, on 23 March 2018, he gave her a ‘scientific’ award (the annual Peithner award) for her ‘research’.

So far so good.

Her paper is unpublished, or at least not available on Medline; therefore, I am unable to evaluate it directly. All I know about it from the announcement is that she did her ‘research at the ‘Zentrum für Traditionelle Chinesische Medizin und Komplementärmedizin‘ of the said university. A quick Medline search revealed that this unit has never published anything, not a single paper, it seems! Disappointed I search for Dr. Christine Schauhuber, the leader of the unit; and again I find no Medline-listed publications in her name. My interim conclusion is thus that this institution might not be at the cutting edge of science.

But what do we know about Dr. Melanie Wölk’s award-winning master thesis *?

The announcement tells us that she investigated all RCTs published between 2010 and 2016. In addition, she evaluated:

On that basis, she arrived at her positive verdict – not just tentatively, but without doubt (“Das Ergebnis steht fest”).

Dr Peithner, the owner of the company and awarder of the prize, was quoted stating that this is a very important piece of work for homeopathy; it shows yet again what we see in our daily routine, namely that homeopathics are effective. Wölk’s investigation demonstrates furthermore that high-quality trials of homeopathy do exist, and that it is time to end the witch-hunt aimed at discrediting an effective therapy. Conventional medicine and homeopathy ought to finally work hand in hand – for the benefit of our patients. (“Für die Homöopathie ist das eine sehr wichtige Arbeit, die wieder zeigt, was wir in der ärztlichen Praxis täglich erleben, nämlich dass homöopathische Arzneimittel wirken. Wölks Untersuchung zeigt weiters deutlich, dass es sehr wohl hochqualitative Homöopathie-Studien gibt und es an der Zeit ist, die Hexenjagd zu beenden, mit der eine wirksame medizinische Therapie diskreditiert werden soll. Konventionelle Medizin und Homöopathie sollten endlich Hand in Hand arbeiten – zum Wohle der Patientinnen und Patienten.”)

I do hope that Dr Wölk uses the prize money (by no means a fortune; see photo) to buy some time for publishing her work (one of my teachers, all those years ago, used to say ‘unpublished research is no research’) so that we can all benefit from it. Until it becomes available, I should perhaps mention that the description of her methodology (publications between 2010 and 2016 [plus a few other papers that nicely fitted the arguments?]; including one Linde review and not his more recent re-analysis [see above]) does not inspire me to think that Dr Wölk’s research was anywhere near rigorous, systematic or complete. In the same vein, I am tempted to point out that the Swiss report is probably the very last document I would select, if I wanted to generate an objective picture about the value of homeopathy.

Taking all this into account, I conclude that we seem to be dealing here with a

  • pseudo-prize (given by a commercial firm to further its business) for a piece of
  • pseudo-research (the project seems to have been aimed to white-wash homeopathy) into
  • pseudo-medicine (a treatment that has been tested extensively but has not been shown to work beyond placebo).

*Wölk, Melanie: Eminenz oder Evidenz: Die Homöopathie auf dem Prüfstand der Evidence based Medicine. Masterarbeit zur Erlangung des akademischen Abschlusses Master of Science im Universitätslehrgang Natural Medicine. Donau-Universität Krems, Department für Gesundheitswissenschaften und Biomedizin. Krems, Mai 2016.

We all know that there is a plethora of interventions for and specialists in low back pain (chiropractors, osteopaths, massage therapists, physiotherapists etc., etc.); and, depending whether you are an optimist or a pessimist, each of these therapies is as good or as useless as the next. Today, a widely-publicised series of articles in the Lancet confirms that none of the current options is optimal:

Almost everyone will have low back pain at some point in their lives. It can affect anyone at any age, and it is increasing—disability due to back pain has risen by more than 50% since 1990. Low back pain is becoming more prevalent in low-income and middle-income countries (LMICs) much more rapidly than in high-income countries. The cause is not always clear, apart from in people with, for example, malignant disease, spinal malformations, or spinal injury. Treatment varies widely around the world, from bed rest, mainly in LMICs, to surgery and the use of dangerous drugs such as opioids, usually in high-income countries.

The Lancet publishes three papers on low back pain, by an international group of authors led by Prof Rachelle Buchbinder, Monash University, Melbourne, Australia, which address the issues around the disorder and call for worldwide recognition of the disability associated with the disorder and the removal of harmful practices. In the first paper, Jan Hartvigsen, Mark Hancock, and colleagues draw our attention to the complexity of the condition and the contributors to it, such as psychological, social, and biophysical factors, and especially to the problems faced by LMICs. In the second paper, Nadine Foster, Christopher Maher, and their colleagues outline recommendations for treatment and the scarcity of research into prevention of low back pain. The last paper is a call for action by Rachelle Buchbinder and her colleagues. They say that persistence of disability associated with low back pain needs to be recognised and that it cannot be separated from social and economic factors and personal and cultural beliefs about back pain.

Overview of interventions endorsed for non-specific low back pain in evidence-based clinical practice guidelines (Danish, US, and UK guidelines)

In this situation, it makes sense, I think, to opt for a treatment (amongst similarly effective/ineffective therapies) that is at least safe, cheap and readily available. This automatically rules out chiropractic, osteopathy and many others. Exercise, however, does come to mind – but what type of exercise?

The aim of this meta-analysis of randomized controlled trials was to gain insight into the effectiveness of walking intervention on pain, disability, and quality of life in patients with chronic low back pain (LBP) at post intervention and follow ups.

Six electronic databases (PubMed, Science Direct, Web of Science, Scopus, PEDro and The Cochrane library) were searched from 1980 to October 2017. Randomized controlled trials (RCTs) in patients with chronic LBP were included, if they compared the effects of walking intervention to non-pharmacological interventions. Pain, disability, and quality of life were the primary health outcomes.

Nine RCTs were suitable for meta-analysis. Data was analysed according to the duration of follow-up (short-term, < 3 months; intermediate-term, between 3 and 12 months; long-term, > 12 months). Low- to moderate-quality evidence suggests that walking intervention in patients with chronic LBP was as effective as other non-pharmacological interventions on pain and disability reduction in both short- and intermediate-term follow ups.

The authors concluded that, unless supplementary high-quality studies provide different evidence, walking, which is easy to perform and highly accessible, can be recommended in the management of chronic LBP to reduce pain and disability.

I know – this will hardly please the legions of therapists who earn their daily bread with pretending their therapy is the best for LBP. But healthcare is clearly not about the welfare of the therapists, it is/should be about patients. And patients should surely welcome this evidence. I know, walking is not always easy for people with severe LBP, but it seems effective and it is safe, free and available to everyone.

My advice to patients is therefore to walk (slowly and cautiously) to the office of their preferred therapist, have a little rest there (say hello to the staff perhaps) and then walk straight back home.

The plethora of dodgy meta-analyses in alternative medicine has been the subject of a recent post – so this one is a mere update of a regular lament.

This new meta-analysis was to evaluate evidence for the effectiveness of acupuncture in the treatment of lumbar disc herniation (LDH). (Call me pedantic, but I prefer meta-analyses that evaluate the evidence FOR AND AGAINST a therapy.) Electronic databases were searched to identify RCTs of acupuncture for LDH, and 30 RCTs involving 3503 participants were included; 29 were published in Chinese and one in English, and all trialists were Chinese.

The results showed that acupuncture had a higher total effective rate than lumbar traction, ibuprofen, diclofenac sodium and meloxicam. Acupuncture was also superior to lumbar traction and diclofenac sodium in terms of pain measured with visual analogue scales (VAS). The total effective rate in 5 trials was greater for acupuncture than for mannitol plus dexamethasone and mecobalamin, ibuprofen plus fugui gutong capsule, loxoprofen, mannitol plus dexamethasone and huoxue zhitong decoction, respectively. Two trials showed a superior effect of acupuncture in VAS scores compared with ibuprofen or mannitol plus dexamethasone, respectively.

The authors from the College of Traditional Chinese Medicine, Jinan University, Guangzhou, Guangdong, China, concluded that acupuncture showed a more favourable effect in the treatment of LDH than lumbar traction, ibuprofen, diclofenac sodium, meloxicam, mannitol plus dexamethasone and mecobalamin, fugui gutong capsule plus ibuprofen, mannitol plus dexamethasone, loxoprofen and huoxue zhitong decoction. However, further rigorously designed, large-scale RCTs are needed to confirm these findings.

Why do I call this meta-analysis ‘dodgy’? I have several reasons, 10 to be exact:

  1. There is no plausible mechanism by which acupuncture might cure LDH.
  2. The types of acupuncture used in these trials was far from uniform and  included manual acupuncture (MA) in 13 studies, electro-acupuncture (EA) in 10 studies, and warm needle acupuncture (WNA) in 7 studies. Arguably, these are different interventions that cannot be lumped together.
  3. The trials were mostly of very poor quality, as depicted in the table above. For instance, 18 studies failed to mention the methods used for randomisation. I have previously shown that some Chinese studies use the terms ‘randomisation’ and ‘RCT’ even in the absence of a control group.
  4. None of the trials made any attempt to control for placebo effects.
  5. None of the trials were conducted against sham acupuncture.
  6. Only 10 studies 10 trials reported dropouts or withdrawals.
  7. Only two trials reported adverse reactions.
  8. None of these shortcomings were critically discussed in the paper.
  9. Despite their affiliation, the authors state that they have no conflicts of interest.
  10. All trials were conducted in China, and, on this blog, we have discussed repeatedly that acupuncture trials from China never report negative results.

And why do I find the journal ‘dodgy’?

Because any journal that publishes such a paper is likely to be sub-standard. In the case of ‘Acupuncture in Medicine’, the official journal of the British Medical Acupuncture Society, I see such appalling articles published far too frequently to believe that the present paper is just a regrettable, one-off mistake. What makes this issue particularly embarrassing is, of course, the fact that the journal belongs to the BMJ group.

… but we never really thought that science publishing was about anything other than money, did we?

What an odd title, you might think.

Systematic reviews are the most reliable evidence we presently have!

Yes, this is my often-voiced and honestly-held opinion but, like any other type of research, systematic reviews can be badly abused; and when this happens, they can seriously mislead us.

new paper by someone who knows more about these issues than most of us, John Ioannidis from Stanford university, should make us think. It aimed at exploring the growth of published systematic reviews and meta‐analyses and at estimating how often they are redundant, misleading, or serving conflicted interests. Ioannidis demonstrated that publication of systematic reviews and meta‐analyses has increased rapidly. In the period January 1, 1986, to December 4, 2015, PubMed tags 266,782 items as “systematic reviews” and 58,611 as “meta‐analyses.” Annual publications between 1991 and 2014 increased 2,728% for systematic reviews and 2,635% for meta‐analyses versus only 153% for all PubMed‐indexed items. Ioannidis believes that probably more systematic reviews of trials than new randomized trials are published annually. Most topics addressed by meta‐analyses of randomized trials have overlapping, redundant meta‐analyses; same‐topic meta‐analyses may exceed 20 sometimes.

Some fields produce massive numbers of meta‐analyses; for example, 185 meta‐analyses of antidepressants for depression were published between 2007 and 2014. These meta‐analyses are often produced either by industry employees or by authors with industry ties and results are aligned with sponsor interests. China has rapidly become the most prolific producer of English‐language, PubMed‐indexed meta‐analyses. The most massive presence of Chinese meta‐analyses is on genetic associations (63% of global production in 2014), where almost all results are misleading since they combine fragmented information from mostly abandoned era of candidate genes. Furthermore, many contracting companies working on evidence synthesis receive industry contracts to produce meta‐analyses, many of which probably remain unpublished. Many other meta‐analyses have serious flaws. Of the remaining, most have weak or insufficient evidence to inform decision making. Few systematic reviews and meta‐analyses are both non‐misleading and useful.

The author concluded that the production of systematic reviews and meta‐analyses has reached epidemic proportions. Possibly, the large majority of produced systematic reviews and meta‐analyses are unnecessary, misleading, and/or conflicted.

Ioannidis makes the following ‘Policy Points’:

  • Currently, there is massive production of unnecessary, misleading, and conflicted systematic reviews and meta‐analyses. Instead of promoting evidence‐based medicine and health care, these instruments often serve mostly as easily produced publishable units or marketing tools.
  • Suboptimal systematic reviews and meta‐analyses can be harmful given the major prestige and influence these types of studies have acquired.
  • The publication of systematic reviews and meta‐analyses should be realigned to remove biases and vested interests and to integrate them better with the primary production of evidence.

Obviously, Ioannidis did not have alternative medicine in mind when he researched and published this article. But he easily could have! Virtually everything he stated in his paper does apply to it. In some areas of alternative medicine, things are even worse than Ioannidis describes.

Take TCM, for instance. I have previously looked at some of the many systematic reviews of TCM that currently flood Medline, based on Chinese studies. This is what I concluded at the time:

Why does that sort of thing frustrate me so much? Because it is utterly meaningless and potentially harmful:

  • I don’t know what treatments the authors are talking about.
  • Even if I managed to dig deeper, I cannot get the information because practically all the primary studies are published in obscure journals in Chinese language.
  • Even if I  did read Chinese, I do not feel motivated to assess the primary studies because we know they are all of very poor quality – too flimsy to bother.
  • Even if they were formally of good quality, I would have my doubts about their reliability; remember: 100% of these trials report positive findings!
  • Most crucially, I am frustrated because conclusions of this nature are deeply misleading and potentially harmful. They give the impression that there might be ‘something in it’, and that it (whatever ‘it’ might be) could be well worth trying. This may give false hope to patients and can send the rest of us on a wild goose chase.

So, to ease the task of future authors of such papers, I decided give them a text for a proper EVIDENCE-BASED conclusion which they can adapt to fit every review. This will save them time and, more importantly perhaps, it will save everyone who might be tempted to read such futile articles the effort to study them in detail. Here is my suggestion for a conclusion soundly based on the evidence, not matter what TCM subject the review is about:

OUR SYSTEMATIC REVIEW HAS SHOWN THAT THERAPY ‘X’ AS A TREATMENT OF CONDITION ‘Y’ IS CURRENTLY NOT SUPPORTED BY SOUND EVIDENCE.

On another occasion, I stated that I am getting very tired of conclusions stating ‘…XY MAY BE EFFECTIVE/HELPFUL/USEFUL/WORTH A TRY…’ It is obvious that the therapy in question MAY be effective, otherwise one would surely not conduct a systematic review. If a review fails to produce good evidence, it is the authors’ ethical, moral and scientific obligation to state this clearly. If they don’t, they simply misuse science for promotion and mislead the public. Strictly speaking, this amounts to scientific misconduct.

In yet another post on the subject of systematic reviews, I wrote that if you have rubbish trials, you can produce a rubbish review and publish it in a rubbish journal (perhaps I should have added ‘rubbish researchers).

And finally this post about a systematic review of acupuncture: it is almost needless to mention that the findings (presented in a host of hardly understandable tables) suggest that acupuncture is of proven or possible effectiveness/efficacy for a very wide array of conditions. It also goes without saying that there is no critical discussion, for instance, of the fact that most of the included evidence originated from China, and that it has been shown over and over again that Chinese acupuncture research never seems to produce negative results.

The main point surely is that the problem of shoddy systematic reviews applies to a depressingly large degree to all areas of alternative medicine, and this is misleading us all.

So, what can be done about it?

My preferred (but sadly unrealistic) solution would be this:

STOP ENTHUSIASTIC AMATEURS FROM PRETENDING TO BE RESEARCHERS!

Research is not fundamentally different from other professional activities; to do it well, one needs adequate training; and doing it badly can cause untold damage.

Few people would argue that Cochrane reviews tend to be the most rigorous, independent and objective assessments of therapeutic interventions we currently have. Therefore, it is relevant to see what they tell us about the value of acupuncture.

Here is a fascinating overview of all Cochrane reviews of acupuncture. It was compiled by the formidable guys at ‘FRIENDS OF SCIENCE-BEASED MEDICINE‘ in Australia. They gave me the permission to publish it here (thanks Loretta!).

 

Considering this collective evidence, it would be hard to dispute the conclusion that there is no convincing evidence that acupuncture is an effective therapy, I believe.

What do you think?

We have discussed the NHMRC report on homeopathy several times – see, for instance, here, here and here. Perhaps understandably, homeopaths have great difficulties accepting its negative findings, and have complained about it ever since it was published. Now, a very detailed and well-researched analysis has become available of both the report and its criticism. Here I take the liberty to copy and (clumsily) translate its conclusions; if you can read German, I highly recommend studying the full document.

START OF MY TRANSLATION

The criticism of the NHMRC review is very voluminous and highlights many different aspects of the background, the methodology, the execution and the unwanted results from a homeopathic perspective. The very engaging discussions in the general public about this document and its flaws are, however, relatively meaningless: the NHMRC arrives at exactly the same conclusions as the employee of the Homeopathic Research Institute (HRI), Mathie, in his reviews of 2014 and 2017.

In both reviews, Mathie evaluated a total of 107 primary studies and found only 2 trials that could be rated as qualitatively good, that is to say constituting reliable evidence. Mathie did upgrade 2 further studies to the category of reliable evidence, however, this was in violation of the procedures proscribed in the study protocol.

The criticism of the NHMRC review was not able to make a single valid rebuttal. No condition could be identified for which homeopathy is clearly superior to placebo. This is all the more important, as Mathie avoided the mistakes that constituted the most prominent alleged criticisms of the NHMRC report.

  • Since Mathie and most of his co-authors are affiliated with organisations of homeopathy, an anti-homeopathy bias can be excluded.
  • Mathie conducted classic reviews and even differentiated between individualised and non-individualised homeopathy.
  • Mathie did not exclude studies below a certain sample size.

Yet, in both reviews, he draws the same conclusion.

In view of the truly independent replications of an employee of the HRI, we can be sure that there are, in fact, no solid proofs for the effectiveness of homeopathy. The claim of a  strong efficacy, equivalent to conventional medicines, that is made by homeopathy’s advocates is therefore not true.

END OF MY TRANSLATION

And here is the original German text:

Die Kritik an dem Review des NHMRC ist sehr umfangreich und beleuchtet sehr viele verschiedene Facetten über das Umfeld, die Methodik und die Durchführung sowie das aus Sicht der Homöopathen unerwünschte Ergebnis selbst. Die in der Öffentlichkeit sehr engagierte Diskussion um diese Arbeit und ihre möglichen Unzulänglichkeiten sind jedoch relativ bedeutungslos: Das NHMRC kommt zu genau dem gleichen Ergebnis wie Mathie als Mitarbeiter des HRI in seinen in 2014 und 2017 veröffentlichten systematischen Reviews:

Insgesamt hat Mathie in beiden Reviews 107 Einzelstudien untersucht und fand nur zwei Studien, die als qualitativ gut („low risk of bias“), also als zuverlässige Evidenz betrachtet werden können. Mathie hat zwar vier weitere Studien zur zuverlässigen Evidenz aufgewertet, was allerdings im Widerspruch zu den üblichen Vorgehensweisen steht und im Studienprotokoll nicht vorgesehen war.

Die Kritik am Review des NHMRC hat keinen einzigen Punkt fundiert widerlegen können. Man konnte keine Indikation finden, bei der sich die Homöopathie als klar über Placebo hinaus wirksam erwiesen hätte. Diese Punkte sind umso bedeutsamer, weil Mathie die am NHMRC hauptsächlich kritisierten Fehler nicht gemacht hat:

  • Als Mitarbeiter des HRI und mit Autoren, die überwiegend für Homöopathie-affine Organisationen arbeiten, ist eine Voreingenommenheit gegen die Homöopathie auszuschließen.
  • Mathie hat klassische Reviews ausgeführt, sogar getrennt zwischen einzelnen Ausprägungen (individualisierte Homöopathie und nicht-individualisierte Homöopathie).
  • Mathie hat keine Größenbeschränkung der Studien berücksichtigt.

Er kommt aber dennoch zweimal zum gleichen Ergebnis wie das NHMRC.

Angesichts der wirklich als unabhängig anzusehenden Bestätigung der Ergebnisse des NHMRC durch einen Mitarbeiter des Homeopathy Research Institute kann man sicher davon ausgehen, dass es tatsächlich keine belastbaren Wirkungsnachweise für die Homöopathie gibt und dass die von ihren Anhängern behauptete starke, der konventionellen Medizin gleichwertige oder gar überlegene Wirksamkeit der Homöopathie nicht gegeben ist.

I do apologise for my clumsy translation and once again encourage those who can to study the detailed original in full.

My conclusion of this (and indeed of virtually all criticism of homeopathy) is that homeopaths are just as unable to accept criticism as an evangelic believer is going to accept any rational argument against his belief. In other words, regardless of how convincing the evidence, homeopaths will always dismiss it – or, to put it in a nutshell: HOMEOPATHY IS A CULT.

A comprehensive review of the evidence relating to acupuncture entitled “The Acupuncture Evidence Project: A Comparative Literature Review” has just been published. The document aims to provide “an updated review of the literature with greater rigour than was possible in the past.” That sounds great! Let’s see just how rigorous the assessment is.

The review was conducted by John McDonald who no stranger to this blog; we have mentioned him here, for instance. To call him an unbiased, experienced, or expert researcher would, in my view, be more than a little optimistic.

The review was financed by the ‘Australian Acupuncture and Chinese Medicine Association Ltd.’ – call me a pessimist, but I do wonder whether this bodes well for the objectivity of the findings.

The research seems to have been assisted by a range of experts: Professor Caroline Smith, National Institute of Complementary Medicine, Western Sydney University, provided advice regarding evidence levels for assisted reproduction trials; Associate Professor Zhen Zheng, RMIT University identified the evidence levels for postoperative nausea and vomiting and post-operative pain; Dr Suzanne Cochrane, Western Sydney University; Associate Professor Chris Zaslawski, University of Technology Sydney; and Associate Professor Zhen Zheng, RMIT University provided prepublication commentary and advice. I fail to see anyone in this list who is an expert in EBM or who is even mildly critical of acupuncture and the many claims that are being made for it.

The review has not been published in a journal. This means, it has not been peer-reviewed. As we will see shortly, there is reason to doubt that it could pass the peer-review process of any serious journal.

There is an intriguing declaration of conflicts of interest: “Dr John McDonald was a co-author of three of the research papers referenced in this review. Professor Caroline Smith was a co-author of six of the research papers referenced in this review, and Associate Professor Zhen Zheng was co-author of one of the research papers in this review. There were no other conflicts of interest.” Did they all forget to mention that they earn their livelihoods through acupuncture? Or is that not a conflict?

I do love the disclaimer: “The authors and the Australian Acupuncture and Chinese Medicine Association Ltd (AACMA) give no warranty that the information contained in this publication and within any online updates available on the AACMA website are correct or complete.” I think they have a point here.

But let’s not be petty, let’s look at the actual review and how well it was done!

Systematic reviews must first formulate a precise research question, then disclose the exact methodology, reveal the results and finally discuss them critically. I am afraid, I miss almost all of these essential elements in the document in question.

The methods section includes statements which puzzle me (my comments are in bold):

  • A total of 136 systematic reviews, including 27 Cochrane systematic reviews were included in this review, along with three network meta-analyses, nine reviews of reviews and 20 other reviews. Does that indicate that non-systematic reviews were included too? Yes, it does – but only, if they reported a positive result, I presume.
  • Some of the included systematic reviews included studies which were not randomised controlled trials. In this case, they should have not been included at all, in my view.
  • … evidence from individual randomised controlled trials has been included occasionally where new high quality randomised trials may have changed the conclusions from the most recent systematic review. ‘Occasionally’ is the antithesis of systematic. This discloses the present review as being non-systematic and therefore worthless.
  • Some systematic reviews have not reported an assessment of quality of evidence of included trials, and due to time constraints, this review has not attempted to make such an assessment. Say no more!

It is almost needless to mention that the findings (presented in a host of hardly understandable tables) suggest that acupuncture is of proven or possible effectiveness/efficacy for a very wide array of conditions. It also goes without saying that there is no critical discussion, for instance, of the fact that most of the included evidence originated from China, and that it has been shown over and over again that Chinese acupuncture research never seems to produce negative results.

So, what might we conclude from all this?

I don’t know about you, but for me this new review is nothing but an orgy in deceit and wishful thinking!

1 2 3 5
Recent Comments

Note that comments can be edited for up to five minutes after they are first submitted.


Click here for a comprehensive list of recent comments.

Categories