Disgracefully low methodological quality of systematic reviews on acupuncture

Well-conducted systematic reviews (SRs) should in principle provide the most reliable evidence on the effectiveness of acupuncture. However, limitations on the methodological rigour of SRs may impact the trustworthiness of their conclusions. This cross-sectional study was aimed at evaluating the methodological quality of recent SRs of acupuncture.

The Cochrane Database of Systematic Reviews, MEDLINE, and EMBASE were searched for SRs focusing on manual acupuncture or electro-acupuncture published during January 2018 and March 2020. Eligible SRs needed to contain at least one meta-analysis and be published in the English language. Two independent reviewers extracted the bibliographical characteristics of the included SRs with a pre-designed questionnaire and appraised the methodological quality of the reviews with the validated AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2). The associations between bibliographical characteristics and methodological quality ratings were explored using Kruskal-Wallis rank tests and Spearman’s rank correlation coefficients.

A total of 106 SRs were appraised. The results were as follows:

• one (0.9%) SR was of high methodological quality,
• no review (0%) was of moderate quality,
• six (5.7%) were of low quality,
• 99 (93.4%) were of critically low quality.

Only ten (9.4%) provided an a priori protocol, only four (3.8%) conducted a comprehensive literature search, only five (4.7%) provided a list of excluded studies, and only six (5.7%) performed a meta-analysis appropriately. Cochrane SRs, updated SRs, and SRs that did not search non-English databases had relatively higher overall quality. The vast majority (87.7%) of the 106 reviews included in this analysis originated from Asia. Conflicts of interest of the review authors were declared in only 2 of the 106 reviews.

The authors concluded that the methodological quality of SRs on acupuncture is unsatisfactory. Future reviewers should improve critical methodological aspects of publishing protocols, performing comprehensive search, providing a list of excluded studies with justifications for exclusion, and conducting appropriate meta-analyses. These recommendations can be implemented via enhancing the technical competency of reviewers in SR methodology through established education approaches as well as quality gatekeeping by journal editors and reviewers. Finally, for evidence users, skills in SR critical appraisal remain to be essential as relevant evidence may not be available in pre-appraised formats.

On this blog, I have often complained about the lack of critical input and the poor quality of systematic reviews of so-called alternative medicine (SCAM), particularly of acupuncture, and especially of Chinese reviews, and even more especially Chinese reviews of (mostly) Chinese studies. This new paper is a valuable confirmation of this fast-growing deficit.

One does not need to be a prophet to predict that this pollution of the literature with complete rubbish will have detrimental effects. Because poor reviews almost always draw an over-optimistic picture of the value of acupuncture, this phenomenon must seriously mislead the public. The end result will be that the public believes acupuncture to be effective.

I cannot help thinking that this is, in fact, the intended aim of the authors of such poor, false-positive reviews. Moreover, a glance at the subject areas of the reviews in the list below gives the impression that China is heavily promoting the idea that acupuncture is a panacea. Yet there is good evidence to show that acupuncture is little more than placebo therapy.

In my last post, I have reported that I am an author of many of the frequently-cited systematic acupuncture reviews. You might thus assume that I am a significant part of this pollution by rubbish reviews. This would, however, be an entirely wrong conclusion. The above analysis covers a period when my unit had already been closed, and I am thus not responsible for a single of the papers included in the above analysis.

List of included systematic reviews