I remember reading this paper entitled ‘Comparison of acupuncture and other drugs for chronic constipation: A network meta-analysis’ when it first came out. I considered discussing it on my blog, but then decided against it for a range of reasons which I shall explain below. The abstract of the original meta-analysis is copied below:
The objective of this study was to compare the efficacy and side effects of acupuncture, sham acupuncture and drugs in the treatment of chronic constipation. Randomized controlled trials (RCTs) assessing the effects of acupuncture and drugs for chronic constipation were comprehensively retrieved from electronic databases (such as PubMed, Cochrane Library, Embase, CNKI, Wanfang Database, VIP Database and CBM) up to December 2017. Additional references were obtained from review articles. With quality evaluations and data extraction, a network meta-analysis (NMA) was performed using a random-effects model under a frequentist framework. A total of 40 studies (n = 11032) were included: 39 were high-quality studies and 1 was a low-quality study. NMA showed that (1) acupuncture improved the symptoms of chronic constipation more effectively than drugs; (2) the ranking of treatments in terms of efficacy in diarrhoea-predominant irritable bowel syndrome was acupuncture, polyethylene glycol, lactulose, linaclotide, lubiprostone, bisacodyl, prucalopride, sham acupuncture, tegaserod, and placebo; (3) the ranking of side effects were as follows: lactulose, lubiprostone, bisacodyl, polyethylene glycol, prucalopride, linaclotide, placebo and tegaserod; and (4) the most commonly used acupuncture point for chronic constipation was ST25. Acupuncture is more effective than drugs in improving chronic constipation and has the least side effects. In the future, large-scale randomized controlled trials are needed to prove this. Sham acupuncture may have curative effects that are greater than the placebo effect. In the future, it is necessary to perform high-quality studies to support this finding. Polyethylene glycol also has acceptable curative effects with fewer side effects than other drugs.
END OF 1st QUOTE
This meta-analysis has now been retracted. Here is what the journal editors have to say about the retraction:
After publication of this article [1], concerns were raised about the scientific validity of the meta-analysis and whether it provided a rigorous and accurate assessment of published clinical studies on the efficacy of acupuncture or drug-based interventions for improving chronic constipation. The PLOS ONE Editors re-assessed the article in collaboration with a member of our Editorial Board and noted several concerns including the following:
- Acupuncture and related terms are not mentioned in the literature search terms, there are no listed inclusion or exclusion criteria related to acupuncture, and the outcome measures were not clearly defined in terms of reproducible clinical measures.
- The study included acupuncture and electroacupuncture studies, though this was not clearly discussed or reported in the Title, Methods, or Results.
- In the “Routine paired meta-analysis” section, both acupuncture and sham acupuncture groups were reported as showing improvement in symptoms compared with placebo. This finding and its implications for the conclusions of the article were not discussed clearly.
- Several included studies did not meet the reported inclusion criteria requiring that studies use adult participants and assess treatments of >2 weeks in duration.
- Data extraction errors were identified by comparing the dataset used in the meta-analysis (S1 Table) with details reported in the original research articles. Errors included aspects of the study design such as the experimental groups included in the study, the number of study arms in the trial, number of participants, and treatment duration. There are also several errors in the Reference list.
- With regard to side effects, 22 out of 40 studies were noted as having reported side effects. It was not made clear whether side effects were assessed as outcome measures for the other 18 studies, i.e. did the authors collect data clarifying that there were no side effects or was this outcome measure not assessed or reported in the original article. Without this clarification the conclusion comparing side effect frequencies is not well supported.
- The network geometry presented in Fig 5 is not correct and misrepresents some of the study designs, for example showing two-arm studies as three-arm studies.
- The overall results of the meta-analysis are strongly reliant on the evidence comparing acupuncture versus lactulose treatment. Several of the trials that assessed this comparison were poorly reported, and the meta-analysis dataset pertaining to these trials contained data extraction errors. Furthermore, potential bias in studies assessing lactulose efficacy in acupuncture trials versus lactulose efficacy in other trials was not sufficiently addressed.
While some of the above issues could be addressed with additional clarifications and corrections to the text, the concerns about study inclusion, the accuracy with which the primary studies’ research designs and data were represented in the meta-analysis, and the reporting quality of included studies directly impact the validity and accuracy of the dataset underlying the meta-analysis. As a consequence, we consider that the overall conclusions of the study are not reliable. In light of these issues, the PLOS ONE Editors retract the article. We apologize that these issues were not adequately addressed during pre-publication peer review.
LZ disagreed with the retraction. YM and XD did not respond.
END OF 2nd QUOTE
Let me start by explaining why I initially decided not to discuss this paper on my blog. Already the first sentence of the abstract put me off, and an entire chorus of alarm-bells started ringing once I read further.
- A meta-analysis is not a ‘study’ in my book, and I am somewhat weary of researchers who employ odd or unprecise language.
- We all know (and I have discussed it repeatedly) that studies of acupuncture frequently fail to report adverse effects (in doing this, their authors violate research ethics!). So, how can it be a credible aim of a meta-analysis to compare side-effects in the absence of adequate reporting?
- The methodology of a network meta-analysis is complex and I know not a lot about it.
- Several things seemed ‘too good to be true’, for instance, the funnel-plot and the overall finding that acupuncture is the best of all therapeutic options.
- Looking at the references, I quickly confirmed my suspicion that most of the primary studies were in Chinese.
In retrospect, I am glad I did not tackle the task of criticising this paper; I would probably have made not nearly such a good job of it as PLOS ONE eventually did. But it was only after someone raised concerns that the paper was re-reviewed and all the defects outlined above came to light.
While some of my concerns listed above may have been trivial, my last point is the one that troubles me a lot. As it also related to dozens of Cochrane reviews which currently come out of China, it is worth our attention, I think. The problem, as I see it, is as follows:
- Chinese (acupuncture, TCM and perhaps also other) trials are almost invariably reporting positive findings, as we have discussed ad nauseam on this blog.
- Data fabrication seems to be rife in China.
- This means that there is good reason to be suspicious of such trials.
- Many of the reviews that currently flood the literature are based predominantly on primary studies published in Chinese.
- Unless one is able to read Chinese, there is no way of evaluating these papers.
- Therefore reviewers of journal submissions tend to rely on what the Chinese review authors write about the primary studies.
- As data fabrication seems to be rife in China, this trust might often not be justified.
- At the same time, Chinese researchers are VERY keen to publish in top Western journals (this is considered a great boost to their career).
- The consequence of all this is that reviews of this nature might be misleading, even if they are published in top journals.
I have been struggling with this problem for many years and have tried my best to alert people to it. However, it does not seem that my efforts had even the slightest success. The stream of such reviews has only increased and is now a true worry (at least for me). My suspicion – and I stress that it is merely that – is that, if one would rigorously re-evaluate these reviews, their majority would need to be retracted just as the above paper. That would mean that hundreds of papers would disappear because they are misleading, a thought that should give everyone interested in reliable evidence sleepless nights!
So, what can be done?
Personally, I now distrust all of these papers, but I admit, that is not a good, constructive solution. It would be better if Journal editors (including, of course, those at the Cochrane Collaboration) would allocate such submissions to reviewers who:
- are demonstrably able to conduct a CRITICAL analysis of the paper in question,
- can read Chinese,
- have no conflicts of interest.
In the case of an acupuncture review, this would narrow it down to perhaps just a handful of experts worldwide. This probably means that my suggestion is simply not feasible.
But what other choice do we have?
One could oblige the authors of all submissions to include full and authorised English translations of non-English articles. I think this might work, but it is, of course, tedious and expensive. In view of the size of the problem (I estimate that there must be around 1 000 reviews out there to which the problem applies), I do not see a better solution.
(I would truly be thankful, if someone had a better one and would tell us)
12 Responses to Finally! A retraction of an acupuncture meta-analysis published in a top-journal
Leave a Reply to Jashak Cancel reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
The Wikipedia article on academic publishing in China makes interesting reading in the context of this post. See also this piece about how much Chinese scientists are paid cash if their work is published in prestigious Western journals — a sure-fire recipe for dishonesty in science.
I would add to your three criteria for ideal peer reviewers “…has plenty of time to devote to the review”. The reality is that it takes longer to assemble the detailed arguments — supported by appropriate citations — for rejecting a paper than it does to recommend acceptance.
The combined impact of an ever-growing list of scientific publications with ever-decreasing numbers of support staff in universities makes the most critical and knowledgeable peer reviewers so busy they’re more likely than ever to refuse to referee a paper that looks even a little bit dodgy from the abstract.
good points!
when I was still at Exeter, I used to do about 3 reviews/week on average. but I had lots of support from secretaries and co-workers. now that I have none of this, I reject almost all invitations to review journal submissions – it’s a lot of work when done properly, and not responsible to do it superficially.
It strikes me as irresponsiblity on the part of editors to submit papers to a reviewer if the reviewer does not read the language in which most of the articles cited are written, or in this case, included in the corpus of the meta-analysis.
My experience has been that what the cited paper says and what the author of the paper who cited it thinks it said are sometimes completely at odds. Sometimes what the reported data says and the conclusions drawn are completely at odds within the paper.
YOU ARE RIGHT!
the journal editors must take the responsibility. the problem is, I think, that they usually do not have the funds that would be needed to do so.
What about working on opening positions for medical post-grad students with a good mastering of both English and Chinese, like, for instance, Chinese students in England and the US? I’m not sure what form this kind of appointment would take, but I understand that there are quite a bit of Chinese studying abroad, without reckoning with Anglo-Saxon students from Chinese families. Maybe, there might be a way to build a network of such students who could share the work and also get deeper insight in critical thinking about the scientific method. But being completely outside the institutional world of medical studies and researches, I don’t know how that could work.
good idea!
one problem is that critical assessment skills need a lot of training and reviewers are usually paid NOTHING.
Respected people such as Tricia Greenhalgh are asking for open peer review. There was a total failure of peer review here, as the errors are many and obvious. The usual formula was applied – quack papers are reviewed by other quacks. Frankly though the errors were so glaring that the editors should not even have invited reviewers. Maybe we should move beyond peer review and try something different, such as:
1. Screen submissions for obvious flaws, but this will cost money for editorial staff.
2. Publish in draft and invite critique.
3. Authors to rework paper in response to critiques.
4. Each paper goes through a version control system until all points of criticism are addressed. Any draft can be cited but is prominently flagged as draft and may have errors.
5. Final version published after a set number of versions.
Some online journals are doing something like this. There are problems with it, such as malicious rivals deliberately delaying finalisation of a paper they don’t like. Right now we often have flawed papers that have been reworked, but other authors commonly cite the earlier flawed version because it suits their beliefs. The only solution to this is retraction – but retracted papers still get cited and reviewers often miss this! Whatever the solution, traditional peer review is not fit for purpose.
Given the proclamation by China’s leader that TCM is a national treasure to be given equal status with western medicine I believe that the default position of reputable publishers outside China should be that papers originating from within China have a serious risk of bias in favour of TCM.
Accordingly reviewers should be alert to and capable of bringing all such biases to light. When acupuncture is claimed to be more effective than drugs for chronic constipation, red lights and alarm bells should be flashing and ringing in any scientifically minded brain.
Retraction of this paper might be good news, however, I wonder if the overall fight against TCM is lost… at least for the foreseeable future.
The reason for this depressing notion is that the WHO apparently is on the side of TCM. This is e.g. evidenced by the recent decision of the WHO to put TCM into its International Statistical Classification of Diseases and Related Health Problems (ICD) catalogue next year.
https://www.nature.com/articles/d41586-018-06782-7
The ongoing effort of the WHO to integrate TCM into an “universal health coverage” strategy -despite the fact that evidence for efficacy of pretty much all TCM practices is lacking- is a big scandal in my opinion.
@Jashak
The concept and philosophy of TCM is being marketed as a commodity including related goods and paraphernalia from a certain large and very productive country that is rapidly increasing its foothold abroad. The former director of WHO, Margaret Chan Fung Fu-chun, OBE, JP, established her own agenda that has nothing to do with genuine medicine or public health. She was blatantly marketing outdated medicine as a commodity. In addition to the incredible new addition of TCM diagnostic terms to ICD, the literally naive WHO traditional medicine strategy: 2014-2023 document was another embodiment of this misuse of WHO.
The sheer silliness of the contents of the TCM chapter in ICD-11 can be used to advantage by us who are critical to this “bait and switch” promotion of archaeological medicine? I for one have been showing the TCM chapter to colleagues, pointing out the outragrousness of the terms along with an explanation of the ulterior motives of those who have an agenda topopularise TCM.
@ Björn Geir,
thank you for your comment.
Although I understand the hope that once the TCM terms have been defined and put into the ICD, it could become more easy to criticize TCM methods by rational people like yourself.
But don’t you agree that accepting TCM into the WHO register will increase the appearance of legitimacy of these hocus-pocus treatments and that the negative consequences of this act will far outweigh the hope of rational people?
@Jashak
I wholeheartedly agree that the inclusion of this marketing of fake oriental medicine in the ICD is an outrage.
My point is that since it is in there, it should be made known far and wide, how and why it came to be and why it constitutes a risk factor for public health.