The objective of this ‘real world’ study was to evaluate the effectiveness of integrative medicine (IM) on patients with coronary artery disease (CAD) and investigate the prognostic factors of CAD in a real-world setting.
A total of 1,087 hospitalized patients with CAD from 4 hospitals in Beijing, China were consecutively selected between August 2011 and February 2012. The patients were assigned to two groups:
- Chinese medicine (CM) plus conventional treatment, i.e., IM therapy (IM group). IM therapy meant that the patients accepted the conventional treatment of Western medicine and the treatment of Chinese herbal medicine including herbal-based injection and Chinese patent medicine as well as decoction for at least 7 days in the hospital or 3 months out of the hospital.
- Conventional treatment alone (CT group).
The endpoint was a major cardiac event [MCE; including cardiac death, myocardial infarction (MI), and the need for revascularization].
A total of 1,040 patients finished the 2-year follow-up. Of them, 49.4% received IM therapy. During the 2-year follow-up, the total incidence of MCE was 11.3%. Most of the events involved revascularization (9.3%). Cardiac death/MI occurred in 3.0% of cases. For revascularization, logistic stepwise regression analysis revealed that age ⩾ 65 years [odds ratio (OR), 2.224], MI (OR, 2.561), diabetes mellitus (OR, 1.650), multi-vessel lesions (OR, 2.554), baseline high sensitivity C-reactive protein level ⩾ 3 mg/L (OR, 1.678), and moderate or severe anxiety/depression (OR, 1.849) were negative predictors (P<0.05); while anti-platelet agents (OR, 0.422), β-blockers (OR, 0.626), statins (OR, 0.318), and IM therapy (OR, 0.583) were protective predictors (P<0.05). For cardiac death/MI, age ⩾ 65 years (OR, 6.389) and heart failure (OR, 7.969) were negative predictors (P<0.05), while statin use (OR, 0.323) was a protective predictor (P<0.05) and IM therapy showed a beneficial tendency (OR, 0.587), although the difference was not statistically significant (P=0.218).
The authors concluded that in a real-world setting, for patients with CAD, IM therapy was associated with a decreased incidence of revascularization and showed a potential benefit in reducing the incidence of cardiac death or MI.
What the authors call ‘real world setting’ seems to be a synonym of ‘lousy science’, I fear. I am not aware of good evidence to show that herbal injections and concoctions are effective treatments for CAD, and this study can unfortunately not change this. In the methods section of the paper, we read that the treatment decisions were made by the responsible physicians without restriction. That means the two groups were far from comparable. In their discussion section, the authors state; we found that IM therapy was efficacious in clinical practice. I think that this statement is incorrect. All they have shown is that two groups of patients with similar diagnoses can differ in numerous ways, including clinical outcomes.
The lessons here are simple:
- In clinical trials, lack of randomisation (the only method to create reliably comparable groups) often leads to false results.
- Flawed research is currently being used by many proponents of SCAM (so-called alternative medicine) to mislead us about the value of SCAM.
- The integration of dubious treatments into routine care does not lead to better outcomes.
- Integrative medicine, as currently advocated by SCAM-proponents, is a nonsense.
Ginkgo biloba is a well-researched herbal medicine which has shown promise for a number of indications. But does this include coronary heart disease?
The aim of this systematic review was to provide information about the effectiveness and safety of Ginkgo Leaf Extract and Dipyridamole Injection (GD) as one adjuvant therapy for treating angina pectoris (AP) and to evaluate the relevant randomized controlled trials (RCTs) with meta-analysis. (Ginkgo Leaf Extract and Dipyridamole Injection is a Chinese compound preparation, which consists of ginkgo ﬂavone glycosides (24%), terpene lactones (ginkgolide about 13%, ginkgolide about 2.9%) and dipyridamole.)
RCTs concerning AP treated by GD were searched and the Cochrane Risk Assessment Tool was adopted to assess the methodological quality of the RCTs. A total of 41 RCTs involving 4,462 patients were included in the meta-analysis. The results indicated that the combined use of GD and Western medicine (WM) against AP was associated with a higher total effective rate [risk ratio (RR)=1.25, 95% confidence interval (CI): 1.21–1.29, P<0.01], total effective rate of electrocardiogram (RR=1.29, 95% CI: 1.21–1.36, P<0.01). Additional, GD combined with WM could decrease the level of plasma viscosity [mean difference (MD)=–0.56, 95% CI:–0,81 to–0.30, P<0.01], fibrinogen [MD=–1.02, 95% CI:–1.50 to–0.54, P<0.01], whole blood low shear viscosity [MD=–2.27, 95% CI:–3.04 to–1.49, P<0.01], and whole blood high shear viscosity (MD=–0.90, 95% CI: 1.37 to–0.44, P<0.01).
The authors concluded that comparing with receiving WM only, the combine use of GD and WM was associated with a better curative effect for patients with AP. Nevertheless, limited by the methodological quality of included RCTs more large-sample, multi-center RCTs were needed to confirm our findings and provide further evidence for the clinical utility of GD.
If one reads this conclusion, one might be tempted to use GD to cure AP. I would, however, strongly warn everyone from doing so. There are many reasons for my caution:
- All the 41 RCTs originate from China, and we have repeatedly discussed that Chinese TCM trials are highly unreliable.
- The methodological quality of the primary RCTs was, according to the review authors ‘moderate’. This is not true; it was, in fact, lousy.
- Dipyridamole is not indicated in angina pectoris.
- To the best of my knowledge, there is no good evidence from outside China to suggest that Ginkgo biloba is effective for angina pectoris.
- Angina pectoris is caused by coronary artery disease (a narrowing of one or more coronary arteries due to atherosclerosis), and it seems implausible that this condition can be ‘cured’ with any medication.
So, what we have here is yet another nonsensical paper, published in a dubious journal, employing evidently irresponsible reviewers, run by evidently irresponsible editors, hosted by a seemingly reputable publisher (Springer). This is reminiscent of my previous post (and many posts before). Alarmingly, it is also what I encounter on a daily basis when scanning the new publications in my field.
The effects of this incessant stream of nonsense can only have one of two effects:
- People take this ‘evidence’ seriously. In this case, many patients might pay with their lives for this collective incompetence.
- People conclude that alt med research cannot be taken seriously. In this case, we are unlikely to ever see anything useful emerging from it.
Either way, the result will be profoundly negative!
It is high time to stop this idiocy; but how?
I wish, I knew the answer.
This systematic review was aimed at evaluating the effects of acupuncture on the quality of life of migraineurs. Only randomized controlled trials that were published in Chinese and English were included. In total, 62 trials were included for the final analysis; 50 trials were from China, 3 from Brazil, 3 from Germany, 2 from Italy and the rest came from Iran, Israel, Australia and Sweden.
Acupuncture resulted in lower Visual Analog Scale scores than medication at 1 month after treatment and 1-3 months after treatment. Compared with sham acupuncture, acupuncture resulted in lower Visual Analog Scale scores at 1 month after treatment.
The authors concluded that acupuncture exhibits certain efficacy both in the treatment and prevention of migraines, which is superior to no treatment, sham acupuncture and medication. Further, acupuncture enhanced the quality of life more than did medication.
The authors comment in the discussion section that the overall quality of the evidence for most outcomes was of low to moderate quality. Reasons for diminished quality consist of the following: no mentioned or inadequate allocation concealment, great probability of reporting bias, study heterogeneity, sub-standard sample size, and dropout without analysis.
Further worrisome deficits are that only 14 of the 62 studies reported adverse effects (this means that 48 RCTs violated research ethics!) and that there was a high level of publication bias indicating that negative studies had remained unpublished. However, the most serious concern is the fact that 50 of the 62 trials originated from China, in my view. As I have often pointed out, such studies have to be categorised as highly unreliable.
In view of this multitude of serious problems, I feel that the conclusions of this review must be re-formulated:
Despite the fact that many RCTs have been published, the effect of acupuncture on the quality of life of migraineurs remains unproven.
I only recently came across this review; it was published a few years ago but is still highly relevant. It summarizes the evidence of controlled clinical studies of TCM for cancer.
The authors searched all the controlled clinical studies of TCM therapies for all kinds of cancers published in Chinese in four main Chinese electronic databases from their inception to November 2011. They found a total of 2964 reports (involving 253,434 cancer patients) including 2385 randomized controlled trials and 579 non-randomized controlled studies.
The top seven cancer types treated were lung cancer, liver cancer, stomach cancer, breast cancer, esophagus cancer, colorectal cancer and nasopharyngeal cancer by both study numbers and case numbers. The majority of studies (72%) applied TCM therapy combined with conventional treatment, whilst fewer (28%) applied only TCM therapy in the experimental groups. Herbal medicine was the most frequently applied TCM therapy (2677 studies, 90.32%). The most frequently reported outcome was clinical symptom improvement (1667 studies, 56.24%) followed by biomarker indices (1270 studies, 42.85%), quality of life (1129 studies, 38.09%), chemo/radiotherapy induced side effects (1094 studies, 36.91%), tumour size (869 studies, 29.32%) and safety (547 studies, 18.45%).
The authors concluded that data from controlled clinical studies of TCM therapies in cancer treatment is substantial, and different therapies are applied either as monotherapy or in combination with conventional medicine. Reporting of controlled clinical studies should be improved based on the CONSORT and TREND Statements in future. Further studies should address the most frequently used TCM therapy for common cancers and outcome measures should address survival, relapse/metastasis and quality of life.
This paper is important, in my view, predominantly because it exemplifies the problem with TCM research from China and with uncritical reviews on this subject. If a cancer patient, who does not know the background, reads this paper, (s)he might think that TCM is worth trying. This conclusion could easily shorten his/her life.
The often-shown fact is that TCM studies from China are not reliable. They are almost invariably positive, their methodological quality is low, and they are frequently based on fabricated data. In my view, it is irresponsible to publish a review that omits discussing these facts in detail and issuing a stark warning.
TCM FOR CANCER IS A VERY BAD CHOICE!
One theory as to how acupuncture works is that it increases endorphin levels in the brain. These ‘feel-good’ chemicals could theoretically be helpful for weaning alcohol-dependent people off alcohol. So, for once, we might have a (semi-) plausible mechanism as to how acupuncture could be clinically effective. But a ‘beautiful hypothesis’ does not necessarily mean acupuncture works for alcohol dependence. To answer this question, we need clinical trials or systematic reviews of clinical trials.
A new systematic review assessed the effects and safety of acupuncture for alcohol withdrawal syndrome (AWS). All RCTs of drug plus acupuncture or acupuncture alone for the treatment of AWS were included. Eleven RCTs with a total of 875 participants were included. In the acute phase, two trials reported no difference between drug plus acupuncture and drug plus sham acupuncture in the reduction of craving for alcohol; however, two positive trials reported that drug plus acupuncture was superior to drug alone in the alleviation of psychological symptoms. In the protracted phase, one trial reported acupuncture was superior to sham acupuncture in reducing the craving for alcohol, one trial reported no difference between acupuncture and drug (disulfiram), and one trial reported acupuncture was superior to sham acupuncture for the alleviation of psychological symptoms. Adverse effects were tolerable and not severe.
The authors concluded that there was no significant difference between acupuncture (plus drug) and sham acupuncture (plus drug) with respect to the primary outcome measure of craving for alcohol among participants with AWS, and no difference in completion rates (pooled results). There was limited evidence from individual trials that acupuncture may reduce alcohol craving in the protracted phase and help alleviate psychological symptoms; however, given concerns about the quantity and quality of included studies, further large-scale and well-conducted RCTs are needed.
There is little to add here. Perhaps just two short points:
1. The quality of the trials was poor; only one study of the 11 trials was of acceptable rigor. Here is its abstract:
We report clinical data on the efficacy of acupuncture for alcohol dependence. 503 patients whose primary substance of abuse was alcohol participated in this randomized, single blind, placebo controlled trial. Patients were assigned to either specific acupuncture, nonspecific acupuncture, symptom based acupuncture or convention treatment alone. Alcohol use was assessed, along with depression, anxiety, functional status, and preference for therapy. This article will focus on results pertaining to alcohol use. Significant improvement was shown on nearly all measures. There were few differences associated with treatment assignment and there were no treatment differences on alcohol use measures, although 49% of subjects reported acupuncture reduced their desire for alcohol. The placebo and preference for treatment measures did not materially effect the results. Generally, acupuncture was not found to make a significant contribution over and above that achieved by conventional treatment alone in reduction of alcohol use.
To me, this does not sound all that encouraging.
2. Of the 11 RCTs, 8 failed to report on adverse effects of acupuncture. In my book, this means these trials were in violation with basic research ethics.
My conclusion of all this: another ugly fact kills a beautiful hypothesis.
Acupuncture research does not have a good name; if it originates from China, even less so.
And this note in ‘ACUPUNCTURE IN MEDICINE’ is not likely to change this image:
Fang J, Keller CL, Chen L, et al. Effect of acupuncture and Chinese herbal medicine on subacute stroke outcomes: a single-centre randomised controlled trial. Acupuncture in Medicine Published online first 10 November 2017. doi: 10.1136/acupmed-2016-011167.
This article is retracted by the Editor-in-Chief on grounds of redundant publication.
The above article reports that a trial originally planned to be carried out at three hospitals was reduced to a single centre for reasons of cost. This is incorrect. The full three-centre trial was run and reported elsewhere (Scientific Reports 6, Article number: 25850 (2016) DOI: 10.1038/srep25850).
The Scientific Reports paper was accepted for publication prior to submission of the above paper to Acupuncture in Medicine. The third author takes responsibility for the mistake. All authors have agreed to this retraction.
The abstract of the paper in SCIENTIFIC REPORTS is here:
To determine whether integrative medicine rehabilitation (IMR) that combines conventional rehabilitation (CR) with acupuncture and Chinese herbal medicine has better effects for subacute stroke than CR alone, we conducted a multicenter randomized controlled trial that involved three hospitals in China. Three hundred sixty patients with subacute stroke were randomized into IMR and CR groups. The primary outcome was the Modified Barthel Index (MBI). The secondary outcomes were the National Institutes of Health Stroke Scale (NIHSS), the Fugl-Meyer Assessment (FMA), the mini-mental state examination (MMSE), the Montreal Cognitive Assessment (MoCA), Hamilton’s Depression Scale (HAMD), and the Self-Rating Depression Scale (SDS). All variables were evaluated at week 0 (baseline), week 4 (half-way of intervention), week 8 (after treatment) and week 20 (follow-up). In comparison with the CR group, the IMR group had significantly better improvements (P < 0.01 or P < 0.05) in all the primary and secondary outcomes. There were also significantly better changes from baseline in theses outcomes in the IMR group than in the CR group (P < 0.01). A low incidence of adverse events with mild symptoms was observed in the IMR group. We conclude that conventional rehabilitation combined with integrative medicine is safe and more effective for subacute stroke rehabilitation.
I find all this odd in several ways:
- The publication of the ACUPUNCTURE IN MEDICINE article does not seem to have been a ‘mistake‘ but plain scientific fraud, in my view.
- The paper in SCIENTIFIC REPORTS (SR) was published in May 2016. Therefore the reviewers and editor of AIM could and should have spotted the fraud.
- In the SR paper, the authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. As the authors were affiliated to three different departments of acupuncture, I feel this to be debatable.
What do you think?
Bee venom acupuncture is a form of acupuncture in which bee venom is applied to the tips of acupuncture needles, stingers are extracted from bees, or bees are held with an instrument exposing the stinger, and applied to acupoints on the skin.
Bee venom consisting of multiple anti-inflammatory compounds such as melittin, adolapin, apamin. Other substances such as phospholipase A2 can be anti-inflammatory in low concentrations and pro-inflammatory in others. However, bee venom also contains proinflammatory substances, melittin, mast cell degranulation peptide 401, and histamine.
Bee venom acupuncture has been used to treat a number of conditions such as lumbar disc disease, osteoarthritis of the knee, rheumatoid arthritis, adhesive capsulitis, lateral epicondylitis, peripheral neuropathies, stroke and Parkinson’s Disease. The quality of these studies tends to be so poor that any verdict on the effectiveness of bee venom acupuncture would be premature.
A new clinical trial of bee-venom acupuncture for rheumatoid arthritis (RA) might change this situation. A total of 120 cases of RA patients were randomized into bee-sting acupuncture group (treatment) and western medicine group (control). The patients of the control group were treated by oral administration of Methotrexate (10 mg, once a week) and Celecoxlb (0.2 g, once a day). Those of the treatment group received 5 to 15 bee stings of Ashi-points or acupoints according to different conditions and corporeity, and with the bee-sting retained for about 5 min every time, once every other day. The treatment lasted for 8 weeks. The therapeutic effect was assessed by examining:
- symptoms and signs of the affected joints as morning stiffness duration,
- swollen/tender joint counts (indexes),
- handgrip strength,
- 15 m-walking time,
- visual analogue scale (VAS),
- Disease Activity Score including a 28-joint count (DAS 28),
- rheumatoid factor (RF),
- erythrocyte sedimentation rate (ESR),
- C-reactive protein (CRP),
- anti-cyclic citrullinated peptide antibody (ACCPA).
For assessing the safety of bee-venom acupuncture, the patients’ responses of fever, enlargement of lymph nodes, regional red and swollen, itching, blood and urine tests for routine were examined.
Findings of DAS 28 responses displayed that of the two 60 cases in the control and bee-venom acupuncture groups, 15 and 18 experienced marked improvement, 33 and 32 were effective, 12 and 10 ineffective, with the effective rates being 80% and 83. 33%, respectively. No significant difference was found between the two groups in the effective rate (P>0.05). After the treatment, both groups have witnessed a marked decrease in the levels of morning stiffness duration, arthralgia index, swollen joint count index, joint tenderness index, 15 m walking time, VAS, RF, ESR, CRP and ACCPA, and an obvious increase of handgrip strength relevant to their own levels of pre-treatment in each group (P<0.05). There were no significant differences between the two groups in the abovementioned indexes (P>0.05). The routine blood test, routine urine test, routine stool test, electrocardiogram result, the function of liver and kidney and other security index were within the normal range, without any significant adverse effects found after bee-stinging treatment.
The authors (from the Department of Acupuncture and Moxibustion, Bao’an Hospital of Traditional Chinese Medicine, Shenzhen, China) concluded that bee-venom acupuncture therapy for RA patients is safe and effective, worthy of popularization and application in clinical practice.
Where to start? There is so much – perhaps I just comment on the conclusion:
- Safety cannot be assessed on the basis of such a small sample. Bee venom can cause anaphylaxis, and several deaths have been reported in patients who successfully received the therapy prior to the adverse event. Because there is no adverse-effect monitoring system, the incidence of adverse events is unknown. Stating that it is safe, is therefore a big mistake.
- The trial was a non-superiority study. As such, it needs a much larger sample to be able to make claims about effectiveness.
- From the above two points, it follows that popularization and application in clinical practice would be a stupid exercise.
So, what is left over from this seemingly rigorous RCT?
(except perhaps a re-affirmation of my often-voiced fear that we must take TCM-studies from China with more than just one pinch of salt)
Traditional Chinese Medicine (TCM) is the umbrella term for modalities historically used in ancient China. TCM includes many therapeutic and some diagnostic modalities. Even though, these modalities differ in many respects, they are claimed to have in common that they are based on assumptions most of which originate from Taoist philosophy:
- The human body is a miniature version of the universe.
- Harmony between the two opposing forces, yin and yang, means health.
- Disease is caused by an imbalance between these forces.
- Five elements—fire, earth, wood, metal, and water—symbolically represent all phenomena, including the stages of human life, and explain the functioning of the body and how it changes during disease.
- The vital energy, qi or chi, flows through the body in meridians, is essential for maintaining health.
TCM is a construct of Mao Zedong who lumped all historical Chinese treatments together under this umbrella and created the ‘barefoot doctor’ to practice TCM nationwide – not because he believed in TCM, but because China was desperately short of real doctors and needed at least a semblance of healthcare.
Over the past few years, China has been aggressively promoting TCM for expanding its global influence and for a share of the estimated US$50-billion global market (of products of dubious quality). A recent article in ‘Nature’ explains that the WHO’s governing body, the World Health Assembly, is set to adopt the 11th version of the organization’s global compendium — known as the International Statistical Classification of Diseases and Related Health Problems (ICD). For the first time, the ICD will include information about TCM. Chapter 26 of the ICD will feature a classification system on TCM, largely based not on science or facts, but on obsolete nonsense.
The WHO’s support applies to all traditional medicines, but its relationship with Chinese medicine, and with China, has grown especially close, in particular during the tenure of Margaret Chan, who ran the organization from 2006 to 2017 and made sure that several documents favourable to TCM were passed. The WHO’s declarations about traditional medicine are puzzling. Various of these WHO documents call for the integration of “traditional medicine, of proven quality, safety and efficacy”, while being silent as to which traditional medicines and diagnostics are proven. Wu Linlin, a WHO representative in the Beijing office, told Nature that the “WHO does not endorse particular traditional and complementary medicine procedures or remedies”.
But this is evidently not the case and in sharp contrast to the WHO’s actions in other areas. The agency provides, for instance, specific advice on what vaccines and drugs to use and what foods to avoid. With traditional medicines, however, such specifics are missing. The message therefore can only be that the WHO endorses TCM as safe and effective.
The evidence, however, tells us a different story. On this blog, we have repeatedly discussed that:
- the clinical trials of TCM are often of poor quality,
- there are several good reasons why one has to be very sceptical about research coming out of China,
- critics are silenced,
- in general, the evidence on TCM is neither positive nor trustworthy.
China’s drug regulator gets more than 230,000 reports of adverse effects from TCM each year, and Chinese herbal medicines carry multiple direct risks:
- one or more ingredients can be toxic,
- some have been shown to be contaminated with toxic materials such as heavy metals,
- others are adulterated with synthetic drugs such as steroids,
- others again can interact with prescription drugs taken concomitantly,
- and not to forget, the risk for endangered species.
To this, we have to add the indirect risk of employing useless treatments for otherwise treatable conditions.
In view of all this, the WHO’s endorsement of TCM and its obsolete concepts is not just not understandable, it is a dangerous step backwards and, in my view, even intolerable.
I remember reading this paper entitled ‘Comparison of acupuncture and other drugs for chronic constipation: A network meta-analysis’ when it first came out. I considered discussing it on my blog, but then decided against it for a range of reasons which I shall explain below. The abstract of the original meta-analysis is copied below:
The objective of this study was to compare the efficacy and side effects of acupuncture, sham acupuncture and drugs in the treatment of chronic constipation. Randomized controlled trials (RCTs) assessing the effects of acupuncture and drugs for chronic constipation were comprehensively retrieved from electronic databases (such as PubMed, Cochrane Library, Embase, CNKI, Wanfang Database, VIP Database and CBM) up to December 2017. Additional references were obtained from review articles. With quality evaluations and data extraction, a network meta-analysis (NMA) was performed using a random-effects model under a frequentist framework. A total of 40 studies (n = 11032) were included: 39 were high-quality studies and 1 was a low-quality study. NMA showed that (1) acupuncture improved the symptoms of chronic constipation more effectively than drugs; (2) the ranking of treatments in terms of efficacy in diarrhoea-predominant irritable bowel syndrome was acupuncture, polyethylene glycol, lactulose, linaclotide, lubiprostone, bisacodyl, prucalopride, sham acupuncture, tegaserod, and placebo; (3) the ranking of side effects were as follows: lactulose, lubiprostone, bisacodyl, polyethylene glycol, prucalopride, linaclotide, placebo and tegaserod; and (4) the most commonly used acupuncture point for chronic constipation was ST25. Acupuncture is more effective than drugs in improving chronic constipation and has the least side effects. In the future, large-scale randomized controlled trials are needed to prove this. Sham acupuncture may have curative effects that are greater than the placebo effect. In the future, it is necessary to perform high-quality studies to support this finding. Polyethylene glycol also has acceptable curative effects with fewer side effects than other drugs.
END OF 1st QUOTE
This meta-analysis has now been retracted. Here is what the journal editors have to say about the retraction:
After publication of this article , concerns were raised about the scientific validity of the meta-analysis and whether it provided a rigorous and accurate assessment of published clinical studies on the efficacy of acupuncture or drug-based interventions for improving chronic constipation. The PLOS ONE Editors re-assessed the article in collaboration with a member of our Editorial Board and noted several concerns including the following:
- Acupuncture and related terms are not mentioned in the literature search terms, there are no listed inclusion or exclusion criteria related to acupuncture, and the outcome measures were not clearly defined in terms of reproducible clinical measures.
- The study included acupuncture and electroacupuncture studies, though this was not clearly discussed or reported in the Title, Methods, or Results.
- In the “Routine paired meta-analysis” section, both acupuncture and sham acupuncture groups were reported as showing improvement in symptoms compared with placebo. This finding and its implications for the conclusions of the article were not discussed clearly.
- Several included studies did not meet the reported inclusion criteria requiring that studies use adult participants and assess treatments of >2 weeks in duration.
- Data extraction errors were identified by comparing the dataset used in the meta-analysis (S1 Table) with details reported in the original research articles. Errors included aspects of the study design such as the experimental groups included in the study, the number of study arms in the trial, number of participants, and treatment duration. There are also several errors in the Reference list.
- With regard to side effects, 22 out of 40 studies were noted as having reported side effects. It was not made clear whether side effects were assessed as outcome measures for the other 18 studies, i.e. did the authors collect data clarifying that there were no side effects or was this outcome measure not assessed or reported in the original article. Without this clarification the conclusion comparing side effect frequencies is not well supported.
- The network geometry presented in Fig 5 is not correct and misrepresents some of the study designs, for example showing two-arm studies as three-arm studies.
- The overall results of the meta-analysis are strongly reliant on the evidence comparing acupuncture versus lactulose treatment. Several of the trials that assessed this comparison were poorly reported, and the meta-analysis dataset pertaining to these trials contained data extraction errors. Furthermore, potential bias in studies assessing lactulose efficacy in acupuncture trials versus lactulose efficacy in other trials was not sufficiently addressed.
While some of the above issues could be addressed with additional clarifications and corrections to the text, the concerns about study inclusion, the accuracy with which the primary studies’ research designs and data were represented in the meta-analysis, and the reporting quality of included studies directly impact the validity and accuracy of the dataset underlying the meta-analysis. As a consequence, we consider that the overall conclusions of the study are not reliable. In light of these issues, the PLOS ONE Editors retract the article. We apologize that these issues were not adequately addressed during pre-publication peer review.
LZ disagreed with the retraction. YM and XD did not respond.
END OF 2nd QUOTE
Let me start by explaining why I initially decided not to discuss this paper on my blog. Already the first sentence of the abstract put me off, and an entire chorus of alarm-bells started ringing once I read further.
- A meta-analysis is not a ‘study’ in my book, and I am somewhat weary of researchers who employ odd or unprecise language.
- We all know (and I have discussed it repeatedly) that studies of acupuncture frequently fail to report adverse effects (in doing this, their authors violate research ethics!). So, how can it be a credible aim of a meta-analysis to compare side-effects in the absence of adequate reporting?
- The methodology of a network meta-analysis is complex and I know not a lot about it.
- Several things seemed ‘too good to be true’, for instance, the funnel-plot and the overall finding that acupuncture is the best of all therapeutic options.
- Looking at the references, I quickly confirmed my suspicion that most of the primary studies were in Chinese.
In retrospect, I am glad I did not tackle the task of criticising this paper; I would probably have made not nearly such a good job of it as PLOS ONE eventually did. But it was only after someone raised concerns that the paper was re-reviewed and all the defects outlined above came to light.
While some of my concerns listed above may have been trivial, my last point is the one that troubles me a lot. As it also related to dozens of Cochrane reviews which currently come out of China, it is worth our attention, I think. The problem, as I see it, is as follows:
- Chinese (acupuncture, TCM and perhaps also other) trials are almost invariably reporting positive findings, as we have discussed ad nauseam on this blog.
- Data fabrication seems to be rife in China.
- This means that there is good reason to be suspicious of such trials.
- Many of the reviews that currently flood the literature are based predominantly on primary studies published in Chinese.
- Unless one is able to read Chinese, there is no way of evaluating these papers.
- Therefore reviewers of journal submissions tend to rely on what the Chinese review authors write about the primary studies.
- As data fabrication seems to be rife in China, this trust might often not be justified.
- At the same time, Chinese researchers are VERY keen to publish in top Western journals (this is considered a great boost to their career).
- The consequence of all this is that reviews of this nature might be misleading, even if they are published in top journals.
I have been struggling with this problem for many years and have tried my best to alert people to it. However, it does not seem that my efforts had even the slightest success. The stream of such reviews has only increased and is now a true worry (at least for me). My suspicion – and I stress that it is merely that – is that, if one would rigorously re-evaluate these reviews, their majority would need to be retracted just as the above paper. That would mean that hundreds of papers would disappear because they are misleading, a thought that should give everyone interested in reliable evidence sleepless nights!
So, what can be done?
Personally, I now distrust all of these papers, but I admit, that is not a good, constructive solution. It would be better if Journal editors (including, of course, those at the Cochrane Collaboration) would allocate such submissions to reviewers who:
- are demonstrably able to conduct a CRITICAL analysis of the paper in question,
- can read Chinese,
- have no conflicts of interest.
In the case of an acupuncture review, this would narrow it down to perhaps just a handful of experts worldwide. This probably means that my suggestion is simply not feasible.
But what other choice do we have?
One could oblige the authors of all submissions to include full and authorised English translations of non-English articles. I think this might work, but it is, of course, tedious and expensive. In view of the size of the problem (I estimate that there must be around 1 000 reviews out there to which the problem applies), I do not see a better solution.
(I would truly be thankful, if someone had a better one and would tell us)
Psoriasis is one of those conditions that is
- not curable,
- irritating to the point where it reduces quality of life.
In other words, it is a disease for which virtually all alternative treatments on the planet are claimed to be effective. But which therapies do demonstrably alleviate the symptoms?
This review (published in JAMA Dermatology) compiled the evidence on the efficacy of the most studied complementary and alternative medicine (CAM) modalities for treatment of patients with plaque psoriasis and discusses those therapies with the most robust available evidence.
PubMed, Embase, and ClinicalTrials.gov searches (1950-2017) were used to identify all documented CAM psoriasis interventions in the literature. The criteria were further refined to focus on those treatments identified in the first step that had the highest level of evidence for plaque psoriasis with more than one randomized clinical trial (RCT) supporting their use. This excluded therapies lacking RCT data or showing consistent inefficacy.
A total of 457 articles were found, of which 107 articles were retrieved for closer examination. Of those articles, 54 were excluded because the CAM therapy did not have more than 1 RCT on the subject or showed consistent lack of efficacy. An additional 7 articles were found using references of the included studies, resulting in a total of 44 RCTs (17 double-blind, 13 single-blind, and 14 nonblind), 10 uncontrolled trials, 2 open-label nonrandomized controlled trials, 1 prospective controlled trial, and 3 meta-analyses.
Compared with placebo, application of topical indigo naturalis, studied in 5 RCTs with 215 participants, showed significant improvements in the treatment of psoriasis. Treatment with curcumin, examined in 3 RCTs (with a total of 118 participants), 1 nonrandomized controlled study, and 1 uncontrolled study, conferred statistically and clinically significant improvements in psoriasis plaques. Fish oil treatment was evaluated in 20 studies (12 RCTs, 1 open-label nonrandomized controlled trial, and 7 uncontrolled studies); most of the RCTs showed no significant improvement in psoriasis, whereas most of the uncontrolled studies showed benefit when fish oil was used daily. Meditation and guided imagery therapies were studied in 3 single-blind RCTs (with a total of 112 patients) and showed modest efficacy in treatment of psoriasis. One meta-analysis of 13 RCTs examined the association of acupuncture with improvement in psoriasis and showed significant improvement with acupuncture compared with placebo.
The authors concluded that CAM therapies with the most robust evidence of efficacy for treatment of psoriasis are indigo naturalis, curcumin, dietary modification, fish oil, meditation, and acupuncture. This review will aid practitioners in advising patients seeking unconventional approaches for treatment of psoriasis.
I am sorry to say so, but this review smells fishy! And not just because of the fish oil. But the fish oil data are a good case in point: the authors found 12 RCTs of fish oil. These details are provided by the review authors in relation to oral fish oil trials: Two double-blind RCTs (one of which evaluated EPA, 1.8g, and DHA, 1.2g, consumed daily for 12 weeks, and the other evaluated EPA, 3.6g, and DHA, 2.4g, consumed daily for 15 weeks) found evidence supporting the use of oral fish oil. One open-label RCT and 1 open-label non-randomized controlled trial also showed statistically significant benefit. Seven other RCTs found lack of efficacy for daily EPA (216mgto5.4g)or DHA (132mgto3.6g) treatment. The remainder of the data supporting efficacy of oral fish oil treatment were based on uncontrolled trials, of which 6 of the 7 studies found significant benefit of oral fish oil. This seems to support their conclusion. However, the authors also state that fish oil was not shown to be effective at several examined doses and duration. Confused? Yes, me too!
Even more confusing is their failure to mention a single trial of Mahonia aquifolium. A 2013 meta-analysis published in the British Journal of Dermatology included 5 RCTs of Mahonia aquifolium which, according to these authors, provided ‘limited support’ for its effectiveness. How could they miss that?
More importantly, how could the reviewers miss to conduct a proper evaluation of the quality of the studies they included in their review (even in their abstract, they twice speak of ‘robust evidence’ – but how can they without assessing its robustness? [quantity is not remotely the same as quality!!!]). Without a transparent evaluation of the rigour of the primary studies, any review is nearly worthless.
Take the 12 acupuncture trials, for instance, which the review authors included based not on an assessment of the studies but on a dodgy review published in a dodgy journal. Had they critically assessed the quality of the primary studies, they could have not stated that CAM therapies with the most robust evidence of efficacy for treatment of psoriasis …[include]… acupuncture. Instead they would have had to admit that these studies are too dubious for any firm conclusion. Had they even bothered to read them, they would have found that many are in Chinese (which would have meant they had to be excluded in their review [as many pseudo-systematic reviewers, the authors only considered English papers]).
There might be a lesson in all this – well, actually I can think of at least two:
- Systematic reviews might well be the ‘Rolls Royce’ of clinical evidence. But even a Rolls Royce needs to be assembled correctly, otherwise it is just a heap of useless material.
- Even top journals do occasionally publish poor-quality and thus misleading reviews.