scientific misconduct
A ‘pragmatic, superiority, open-label, randomised controlled trial’ of sleep restriction therapy versus sleep hygiene has just been published in THE LANCET. Adults with insomnia disorder were recruited from 35 general practices across England and randomly assigned (1:1) using a web-based randomisation programme to either four sessions of nurse-delivered sleep restriction therapy plus a sleep hygiene booklet or a sleep hygiene booklet only. There was no restriction on usual care for either group. Outcomes were assessed at 3 months, 6 months, and 12 months. The primary endpoint was self-reported insomnia severity at 6 months measured with the insomnia severity index (ISI). The primary analysis included participants according to their allocated group and who contributed at least one outcome measurement. Cost-effectiveness was evaluated from the UK National Health Service and personal social services perspective and expressed in terms of incremental cost per quality-adjusted life year (QALY) gained. The trial was prospectively registered (ISRCTN42499563).
Between Aug 29, 2018, and March 23, 2020 the researchers randomly assigned 642 participants to sleep restriction therapy (n=321) or sleep hygiene (n=321). Mean age was 55·4 years (range 19–88), with 489 (76·2%) participants being female and 153 (23·8%) being male. 580 (90·3%) participants provided data for at least one outcome measurement. At 6 months, mean ISI score was 10·9 (SD 5·5) for sleep restriction therapy and 13·9 (5·2) for sleep hygiene (adjusted mean difference –3·05, 95% CI –3·83 to –2·28; p<0·0001; Cohen’s d –0·74), indicating that participants in the sleep restriction therapy group reported lower insomnia severity than the sleep hygiene group. The incremental cost per QALY gained was £2076, giving a 95·3% probability that treatment was cost-effective at a cost-effectiveness threshold of £20 000. Eight participants in each group had serious adverse events, none of which were judged to be related to intervention.
The authors concluded that brief nurse-delivered sleep restriction therapy in primary care reduces insomnia symptoms, is likely to be cost-effective, and has the potential to be widely implemented as a first-line treatment for insomnia disorder.
I am frankly amazed that this paper was published in a top journal, like THE LANCET. Let me explain why:
The verum treatment was delivered over four consecutive weeks, involving one brief session per week (two in-person sessions and two sessions over the phone). Session 1 introduced the rationale for sleep restriction therapy alongside a review of sleep diaries, helped participants to select bed and rise times, advised on management of daytime sleepiness (including implications for driving), and discussed barriers to and facilitators of implementation. Session 2, session 3, and session 4 involved reviewing progress, discussion of difficulties with implementation, and titration of the sleep schedule according to a sleep efficiency algorithm.
This means that the verum group received fairly extensive attention, while the control group did not. In other words, a host of non-specific effects are likely to have significantly influenced or even entirely determined the outcome. Despite this rather obvious limitation, the authors fail to discuss any of it. On the contrary, that claim that “we did a definitive test of whether brief sleep restriction therapy delivered in primary care is clinically effective and cost-effective.” This is, in my view, highly misleading and unworthy of THE LANCET. I suggest the conclusions of this trial should be re-formulated as follows:
The brief nurse-delivered sleep restriction, or the additional attention provided exclusively to the patients in the verum group, or a placebo-effect or some other non-specific effect reduced insomnia symptoms.
Alternatively, one could just conclude from this study that poor science can make it even into the best medical journals – a problem only too well known in the realm of so-called alternative medicine (SCAM).
We discussed the 2015 Australian NHMRC report on homeopathy many times before, e.g.:
- Homeopathy: the 2015 NHMRC report and its criticism re-analysed
- HOMEOPATHY: the NHMRC report revisited
- Ombudsman investigates ‘flawed’ homeopathic study
- The final verdict on homeopathy: it’s a placebo
In a nutshell, the report was an hugely influential analysis of the effectiveness of homeopathy which came to squarely negative conclusions. Thus it was celebrated as a thorough and conclusive piece evidence demonstrating the madness of homeopathy. Unsurprisingly, homeopaths did not like it at all and produced various criticisms claiming that it was neither thorough nor conclusive.
Now the final evaluation of what has been going on was finally published (ISSUED BY THE COMMONWEALTH OMBUDSMAN, IAIN ANDERSON, ON 4 AUGUST 2023):
The Office of the Commonwealth Ombudsman (the Office) has finalised an investigation relating to the National Health and Medical Research Council’s (NHMRC) review of the evidence for the effectiveness of homeopathy, conducted between 2010 and 2015. We commenced this investigation in September 2017 in response to concerns raised with us about how the NHMRC review had proceeded.
The Office conducts its investigations in private, and the Ombudsman generally does not make a public statement in the absence of a formal report. In the circumstances of this matter, including that the then-Ombudsman released a public statement on 4 June 2021 which acknowledged the Office was investigating, we believe it is important to share publicly the information we can, now that the investigation is complete.
Our investigation was finalised in July 2023. We acknowledge the length of time the investigation has taken. This is in part due to the extensive efforts the Office made to source independent scientific expertise to advise us on some detailed and specific questions of scientific methodology that were raised with our Office, including some that were only brought to our attention as our investigation progressed. Despite our best efforts, it was not possible to engage an expert (or experts) to provide independent advice to our Office on this subject. In the absence of independent, expert scientific expertise we have not been able to conclusively determine those matters of scientific methodology. This did not prevent our Office from forming a view on other aspects of the matter.
Our investigation did not result in any adverse findings about the review or the NHMRC. When finalising investigations, we may offer comments and suggestions to an agency about areas for future improvement. In this instance, we offered comments and suggestions to the NHMRC about how it records and publicly explains decisions about its activities. The NHMRC also independently made several improvements to its processes during the course of our investigation.
________________
In essence, this means that the conclusions of the report stand:
Homeopathy should not be used to treat health conditions that are chronic, serious, or could become serious. People who choose homeopathy may put their health at risk if they reject or delay treatments for which there is good evidence for safety and effectiveness. People who are considering whether to use homeopathy should first get advice from a registered health practitioner. Those who use homeopathy should tell their health practitioner and should keep taking any prescribed treatments.
Thus the matter is closed – that is closed for rational thinkers. For irrationalists, the matter will no doubt continue to be a stone of contention. No, homeopath will be able to accept these conclusions simply because a member of a cult ceases to be a cultist once he/she accepts the criticism agaist the cult.
Guest post by Norbert Aust, Udo Endruscheit, and Edzard Ernst
How do we know whether a treatment is reasonable or just some so-called alternative medicine (SCAM) that is at best useless? A simple answer is that the former is evidence-based, while the latter is not. But how can we tell the difference? High-quality studies, with independent replications or even a systematic review, are the sort of things we are usually looking for. But there is an underlying assumption, namely that, in science, bogus studies are prevented from polluting the scientific database or, if such trials have emerged, there are ways to identify and eliminate them.
And what if this assumption is wrong?
What if respectable universities and research organizations venture into the realm of pseudoscience either knowingly or because it had slipped their attention?
What if the editorial board of a top journal passes bogus studies to peer review?
What if such a paper is eventually reviewed by a proponent of the implausible therapy?
What if the readers of the article, once it is published, are too lethargic to object and do not write letters to the editor in protest?
And what if skeptics do formulate a protest but the journal editor refuses to publish it?
Well, if all the checks that should prevent faulty results from entering the scientific knowledge fail, we have fake evidence: a study that looks like sound science but that, in fact, is invalid. It is not hard to imagine what would happen if SCAM therapies are supported by seemingly respectable studies published in top journals. The fake evidence would accumulate as part of the body of evidence and eventually enter mainstream clinical practice, education, politics, etc., etc. Thus the reputation of bogus therapies would grow unjustifiably.
If you think this cannot happen, you are in the wrong. After the infamous study by Frass et al about homeopathy as an add-on treatment for lung cancer, another homeopathy paper was published in 2022 by Gaertner et al. in Pediatric Research (PR), a Medline-indexed journal with a two-year impact factor of 3.95 belonging to the nature-group of journals. According to this meta-analysis ‘individualized homeopathy showed a clinically relevant and statistically robust effect in the treatment of ADHD’. Shortly after the publication of this paper, we sent a letter to the editor to point out the shortcomings of this study. Here it is:
Sir,
with this letter we like to comment on the systematic review and meta-analysis on childhood ADHD by Gaertner et al. recently published in your journal.
First off, we are surprised, that your journal that is connected to nature does publish a paper on a treatment that has no a-priory probability at all and thus can only contain false positives if any. And this review is no exception as will be seen presently.
Our concerns are:
Out of the six studies included three were mere pilot trials (Fibert_2019, Jacobs_2005, Oberai_2013, ) which cannot provide any evidence for the shortcomings involved in pilots. Three of the six trials show severe issues in blinding (Fibert_2016, Fibert_2019, Oberai_2013), with two of them concerning both of the participants and the test personnel. This usually leads to massive bias in favour of the treatment [Zitat Cochrane Handbook].
Then we compared data from two trials with the data reported in the review and found some major misrepresentations:
(1) Jacobs et al. report an improvement in the T-score of their main outcome (CGI-P) of 4.1 for homeopathy and 9.1 for controls, that is placebo outperformed the homeopathic intervention. But the authors give an effect size of 0.272 in favour of homeopathy which is the opposite of the findings in the trial.
(2) Oberai et al. report effect sizes for their three main outcomes of 0.22, 0.59 and 0.54 (CPRS-R, CGISS, CGIIS repectively). There is no way that this yields a pooled effect size of 1.436 as given in the review.
We conclude that the positive result obtained by the authors is due to a combination of the inclusion of biased trials unsuitable to build evidence together with some major misreporting of study outcomes.
Our recommendation would be that the authors reconsider their review and improve their report. Maybe the editors would like to add a caution-notice to the paper – if not to withdraw it completely.
In June 2023, a full year after our submission, we were informed that Pediatric Research would not publish our criticism because the priority given to it was not sufficient to justify publication. But we were assured, that the journal would take the matter seriously, that they will investigate this matter and take appropriate editorial action. But as of today (End of June 2023) no expression of concern has been published.
Did the journal receive other comments or criticisms related to the paper in question? No, apparently there were none, at least none was published and the paper remains unchallenged to this day. This means that it might be taken for reliable evidence on the effectiveness of homeopathy and mislead patients, carers, practitioners, decision-makers, etc.
We feel this is unacceptable and therefore again wrote to the editors asking to reconsider their decision. Here is our letter:
Dear …
together with my co-authors we would like to comment your decision about our letter to the editor about an extremely faulty and misleading paper that may well create harm to patients. In fact we find it very hard to accept your decision not to publish our comment.
We understand that Pediatric Research is a high impact journal with a 2-year IF of nearly 4. Your journal is member of COPE and is indexed with quite a few first rank institutions. By all standards, any reader will be convinced that a paper published in Pediatric Research is based on solid research and the results are derived by rigorous methodology and are as reliable as can be. Especially if this paper remains unchallenged by any reader’s comments for a full year after publication. This is your responsibility to the scientific community. And to the children that might receive treatment based on knowledge spread through your journal.
How then can it be, that an article about homeopathy, a thoroughly implausible lore, in the treatment of ADHD is published in Pediatric Research, where the authors come to the conclusion “that individualized homeopathy showed a clinically relevant and statistically robust effect in the treatment of ADHD”?
In our comment we point out that the authors made a lot of errors – to say it mildly. They deny the doubtful quality of the studies they included in their meta-analysis, they did not stick to their own exclusion criteria, the data the authors report do not resemble the findings of the studies they were allegedly taken from, the one study setting the results is a mere pilot study.
The reason you give for our letter not being published is that it was not given enough priority to justify publication. We would like to know: Which issues can conceivably receive higher priority than the fact that a paper in your journal is downright wrong and misleading?
What do you need to deem a comment important? Up to now the paper is unchallenged by any reader’s comments, so apparently there was no other letter to the editor that might be given higher priority than ours.
We ask you to review your decision, or better still, consider a retraction of the paper altogether. If so, an expression of concern should be issued at once. After all, the COPE-guidelines for retraction state “clear evidence, that the findings are unreliable, either as a result of major error (…), or as a result of fabrication (…) or faslification (…)’ as a reason to consider retraction.
Otherwise the malpractice of homeopathy will have a first class evidence that will be helpful to promote homeopathy to parents and their children.
Watch this space!
This analysis was aimed at quantifying how many studies registered on the Open Science Framework (OSF) up to November 2017 are performed but not shared after at least 4 years. Examining a sample of 315 registrations, of which 169 were research studies, the researchers found that 104 (62%) were published. They estimated that 5550 out of 9544 (58%) registered studies on the OSF are published.
Researchers use registries to make unpublished studies public, and the OSF policy to open registrations after a four-year embargo substantially increases the number of studies that become known to the scientific community. In responses to emails asking researchers why studies remained unpublished logistical issues (e.g., lack of time, researchers changing jobs) were the most common cause, followed by null results, and rejections during peer review.
The authors concluded that their study shows that a substantial amount of studies researchers perform remain unpublished.
I find this truly shocking!
Researchers are able to do research only because they receive financial and other support from elsewhere. Therefore they have an ethical obligation to publish it. The reasons frequently given for not publishing research are nothing well and truly invalid:
- Lack of time is a mere excuse; if researchers had the time to get the grants, permissions, etc. they simply must have the time to finish the job properly.
- Researchers changing jobs is an equally flawed excuse; if someone changes position, he/she is obliged to finish the job they were doing. A surgeon can also not leave mid-surgery because he has a better offer.
- ‘Null results’ is even worse as a reason. Null results are just as important as positive findings – occasionally they are even more important. If researchers fail to realize this, they simply disqualify themselves as researchers.
- ‘Rejections during peer review’ is complete nonsense. Everyone who submits papers for publication gets rejected once in a while. In this case, one learns from the peer-review comments, improves the paper in question, and re-submits it to another journal.
I have seen many studies of so-called alternative medicine (SCAM) that, for this or that reason, never were published. And I feel strongly that this is a serious violation of research ethics – so much so that I would ban researchers who are guilty of this crime from conducting research in the future. I also feel that, in order to receive the necessary support (financial and other), researchers should sign that they will publish their findings within a given time after finishing their study. Failing to comply could then incur a penalty such as paying back part of the funds wasted. I think such measures would very quickly clear up the current intolerable situation.
In the comments section, someone recently alerted us to a most remarkable article. I had a look at it and thought it would be a pity to let it pass without further comment. Here is the abstract:
There are many types of energy around us, including natural and artificial ones, the first of the ground energies due to the imbalance happened from the treatment of man with the ground (mines-the bases of huge buildings); the result of the Earth rotation, the result of geological faults, the flow of groundwater or energies resulting from other factors that result in radiations that harm organisms in general. Also we are continuously increasing the amount of carrier waves needed for the wireless technology of modern communication in the earth’s atmosphere every day. These electromagnetic waves are thousands of times stronger than the level used in the communication in our body cells. The problem is not the saturation of the earth’s atmosphere through quantity, but also a detrimental quality. Even people who avoid using high technology are not immune. No one is immune because these are carrier waves with penetrating properties. our immune systems are continuously trying to correct the distortion in the transfer of inner information in our body; very soon the threshold will be reached when a total collapse of our body defenses will take place. Balancing the activities of daily life, achieving harmony with our inner and outer environments, humanizing modern technology, integrating science and spirits, and discovering the unified scientific reality behind all religions is the work of some science such as Bio Geometry, Bio Design, Radiesthesia, …ext.
When one runs a blog on so-called alternative medicine (SCAM), it is almost inevitable to run into plenty of bullshit. Thus, over the years, I have gotten used to even the most compact versions of it. Yet, this paper – I do recommend you have a glance also at the full text – is truly outstanding.
In case there is someone amongst my readers who understands what the author wants to express, I would be most obliged to learn.
‘Spagyric’ is a so-called alternative medicine (SCAM) based on the alchemy of Paracelsus (1493-1541). Paracelsus borrowed the term from “separate” (spao) and “combine” (ageiro) to indicate that spagyric preparations are based on the “separation”, “extraction” and “recombination” of the active ingredients of a substance. Plant, mineral as well as animal source materials are used.
The production of spagyric remedies is based on a complex process of maceration and fermentation of a plant extract in alcohol. It takes place in dark, thick-walled glass flasks that are hermetically sealed and kept at a controlled temperature of 37 °C for 28 days. The tincture thus obtained is then decanted and the drug residue is removed from the solution, completely dried, and burned to ash to recover the inorganic components of the plant material. The ash is subsequently dissolved in the alcoholic solution of maceration, and the finished spagyric preparation is left for 12 days before use.
Spagyric is not the most popular of all SCAMs but it certainly does have a significant following. One enthusiast claims that “spagyric essences work on a vibrational level in their action upon the emotional/mind and physical spheres and can be employed in numerous situations. Most people seek help to relieve physical symptoms. Even so, it is often necessary to address the emotional and psychological aspects which may predispose the illness or imbalance. In an era where many people are experiencing life-changing events, the ability to transition smoothly is essential for well-being and vitality. Guidance and help are required to maintain homeostasis. These medicines can help the patient to understand the root cause of their illness and learn to regain control of their lives. Some medicine systems appear to be less effective than in previous times. It has been suggested that the energetic frequency of both the earth and human organism are changing. Therefore these systems may no longer be a vibrational match for the changing frequencies. Spagyric Medicine is designed to ‘tune in with’ these current frequencies. Research suggests that the Spagyric essences may instigate improved health by energetically influencing DNA.”
After reading such weird statements, I ask myself, is there any evidence that spagyric remedies work? In my search for robust studies, I was unsuccessful. There does not seem to be a single controlled study on the subject. However, there are fragmentary reports of a study initiated and conducted by a now largely unknown healer named Karl Hann von Weyhern.
Von Weyhern (1882 – 1954) had taken a few semesters of pharmacy and medicine in Freiburg but remained without a degree. In 1930, he became a member of the NSDAP (Hitler’s Nazi party) and in 1940 he joined the SS. Around 1935, he settled in Munich as a non-medical practitioner (Heilpraktiker), and Heinrich Himmler who has a soft spot for SCAM enlisted as one of his patients. By then von Weyhern had by then made a steep career in the Nazi hierarchy, and he managed to convince Himmler that his spagyric remedies could cure tuberculosis, which was still rampant at the time. They decided to carry out experiments in this regard in the Dachau concentration camp.
Thus, von Weyhern was allowed to test spagyric remedies on forcibly recruited concentration camp prisoners. These experiments lasted for about one year and included around 150 patients who, according to von Weyhern’s iridology diagnosis, suffered from tuberculosis. Half of them were treated with spagyric remedies and the others with conventional treatments. At the end of the experiment, 27 persons were reportedly released into everyday concentration camp life as ‘fit for work’. How many of the 150 prisoners lost their lives due to these experiments is not known. Von Weyhern never filed a final report. It is to be feared that the death toll was considerable. [1]
After the war, von Weyhern denied belonging to the SS, claimed that he had ‘sacrificed himself’ for his patients in the concentration camp, merely had to pay a fine, and was ‘denazified’ in 1948. Subsequently, he resumed his work as a ‘Heilpraktiker’ in Olching, a village near Dachau. [1]
Of course, these infamous experiments cannot be blamed on spagyric medicine. Yet, I feel they are nevertheless important, not least because they seem to reveal the only thing remotely resembling something like evidence.
[1] Die Ärzte der Nazi-Führer: Karrieren und Netzwerke : Mathias Schmidt (Hg.), Dominik Groß (Hg.), Jens Westemeier (Hg.): Amazon.de: BooksThere are debates in acupuncture-related systematic reviews and meta-analyses on whether searching Chinese databases to get more Chinese-language studies may increase the risk of bias and overestimate the effect size, and whether the treatment effects of acupuncture differ between Chinese and non-Chinese populations.
For this meta-epidemiological study, a team of investigators searched the Cochrane Library from its inception until December 2021, and identified systematic reviews and meta-analyses with acupuncture as one of the interventions. Paired reviewers independently screened the reviews and extracted the information. They repeated the meta-analysis of the selected outcomes to separately pool the results of Chinese- and non-Chinese-language acupuncture studies and presented the pooled estimates as odds ratios (OR) with 95% confidence interval (CI). They calculated the Ratio of ORs (ROR) by dividing the OR of the Chinese-language trials by the OR of the non-Chinese-language trials, and the ROR by dividing the OR of trials addressing Chinese population by the OR of trials addressing non-Chinese population. The researchers thus explored whether the impact of a high risk of bias on the effect size differed between studies published in Chinese- and in non-Chinese-language, and whether the treatment effects of acupuncture differed between Chinese and non-Chinese populations.
The researchers identified 84 Cochrane acupuncture reviews involving 33 Cochrane groups, of which 31 reviews (37%) searched Chinese databases. Searching versus not searching Chinese databases significantly increased the contribution of Chinese-language literature both to the total number of included trials (54% vs. 15%) and the sample size (40% vs. 15%). When compared with non-Chinese-language trials, Chinese-language trials were associated with a larger effect size (pooled ROR 0.51, 95% CI 0.29 to 0.91). The researchers also observed a higher risk of bias in Chinese-language trials in blinding of participants and personnel (97% vs. 51%) and blinding of outcome assessment (93% vs. 47%). The higher risk of bias was associated with a larger effect estimate in both Chinese language (allocation concealment: high/unclear risk vs. low risk, ROR 0.43, 95% CI 0.21 to 0.87) and non-Chinese-language studies (blinding of participants and personnel: high/unclear risk vs. low risk, ROR 0.41, 95% CI 0.23 to 0.74). However, the team found no evidence that the higher risk of bias would increase the effect size of acupuncture in Chinese-language studies more often than in non-Chinese-language studies (the confidence intervals of all ROR in the high-risk group included 1, Table 3). The researchers further found acupuncture appeared to be more effective in Chinese than in non-Chinese populations.
The authors concluded that the findings of this study suggest the higher risk of bias may lead to an overestimation of the treatment effects of acupuncture but would not increase the treatment effects in Chinese-language studies more often than in other language studies. The difference in treatment effects of acupuncture was probably associated with differences in population characteristics.
The authors discuss that, although searching Chinese databases can substantially increase the number of eligible studies and sample size in acupuncture reviews, the potentially higher risk of bias is an argument that needs to be considered in the inclusion of Chinese-language studies. Patients, investigators, and guideline panels should be cautious when adopting evidence from acupuncture reviews where studies with a high risk of bias contributed with a high weight to the meta-analysis.
The authors observed larger treatment effects of acupuncture in Chinese-language studies than in studies published in other languages. Although the treatment effects of acupuncture tended to be greater in studies with a high risk of bias, this potential overestimation did not differ between studies published in Chinese and in other languages. In other words, the larger treatment effects in Chinese-language studies cannot be explained by a high risk of bias. Furthermore, our study found acupuncture to be more effective in Chinese populations than in other populations, which could at least partly explain the larger treatment effects observed in Chinese-language studies.
I feel that this analysis obfuscates more than it clarifies. As we have discussed often here, acupuncture studies by Chinese researchers (regardless of what language they are published in) hardly ever report negative results, and their findings are often fabricated. It, therefore, is not surprising that their effect sizes are larger than those of other trials.
The only sensible conclusion from this messy and regrettable situation, in my view, is to be very cautious and exclude them from systematic reviews.
How often do we hear that chiropractic is safe because numerous trials reported no adverse events? This systematic review tested whether there has been a change in the reporting of adverse events associated with spinal manipulation in randomized clinical trials (RCTs) since 2016.
Databases were searched from March 2016 to May 2022: MEDLINE (Ovid), Embase, CINAHL, ICL, PEDro, and Cochrane Library. Domains of interest (pertaining to adverse events) included: completeness and location of reporting; nomenclature and description; spinal location and practitioner delivering manipulation; methodological quality of the studies and details of the publishing journal. Frequencies and proportions of studies reporting on each of these domains were calculated. Univariable and multivariable logistic regression models were fitted to examine the effect of potential predictors on the likelihood of studies reporting on adverse events.
5399 records were identified by the electronic searches, of which 154 (2.9%) were included in the analysis. Of these, 94 (61.0%) reported adverse events with only 23.4% providing an explicit description of what constituted an adverse event. Reporting of adverse events in the abstract has increased (n=29, 30.9%) while reporting in the results section has decreased (n=83, 88.3%) over the past 6 years. Spinal manipulation was delivered to 7518 participants in the included studies. No serious adverse events were reported in any of these studies.
The authors concluded as follows: while the current level of reporting of adverse events associated with spinal manipulation in RCTs has increased since our 2016 publication on the same topic, the level remains low and inconsistent with established standards. As such, it is imperative for authors, journal editors and administrators of clinical trial registries to ensure there is more balanced reporting of both benefits and harms in RCTs involving spinal manipulation.
This article is clearly relevant to our discussions about adverse events after spinal manipulation. However, I find it far too uncritical. This might be due to the affiliations of some of the authors:
- Integrative Spinal Research Group, Department of Chiropractic Medicine, University Hospital Balgrist and University of Zurich, Zurich, Switzerland.
- Department of Chiropractic, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, New South Wales, Australia.
Interestingly, the authors stated that they have no conflict of interest. Also interesting is the fact that they do not cite our paper from 2012. I, therefore, take the liberty of doing it:
Objective: To systematically review the reporting of adverse effects in clinical trials of chiropractic manipulation.
Data sources: Six databases were searched from 2000 to July 2011. Randomised clinical trials (RCTs) were considered, if they tested chiropractic manipulations against any control intervention in human patients suffering from any type of clinical condition. The selection of studies, data extraction, and validation were performed independently by two reviewers.
Results: Sixty RCTs had been published. Twenty-nine RCTs did not mention adverse effects at all. Sixteen RCTs reported that no adverse effects had occurred. Complete information on incidence, severity, duration, frequency and method of reporting of adverse effects was included in only one RCT. Conflicts of interests were not mentioned by the majority of authors.
Conclusions: Adverse effects are poorly reported in recent RCTs of chiropractic manipulations.
In percentage terms the results are similar. What is very different is that the authors of the new paper merely lament that the level remains low and inconsistent with established standards, while we make it clear in the abstract that adverse effect reporting is poor and in the paper identify this deficit as a violation against research ethics and thus as a form of scientific misconduct.
In view of all this, let me re-phrase the last sentence of the authors’ conclusion:
it is imperative for authors, journal editors, and administrators of clinical trial registries to ensure that researchers adhere to accepted ethical standards and that scientific misconduct no longer gets published.
This meta-analysis aimed “to provide better evidence of the efficacy of manual therapy (MT) on adolescent idiopathic scoliosis (AIS)”. 
All RCTs of MT for the management of patients with AIS were included in the present study. The treatment difference between the experimental and control group was mainly MT. The outcomes consisted of the total effective rate, the Cobb angle, and Scoliosis Research Society-22 (SRS-22) questionnaire score. Electronic database searches were conducted from database inception to July 2022, including the Cochrane Library, PubMed, Web of Science, Embase, Wanfang Data, CNKI, and VIP. The pooled data were analyzed using RevMan 5.4 software.
Four RCTs with 213 patients in the experimental groups were finally included. There are 2 studies of standalone MT in the experimental group and 3 studies of MT with identical conservative treatments in the control group. Three trials reported the total effective rate and a statistically significant difference was found (P = 0.004). Three trials reported Cobb angle; a statistical difference was found (P = 0.01). Then, sensitivity analysis showed that there was a significant difference in the additional MT subgroup (P < 0.00001) while not in the standalone MT subgroup (P = 0.41). Three trials reported SRS-22 scores (P = 0.55) without significant differences.
The authors concluded that there is insufficient data to determine the effectiveness of spinal manipulation limited by the very low quality of included studies. High-quality studies with appropriate design and follow-up periods are warranted to determine if MT may be beneficial as an adjunct therapy for AIS. Currently, there is no evidence to support spinal manipulation. 
The treatment of idiopathic scoliosis depends on the age, curve size, and progression of the condition. Therapeutic options include observation, bracing, physiotherapy, and surgery. They do NOT include MT because it is neither a plausible nor effective solution to this problem. It follows that further studies are not warranted and should be discouraged.
And, even if you disagree with me here and feel that further studies might be justified, let me remind you that proper research is never aimed at providing better evidence that a therapy works (as the authors of this odd paper seem to think); it must be aimed at testing whether it is effective!
Sure, the LP is dangerous nonsense, but this begs the question of whether so-called alternative medicine (SCAM) has anything to offer for patients suffering from ME/CFS. If the LP story tells us anything, then it must be this: we should not trust single trials, particularly if they seem dodgy. In other words, we should look at systematic reviews that synthesize ALL clinical trials and evaluate them critically.
To locate this type of evidence I conducted several Medline searches and found several recent systematic reviews that address the issue:
Context: A variety of interventions have been used in the treatment and management of chronic fatigue syndrome (CFS). Currently, debate exists among health care professionals and patients about appropriate strategies for management.
Objective: To assess the effectiveness of all interventions that have been evaluated for use in the treatment or management of CFS in adults or children.
Data sources: Nineteen specialist databases were searched from inception to either January or July 2000 for published or unpublished studies in any language. The search was updated through October 2000 using PubMed. Other sources included scanning citations, Internet searching, contacting experts, and online requests for articles.
Study selection: Controlled trials (randomized or nonrandomized) that evaluated interventions in patients diagnosed as having CFS according to any criteria were included. Study inclusion was assessed independently by 2 reviewers. Of 350 studies initially identified, 44 met inclusion criteria, including 36 randomized controlled trials and 8 controlled trials.
Data extraction: Data extraction was conducted by 1 reviewer and checked by a second. Validity assessment was carried out by 2 reviewers with disagreements resolved by consensus. A qualitative synthesis was carried out and studies were grouped according to type of intervention and outcomes assessed.
Data synthesis: The number of participants included in each trial ranged from 12 to 326, with a total of 2801 participants included in the 44 trials combined. Across the studies, 38 different outcomes were evaluated using about 130 different scales or types of measurement. Studies were grouped into 6 different categories. In the behavioral category, graded exercise therapy and cognitive behavioral therapy showed positive results and also scored highly on the validity assessment. In the immunological category, both immunoglobulin and hydrocortisone showed some limited effects but, overall, the evidence was inconclusive. There was insufficient evidence about effectiveness in the other 4 categories (pharmacological, supplements, complementary/alternative, and other interventions).
Conclusions: Overall, the interventions demonstrated mixed results in terms of effectiveness. All conclusions about effectiveness should be considered together with the methodological inadequacies of the studies. Interventions which have shown promising results include cognitive behavioral therapy and graded exercise therapy. Further research into these and other treatments is required using standardized outcome measures.
Introduction: Chronic fatigue syndrome (CFS) affects between 0.006% and 3% of the population depending on the criteria of definition used, with women being at higher risk than men.
Methods and outcomes: We conducted a systematic review and aimed to answer the following clinical question: What are the effects of treatments for chronic fatigue syndrome? We searched: Medline, Embase, The Cochrane Library, and other important databases up to March 2010 (Clinical Evidence reviews are updated periodically; please check our website for the most up-to-date version of this review). We included harms alerts from relevant organisations such as the US Food and Drug Administration (FDA) and the UK Medicines and Healthcare products Regulatory Agency (MHRA).
Results: We found 46 systematic reviews, RCTs, or observational studies that met our inclusion criteria. We performed a GRADE evaluation of the quality of evidence for interventions.
Conclusions: In this systematic review we present information relating to the effectiveness and safety of the following interventions: antidepressants, cognitive behavioural therapy (CBT), corticosteroids, dietary supplements, evening primrose oil, galantamine, graded exercise therapy, homeopathy, immunotherapy, intramuscular magnesium, oral nicotinamide adenine dinucleotide, and prolonged rest.
Background: Throughout the world, patients with chronic diseases/illnesses use complementary and alternative medicines (CAM). The use of CAM is also substantial among patients with diseases/illnesses of unknown aetiology. Chronic fatigue syndrome (CFS), also termed myalgic encephalomyelitis (ME), is no exception. Hence, a systematic review of randomised controlled trials of CAM treatments in patients with CFS/ME was undertaken to summarise the existing evidence from RCTs of CAM treatments in this patient population.
Methods: Seventeen data sources were searched up to 13th August 2011. All randomised controlled trials (RCTs) of any type of CAM therapy used for treating CFS were included, with the exception of acupuncture and complex herbal medicines; studies were included regardless of blinding. Controlled clinical trials, uncontrolled observational studies, and case studies were excluded.
Results: A total of 26 RCTs, which included 3,273 participants, met our inclusion criteria. The CAM therapy from the RCTs included the following: mind-body medicine, distant healing, massage, tuina and tai chi, homeopathy, ginseng, and dietary supplementation. Studies of qigong, massage and tuina were demonstrated to have positive effects, whereas distant healing failed to do so. Compared with placebo, homeopathy also had insufficient evidence of symptom improvement in CFS. Seventeen studies tested supplements for CFS. Most of the supplements failed to show beneficial effects for CFS, with the exception of NADH and magnesium.
Conclusions: The results of our systematic review provide limited evidence for the effectiveness of CAM therapy in relieving symptoms of CFS. However, we are not able to draw firm conclusions concerning CAM therapy for CFS due to the limited number of RCTs for each therapy, the small sample size of each study and the high risk of bias in these trials. Further rigorous RCTs that focus on promising CAM therapies are warranted.
Background: There is no curative treatment for chronic fatigue syndrome (CFS). Traditional Chinese medicine (TCM) is widely used in the treatment of CFS in China.
Objective: To evaluate the effectiveness and safety of TCM for CFS.
Methods: The protocol of this review is registered at PROSPERO. We searched six main databases for randomized clinical trials (RCTs) on TCM for CFS from their inception to September 2013. The Cochrane risk of bias tool was used to assess the methodological quality. We used RevMan 5.1 to synthesize the results.
Results: 23 RCTs involving 1776 participants were identified. The risk of bias of the included studies was high. The types of TCM interventions varied, including Chinese herbal medicine, acupuncture, qigong, moxibustion, and acupoint application. The results of meta-analyses and several individual studies showed that TCM alone or in combination with other interventions significantly alleviated fatigue symptoms as measured by Chalder’s fatigue scale, fatigue severity scale, fatigue assessment instrument by Joseph E. Schwartz, Bell’s fatigue scale, and guiding principle of clinical research on new drugs of TCM for fatigue symptom. There was no enough evidence that TCM could improve the quality of life for CFS patients. The included studies did not report serious adverse events.
Conclusions: TCM appears to be effective to alleviate the fatigue symptom for people with CFS. However, due to the high risk of bias of the included studies, larger, well-designed studies are needed to confirm the potential benefit in the future.
Background: As the etiology of chronic fatigue syndrome (CFS) is unclear and the treatment is still a big issue. There exists a wide range of literature about acupuncture and moxibustion (AM) for CFS in traditional Chinese medicine (TCM). But there are certain doubts as well in the effectiveness of its treatment due to the lack of a comprehensive and evidence-based medical proof to dispel the misgivings. Current study evaluated systematically the effectiveness of acupuncture and moxibustion treatments on CFS, and clarified the difference among them and Chinese herbal medicine, western medicine and sham-acupuncture.
Methods: We comprehensively reviewed literature including PubMed, EMBASE, Cochrane library, CBM (Chinese Biomedical Literature Database) and CNKI (China National Knowledge Infrastructure) up to May 2016, for RCT clinical research on CFS treated by acupuncture and moxibustion. Traditional direct meta-analysis was adopted to analyze the difference between AM and other treatments. Analysis was performed based on the treatment in experiment and control groups. Network meta-analysis was adopted to make comprehensive comparisons between any two kinds of treatments. The primary outcome was total effective rate, while relative risks (RR) and 95% confidence intervals (CI) were used as the final pooled statistics.
Results: A total of 31 randomized controlled trials (RCTs) were enrolled in analyses. In traditional direct meta-analysis, we found that in comparison to Chinese herbal medicine, CbAM (combined acupuncture and moxibustion, which meant two or more types of acupuncture and moxibustion were adopted) had a higher total effective rate (RR (95% CI), 1.17 (1.09 ~ 1.25)). Compared with Chinese herbal medicine, western medicine and sham-acupuncture, SAM (single acupuncture or single moxibustion) had a higher total effective rate, with RR (95% CI) of 1.22 (1.14 ~ 1.30), 1.51 (1.31-1.74), 5.90 (3.64-9.56). In addition, compared with SAM, CbAM had a higher total effective rate (RR (95% CI), 1.23 (1.12 ~ 1.36)). In network meta-analyses, similar results were recorded. Subsequently, we ranked all treatments from high to low effective rate and the order was CbAM, SAM, Chinese herbal medicine, western medicine and sham-acupuncture.
Conclusions: In the treatment of CFS, CbAM and SAM may have better effect than other treatments. However, the included trials have relatively poor quality, hence high quality studies are needed to confirm our finding.
Objectives: This meta-analysis aimed to assess the effectiveness and safety of Chinese herbal medicine (CHM) in treating chronic fatigue syndrome (CFS). Methods: Nine electronic databases were searched from inception to May 2022. Two reviewers screened studies, extracted the data, and assessed the risk of bias independently. The meta-analysis was performed using the Stata 12.0 software. Results: Eighty-four RCTs that explored the efficacy of 69 kinds of Chinese herbal formulas with various dosage forms (decoction, granule, oral liquid, pill, ointment, capsule, and herbal porridge), involving 6,944 participants were identified. This meta-analysis showed that the application of CHM for CFS can decrease Fatigue Scale scores (WMD: -1.77; 95%CI: -1.96 to -1.57; p < 0.001), Fatigue Assessment Instrument scores (WMD: -15.75; 95%CI: -26.89 to -4.61; p < 0.01), Self-Rating Scale of mental state scores (WMD: -9.72; 95%CI:-12.26 to -7.18; p < 0.001), Self-Rating Anxiety Scale scores (WMD: -7.07; 95%CI: -9.96 to -4.19; p < 0.001), Self-Rating Depression Scale scores (WMD: -5.45; 95%CI: -6.82 to -4.08; p < 0.001), and clinical symptom scores (WMD: -5.37; 95%CI: -6.13 to -4.60; p < 0.001) and improve IGA (WMD: 0.30; 95%CI: 0.20-0.41; p < 0.001), IGG (WMD: 1.74; 95%CI: 0.87-2.62; p < 0.001), IGM (WMD: 0.21; 95%CI: 0.14-0.29; p < 0.001), and the effective rate (RR = 1.41; 95%CI: 1.33-1.49; p < 0.001). However, natural killer cell levels did not change significantly. The included studies did not report any serious adverse events. In addition, the methodology quality of the included RCTs was generally not high. Conclusion: Our study showed that CHM seems to be effective and safe in the treatment of CFS. However, given the poor quality of reports from these studies, the results should be interpreted cautiously. More international multi-centered, double-blinded, well-designed, randomized controlled trials are needed in future research.
What does all that tell us?
Disappointingly, it tells me that SCAM has preciously little to offer for ME/CFS patients.
But what about the TCM treatments? Aren’t the above reviews quite positive TCM?
Yes, they are but I nevertheless recommend taking them with a healthy pinch of salt.
Why?
Because we have seen many times before that, for a range of reasons, Chinese researchers of TCM draw false positive conclusions. That may sound unfair, harsh, or even racist, but I think it’s true. If you disagree, please show me a couple of systematic reviews of TCM for any human disease by Chinese researchers that have drawn negative conclusions.
And what is my advice to patients suffering from ME/CSF?
I think the best I can offer is this: be very cautious about the many claims made by SCAM enthusiasts; if it sounds too good to be true, it probably is!
