scientific misconduct
Exercise is often cited as a major factor contributing to improved cognitive functioning. As a result, the relationship between exercise and cognition has received much attention in scholarly literature. Systematic reviews and meta-analyses present varying and sometimes conflicting results about the extent to which exercise can influence cognition. The aim of this umbrella review was to summarize the effects of physical exercise on cognitive functions (global cognition, executive function, memory, attention, or processing speed) in healthy adults ≥ 55 years of age.
This review of systematic reviews with meta-analyses invested the effect of exercise on cognition. Databases (CINAHL, Cochrane Library, MEDLINE, PsycInfo, Scopus, and Web of Science) were searched from inception until June 2023 for reviews of randomized or non-randomised controlled trials. Full-text articles meeting the inclusion criteria were reviewed and methodological quality assessed. Overlap within included reviews was assessed using the corrected covered area method (CCA). A random effects model was used to calculate overall pooled effect size with sub-analyses for specific cognitive domains, exercise type and timing of exercise.
A total of 20 met the inclusion criteria. They were based on 332 original primary studies. Overall quality of the reviews was considered moderate with most meeting 8 or more of the 16 AMSTAR 2 categories. Overall pooled effects indicated that exercise in general has a small positive effect on cognition (d = 0.22; SE = 0.04; p < 0.01). Mind–body exercise had the greatest effect with a pooled effect size of (d = 0.48; SE = 0.06; p < 0.001). Exercise had a moderate positive effect on global cognition (d = 0.43; SE = 0,11; p < 0,001) and a small positive effect on executive function, memory, attention, and processing speed. Chronic exercise was more effective than acute exercise. Variation across studies due to heterogeneity was considered very high.
The authors concluded that mind–body exercise has moderate positive effects on the cognitive function of people aged 55 or older. To promote healthy aging, mind–body exercise should be used over a prolonged period to complement other types of exercise. Results of this review should be used to inform the development of guidelines to promote healthy aging.
It seems to me that the umbrella review hides the crucial fact that many of the primary studies had major flaws, e.g. in terms of:
- lack of randomisation,
- lack of blinding.
Eleven studies investigated the effects of aerobic exercise on cognition. Only three studies investigated the effects of mind body exercise on cognition, two analysed the effects of resistance exercise, and five investigated the effects of mixed exercise interventions. I am therefore mystified how the authors managed to arrive at such a hyped conclusion in favour of the effectiveness of mind body exercises. Even an optimistic interpretation of the data would allow merely a weak indication that a positive effect might exist. To state that mind body exercises should be promoted for ‘healthy aging’ borders on the irresponsible, in my view. Surely even the most naive researcher must see that, for such a far-reaching recommendation, we would need much more solid evidence.
I strongly suspect that a proper review of the primary studies of mind body exercise with a critical evaluation of the quality of the primary studies would lead to dramatically different conclusion.
This study aimed to compare the effectiveness of three distinct interventions – Yoga, Naturopathy, and Conventional medical management – in alleviating pain, reducing disability, enhancing spinal mobility, and improving the quality of life in individuals with low back pain. Ninety participants were recruited and randomly divided into three groups.
- The first group (group 1) received Yoga,
- the second group (group 2) received Naturopathy treatments,
- the third group served as the control and received conventional medications.
Visual Analogue Scale (VAS) scores, Oswestry Disability Index (ODI), Flexion Test-Finger to Floor Test (FTFT) results, and Quality of Life (QOL) were assessed at baseline and after a 10-day intervention period for all groups.
Overall comparisons between the groups, utilizing ANOVA, revealed marked differences in pain severity, disability index, daily functional capacity, and Quality of Life (QoL) improvements following respective interventions. Substantial improvements were also noted within the yoga and naturopathic medicine groups across multiple variables.
The authors concluded that the results of this comparative analysis emphasize the effectiveness of Yoga, Naturopathy, and Conventional Medical Treatment in managing low back pain. All three interventions demonstrated significant improvements in pain intensity, disability, spinal mobility, and quality of life. This study contributes valuable insights into the diverse therapeutic approaches for low back pain management, highlighting the potential of holistic and alternative treatments to enhance patients’ well-being.
__________________
This is a remarkably poor study. Its flaws are too numerous to account for them all here. Let me focus on just a three that stand out.
- All we learn about the 3 treatment regimen is this (and it clearly not enough to do an independent replication of this trial):
Yoga Group:
Participants in the Yoga Group underwent a specifically designed integrated approach of Yoga therapy (IAYT) for back pain, incorporating relaxation techniques, spinal movements, breathing exercises, pranayama, and deep relaxation techniques. The intervention was conducted by qualified yoga instructors at SDM College of Naturopathy and Yogic Sciences.
Naturopathy Group:
Participants in the Naturopathy Group received neutral spinal baths and partial massages. The spinal bath was administered at Government Yoga & Nature Cure Out Patient Center, Puttur, and massages were performed by trained naturopathy therapists.
Conventional Medicine Group:
Participants in the Conventional Medicine Group received standard medical treatments for low back pain as recommended by orthopedic physicians from S.D.M Medical College, Dharward
- As an equivalence trial, the sample size of this study is far too small. This means that its findings are most likely caused by coincidence and not by the interventions applied.
- There was no attempt of blinding the patients. Therefore, the results – if they were otherwise trustworthy – would be dominated by expectations and not by the effects of the treatments.
Altogether, this study is, I think, a good example for the fact that
poor research often is worse than no research at all.
A ‘pragmatic, superiority, open-label, randomised controlled trial’ of sleep restriction therapy versus sleep hygiene has just been published in THE LANCET. Adults with insomnia disorder were recruited from 35 general practices across England and randomly assigned (1:1) using a web-based randomisation programme to either four sessions of nurse-delivered sleep restriction therapy plus a sleep hygiene booklet or a sleep hygiene booklet only. There was no restriction on usual care for either group. Outcomes were assessed at 3 months, 6 months, and 12 months. The primary endpoint was self-reported insomnia severity at 6 months measured with the insomnia severity index (ISI). The primary analysis included participants according to their allocated group and who contributed at least one outcome measurement. Cost-effectiveness was evaluated from the UK National Health Service and personal social services perspective and expressed in terms of incremental cost per quality-adjusted life year (QALY) gained. The trial was prospectively registered (ISRCTN42499563).
Between Aug 29, 2018, and March 23, 2020 the researchers randomly assigned 642 participants to sleep restriction therapy (n=321) or sleep hygiene (n=321). Mean age was 55·4 years (range 19–88), with 489 (76·2%) participants being female and 153 (23·8%) being male. 580 (90·3%) participants provided data for at least one outcome measurement. At 6 months, mean ISI score was 10·9 (SD 5·5) for sleep restriction therapy and 13·9 (5·2) for sleep hygiene (adjusted mean difference –3·05, 95% CI –3·83 to –2·28; p<0·0001; Cohen’s d –0·74), indicating that participants in the sleep restriction therapy group reported lower insomnia severity than the sleep hygiene group. The incremental cost per QALY gained was £2076, giving a 95·3% probability that treatment was cost-effective at a cost-effectiveness threshold of £20 000. Eight participants in each group had serious adverse events, none of which were judged to be related to intervention.
The authors concluded that brief nurse-delivered sleep restriction therapy in primary care reduces insomnia symptoms, is likely to be cost-effective, and has the potential to be widely implemented as a first-line treatment for insomnia disorder.
I am frankly amazed that this paper was published in a top journal, like THE LANCET. Let me explain why:
The verum treatment was delivered over four consecutive weeks, involving one brief session per week (two in-person sessions and two sessions over the phone). Session 1 introduced the rationale for sleep restriction therapy alongside a review of sleep diaries, helped participants to select bed and rise times, advised on management of daytime sleepiness (including implications for driving), and discussed barriers to and facilitators of implementation. Session 2, session 3, and session 4 involved reviewing progress, discussion of difficulties with implementation, and titration of the sleep schedule according to a sleep efficiency algorithm.
This means that the verum group received fairly extensive attention, while the control group did not. In other words, a host of non-specific effects are likely to have significantly influenced or even entirely determined the outcome. Despite this rather obvious limitation, the authors fail to discuss any of it. On the contrary, that claim that “we did a definitive test of whether brief sleep restriction therapy delivered in primary care is clinically effective and cost-effective.” This is, in my view, highly misleading and unworthy of THE LANCET. I suggest the conclusions of this trial should be re-formulated as follows:
The brief nurse-delivered sleep restriction, or the additional attention provided exclusively to the patients in the verum group, or a placebo-effect or some other non-specific effect reduced insomnia symptoms.
Alternatively, one could just conclude from this study that poor science can make it even into the best medical journals – a problem only too well known in the realm of so-called alternative medicine (SCAM).
We discussed the 2015 Australian NHMRC report on homeopathy many times before, e.g.:
- Homeopathy: the 2015 NHMRC report and its criticism re-analysed
- HOMEOPATHY: the NHMRC report revisited
- Ombudsman investigates ‘flawed’ homeopathic study
- The final verdict on homeopathy: it’s a placebo
In a nutshell, the report was an hugely influential analysis of the effectiveness of homeopathy which came to squarely negative conclusions. Thus it was celebrated as a thorough and conclusive piece evidence demonstrating the madness of homeopathy. Unsurprisingly, homeopaths did not like it at all and produced various criticisms claiming that it was neither thorough nor conclusive.
Now the final evaluation of what has been going on was finally published (ISSUED BY THE COMMONWEALTH OMBUDSMAN, IAIN ANDERSON, ON 4 AUGUST 2023):
The Office of the Commonwealth Ombudsman (the Office) has finalised an investigation relating to the National Health and Medical Research Council’s (NHMRC) review of the evidence for the effectiveness of homeopathy, conducted between 2010 and 2015. We commenced this investigation in September 2017 in response to concerns raised with us about how the NHMRC review had proceeded.
The Office conducts its investigations in private, and the Ombudsman generally does not make a public statement in the absence of a formal report. In the circumstances of this matter, including that the then-Ombudsman released a public statement on 4 June 2021 which acknowledged the Office was investigating, we believe it is important to share publicly the information we can, now that the investigation is complete.
Our investigation was finalised in July 2023. We acknowledge the length of time the investigation has taken. This is in part due to the extensive efforts the Office made to source independent scientific expertise to advise us on some detailed and specific questions of scientific methodology that were raised with our Office, including some that were only brought to our attention as our investigation progressed. Despite our best efforts, it was not possible to engage an expert (or experts) to provide independent advice to our Office on this subject. In the absence of independent, expert scientific expertise we have not been able to conclusively determine those matters of scientific methodology. This did not prevent our Office from forming a view on other aspects of the matter.
Our investigation did not result in any adverse findings about the review or the NHMRC. When finalising investigations, we may offer comments and suggestions to an agency about areas for future improvement. In this instance, we offered comments and suggestions to the NHMRC about how it records and publicly explains decisions about its activities. The NHMRC also independently made several improvements to its processes during the course of our investigation.
________________
In essence, this means that the conclusions of the report stand:
Homeopathy should not be used to treat health conditions that are chronic, serious, or could become serious. People who choose homeopathy may put their health at risk if they reject or delay treatments for which there is good evidence for safety and effectiveness. People who are considering whether to use homeopathy should first get advice from a registered health practitioner. Those who use homeopathy should tell their health practitioner and should keep taking any prescribed treatments.
Thus the matter is closed – that is closed for rational thinkers. For irrationalists, the matter will no doubt continue to be a stone of contention. No, homeopath will be able to accept these conclusions simply because a member of a cult ceases to be a cultist once he/she accepts the criticism agaist the cult.
Guest post by Norbert Aust, Udo Endruscheit, and Edzard Ernst
How do we know whether a treatment is reasonable or just some so-called alternative medicine (SCAM) that is at best useless? A simple answer is that the former is evidence-based, while the latter is not. But how can we tell the difference? High-quality studies, with independent replications or even a systematic review, are the sort of things we are usually looking for. But there is an underlying assumption, namely that, in science, bogus studies are prevented from polluting the scientific database or, if such trials have emerged, there are ways to identify and eliminate them.
And what if this assumption is wrong?
What if respectable universities and research organizations venture into the realm of pseudoscience either knowingly or because it had slipped their attention?
What if the editorial board of a top journal passes bogus studies to peer review?
What if such a paper is eventually reviewed by a proponent of the implausible therapy?
What if the readers of the article, once it is published, are too lethargic to object and do not write letters to the editor in protest?
And what if skeptics do formulate a protest but the journal editor refuses to publish it?
Well, if all the checks that should prevent faulty results from entering the scientific knowledge fail, we have fake evidence: a study that looks like sound science but that, in fact, is invalid. It is not hard to imagine what would happen if SCAM therapies are supported by seemingly respectable studies published in top journals. The fake evidence would accumulate as part of the body of evidence and eventually enter mainstream clinical practice, education, politics, etc., etc. Thus the reputation of bogus therapies would grow unjustifiably.
If you think this cannot happen, you are in the wrong. After the infamous study by Frass et al about homeopathy as an add-on treatment for lung cancer, another homeopathy paper was published in 2022 by Gaertner et al. in Pediatric Research (PR), a Medline-indexed journal with a two-year impact factor of 3.95 belonging to the nature-group of journals. According to this meta-analysis ‘individualized homeopathy showed a clinically relevant and statistically robust effect in the treatment of ADHD’. Shortly after the publication of this paper, we sent a letter to the editor to point out the shortcomings of this study. Here it is:
Sir,
with this letter we like to comment on the systematic review and meta-analysis on childhood ADHD by Gaertner et al. recently published in your journal.
First off, we are surprised, that your journal that is connected to nature does publish a paper on a treatment that has no a-priory probability at all and thus can only contain false positives if any. And this review is no exception as will be seen presently.
Our concerns are:
Out of the six studies included three were mere pilot trials (Fibert_2019, Jacobs_2005, Oberai_2013, ) which cannot provide any evidence for the shortcomings involved in pilots. Three of the six trials show severe issues in blinding (Fibert_2016, Fibert_2019, Oberai_2013), with two of them concerning both of the participants and the test personnel. This usually leads to massive bias in favour of the treatment [Zitat Cochrane Handbook].
Then we compared data from two trials with the data reported in the review and found some major misrepresentations:
(1) Jacobs et al. report an improvement in the T-score of their main outcome (CGI-P) of 4.1 for homeopathy and 9.1 for controls, that is placebo outperformed the homeopathic intervention. But the authors give an effect size of 0.272 in favour of homeopathy which is the opposite of the findings in the trial.
(2) Oberai et al. report effect sizes for their three main outcomes of 0.22, 0.59 and 0.54 (CPRS-R, CGISS, CGIIS repectively). There is no way that this yields a pooled effect size of 1.436 as given in the review.
We conclude that the positive result obtained by the authors is due to a combination of the inclusion of biased trials unsuitable to build evidence together with some major misreporting of study outcomes.
Our recommendation would be that the authors reconsider their review and improve their report. Maybe the editors would like to add a caution-notice to the paper – if not to withdraw it completely.
In June 2023, a full year after our submission, we were informed that Pediatric Research would not publish our criticism because the priority given to it was not sufficient to justify publication. But we were assured, that the journal would take the matter seriously, that they will investigate this matter and take appropriate editorial action. But as of today (End of June 2023) no expression of concern has been published.
Did the journal receive other comments or criticisms related to the paper in question? No, apparently there were none, at least none was published and the paper remains unchallenged to this day. This means that it might be taken for reliable evidence on the effectiveness of homeopathy and mislead patients, carers, practitioners, decision-makers, etc.
We feel this is unacceptable and therefore again wrote to the editors asking to reconsider their decision. Here is our letter:
Dear …
together with my co-authors we would like to comment your decision about our letter to the editor about an extremely faulty and misleading paper that may well create harm to patients. In fact we find it very hard to accept your decision not to publish our comment.
We understand that Pediatric Research is a high impact journal with a 2-year IF of nearly 4. Your journal is member of COPE and is indexed with quite a few first rank institutions. By all standards, any reader will be convinced that a paper published in Pediatric Research is based on solid research and the results are derived by rigorous methodology and are as reliable as can be. Especially if this paper remains unchallenged by any reader’s comments for a full year after publication. This is your responsibility to the scientific community. And to the children that might receive treatment based on knowledge spread through your journal.
How then can it be, that an article about homeopathy, a thoroughly implausible lore, in the treatment of ADHD is published in Pediatric Research, where the authors come to the conclusion “that individualized homeopathy showed a clinically relevant and statistically robust effect in the treatment of ADHD”?
In our comment we point out that the authors made a lot of errors – to say it mildly. They deny the doubtful quality of the studies they included in their meta-analysis, they did not stick to their own exclusion criteria, the data the authors report do not resemble the findings of the studies they were allegedly taken from, the one study setting the results is a mere pilot study.
The reason you give for our letter not being published is that it was not given enough priority to justify publication. We would like to know: Which issues can conceivably receive higher priority than the fact that a paper in your journal is downright wrong and misleading?
What do you need to deem a comment important? Up to now the paper is unchallenged by any reader’s comments, so apparently there was no other letter to the editor that might be given higher priority than ours.
We ask you to review your decision, or better still, consider a retraction of the paper altogether. If so, an expression of concern should be issued at once. After all, the COPE-guidelines for retraction state “clear evidence, that the findings are unreliable, either as a result of major error (…), or as a result of fabrication (…) or faslification (…)’ as a reason to consider retraction.
Otherwise the malpractice of homeopathy will have a first class evidence that will be helpful to promote homeopathy to parents and their children.
Watch this space!
This analysis was aimed at quantifying how many studies registered on the Open Science Framework (OSF) up to November 2017 are performed but not shared after at least 4 years. Examining a sample of 315 registrations, of which 169 were research studies, the researchers found that 104 (62%) were published. They estimated that 5550 out of 9544 (58%) registered studies on the OSF are published.
Researchers use registries to make unpublished studies public, and the OSF policy to open registrations after a four-year embargo substantially increases the number of studies that become known to the scientific community. In responses to emails asking researchers why studies remained unpublished logistical issues (e.g., lack of time, researchers changing jobs) were the most common cause, followed by null results, and rejections during peer review.
The authors concluded that their study shows that a substantial amount of studies researchers perform remain unpublished.
I find this truly shocking!
Researchers are able to do research only because they receive financial and other support from elsewhere. Therefore they have an ethical obligation to publish it. The reasons frequently given for not publishing research are nothing well and truly invalid:
- Lack of time is a mere excuse; if researchers had the time to get the grants, permissions, etc. they simply must have the time to finish the job properly.
- Researchers changing jobs is an equally flawed excuse; if someone changes position, he/she is obliged to finish the job they were doing. A surgeon can also not leave mid-surgery because he has a better offer.
- ‘Null results’ is even worse as a reason. Null results are just as important as positive findings – occasionally they are even more important. If researchers fail to realize this, they simply disqualify themselves as researchers.
- ‘Rejections during peer review’ is complete nonsense. Everyone who submits papers for publication gets rejected once in a while. In this case, one learns from the peer-review comments, improves the paper in question, and re-submits it to another journal.
I have seen many studies of so-called alternative medicine (SCAM) that, for this or that reason, never were published. And I feel strongly that this is a serious violation of research ethics – so much so that I would ban researchers who are guilty of this crime from conducting research in the future. I also feel that, in order to receive the necessary support (financial and other), researchers should sign that they will publish their findings within a given time after finishing their study. Failing to comply could then incur a penalty such as paying back part of the funds wasted. I think such measures would very quickly clear up the current intolerable situation.
In the comments section, someone recently alerted us to a most remarkable article. I had a look at it and thought it would be a pity to let it pass without further comment. Here is the abstract:
There are many types of energy around us, including natural and artificial ones, the first of the ground energies due to the imbalance happened from the treatment of man with the ground (mines-the bases of huge buildings); the result of the Earth rotation, the result of geological faults, the flow of groundwater or energies resulting from other factors that result in radiations that harm organisms in general. Also we are continuously increasing the amount of carrier waves needed for the wireless technology of modern communication in the earth’s atmosphere every day. These electromagnetic waves are thousands of times stronger than the level used in the communication in our body cells. The problem is not the saturation of the earth’s atmosphere through quantity, but also a detrimental quality. Even people who avoid using high technology are not immune. No one is immune because these are carrier waves with penetrating properties. our immune systems are continuously trying to correct the distortion in the transfer of inner information in our body; very soon the threshold will be reached when a total collapse of our body defenses will take place. Balancing the activities of daily life, achieving harmony with our inner and outer environments, humanizing modern technology, integrating science and spirits, and discovering the unified scientific reality behind all religions is the work of some science such as Bio Geometry, Bio Design, Radiesthesia, …ext.
When one runs a blog on so-called alternative medicine (SCAM), it is almost inevitable to run into plenty of bullshit. Thus, over the years, I have gotten used to even the most compact versions of it. Yet, this paper – I do recommend you have a glance also at the full text – is truly outstanding.
In case there is someone amongst my readers who understands what the author wants to express, I would be most obliged to learn.
‘Spagyric’ is a so-called alternative medicine (SCAM) based on the alchemy of Paracelsus (1493-1541). Paracelsus borrowed the term from “separate” (spao) and “combine” (ageiro) to indicate that spagyric preparations are based on the “separation”, “extraction” and “recombination” of the active ingredients of a substance. Plant, mineral as well as animal source materials are used.
The production of spagyric remedies is based on a complex process of maceration and fermentation of a plant extract in alcohol. It takes place in dark, thick-walled glass flasks that are hermetically sealed and kept at a controlled temperature of 37 °C for 28 days. The tincture thus obtained is then decanted and the drug residue is removed from the solution, completely dried, and burned to ash to recover the inorganic components of the plant material. The ash is subsequently dissolved in the alcoholic solution of maceration, and the finished spagyric preparation is left for 12 days before use.
Spagyric is not the most popular of all SCAMs but it certainly does have a significant following. One enthusiast claims that “spagyric essences work on a vibrational level in their action upon the emotional/mind and physical spheres and can be employed in numerous situations. Most people seek help to relieve physical symptoms. Even so, it is often necessary to address the emotional and psychological aspects which may predispose the illness or imbalance. In an era where many people are experiencing life-changing events, the ability to transition smoothly is essential for well-being and vitality. Guidance and help are required to maintain homeostasis. These medicines can help the patient to understand the root cause of their illness and learn to regain control of their lives. Some medicine systems appear to be less effective than in previous times. It has been suggested that the energetic frequency of both the earth and human organism are changing. Therefore these systems may no longer be a vibrational match for the changing frequencies. Spagyric Medicine is designed to ‘tune in with’ these current frequencies. Research suggests that the Spagyric essences may instigate improved health by energetically influencing DNA.”
After reading such weird statements, I ask myself, is there any evidence that spagyric remedies work? In my search for robust studies, I was unsuccessful. There does not seem to be a single controlled study on the subject. However, there are fragmentary reports of a study initiated and conducted by a now largely unknown healer named Karl Hann von Weyhern.
Von Weyhern (1882 – 1954) had taken a few semesters of pharmacy and medicine in Freiburg but remained without a degree. In 1930, he became a member of the NSDAP (Hitler’s Nazi party) and in 1940 he joined the SS. Around 1935, he settled in Munich as a non-medical practitioner (Heilpraktiker), and Heinrich Himmler who has a soft spot for SCAM enlisted as one of his patients. By then von Weyhern had by then made a steep career in the Nazi hierarchy, and he managed to convince Himmler that his spagyric remedies could cure tuberculosis, which was still rampant at the time. They decided to carry out experiments in this regard in the Dachau concentration camp.
Thus, von Weyhern was allowed to test spagyric remedies on forcibly recruited concentration camp prisoners. These experiments lasted for about one year and included around 150 patients who, according to von Weyhern’s iridology diagnosis, suffered from tuberculosis. Half of them were treated with spagyric remedies and the others with conventional treatments. At the end of the experiment, 27 persons were reportedly released into everyday concentration camp life as ‘fit for work’. How many of the 150 prisoners lost their lives due to these experiments is not known. Von Weyhern never filed a final report. It is to be feared that the death toll was considerable. [1]
After the war, von Weyhern denied belonging to the SS, claimed that he had ‘sacrificed himself’ for his patients in the concentration camp, merely had to pay a fine, and was ‘denazified’ in 1948. Subsequently, he resumed his work as a ‘Heilpraktiker’ in Olching, a village near Dachau. [1]
Of course, these infamous experiments cannot be blamed on spagyric medicine. Yet, I feel they are nevertheless important, not least because they seem to reveal the only thing remotely resembling something like evidence.
[1] Die Ärzte der Nazi-Führer: Karrieren und Netzwerke : Mathias Schmidt (Hg.), Dominik Groß (Hg.), Jens Westemeier (Hg.): Amazon.de: BooksThere are debates in acupuncture-related systematic reviews and meta-analyses on whether searching Chinese databases to get more Chinese-language studies may increase the risk of bias and overestimate the effect size, and whether the treatment effects of acupuncture differ between Chinese and non-Chinese populations.
For this meta-epidemiological study, a team of investigators searched the Cochrane Library from its inception until December 2021, and identified systematic reviews and meta-analyses with acupuncture as one of the interventions. Paired reviewers independently screened the reviews and extracted the information. They repeated the meta-analysis of the selected outcomes to separately pool the results of Chinese- and non-Chinese-language acupuncture studies and presented the pooled estimates as odds ratios (OR) with 95% confidence interval (CI). They calculated the Ratio of ORs (ROR) by dividing the OR of the Chinese-language trials by the OR of the non-Chinese-language trials, and the ROR by dividing the OR of trials addressing Chinese population by the OR of trials addressing non-Chinese population. The researchers thus explored whether the impact of a high risk of bias on the effect size differed between studies published in Chinese- and in non-Chinese-language, and whether the treatment effects of acupuncture differed between Chinese and non-Chinese populations.
The researchers identified 84 Cochrane acupuncture reviews involving 33 Cochrane groups, of which 31 reviews (37%) searched Chinese databases. Searching versus not searching Chinese databases significantly increased the contribution of Chinese-language literature both to the total number of included trials (54% vs. 15%) and the sample size (40% vs. 15%). When compared with non-Chinese-language trials, Chinese-language trials were associated with a larger effect size (pooled ROR 0.51, 95% CI 0.29 to 0.91). The researchers also observed a higher risk of bias in Chinese-language trials in blinding of participants and personnel (97% vs. 51%) and blinding of outcome assessment (93% vs. 47%). The higher risk of bias was associated with a larger effect estimate in both Chinese language (allocation concealment: high/unclear risk vs. low risk, ROR 0.43, 95% CI 0.21 to 0.87) and non-Chinese-language studies (blinding of participants and personnel: high/unclear risk vs. low risk, ROR 0.41, 95% CI 0.23 to 0.74). However, the team found no evidence that the higher risk of bias would increase the effect size of acupuncture in Chinese-language studies more often than in non-Chinese-language studies (the confidence intervals of all ROR in the high-risk group included 1, Table 3). The researchers further found acupuncture appeared to be more effective in Chinese than in non-Chinese populations.
The authors concluded that the findings of this study suggest the higher risk of bias may lead to an overestimation of the treatment effects of acupuncture but would not increase the treatment effects in Chinese-language studies more often than in other language studies. The difference in treatment effects of acupuncture was probably associated with differences in population characteristics.
The authors discuss that, although searching Chinese databases can substantially increase the number of eligible studies and sample size in acupuncture reviews, the potentially higher risk of bias is an argument that needs to be considered in the inclusion of Chinese-language studies. Patients, investigators, and guideline panels should be cautious when adopting evidence from acupuncture reviews where studies with a high risk of bias contributed with a high weight to the meta-analysis.
The authors observed larger treatment effects of acupuncture in Chinese-language studies than in studies published in other languages. Although the treatment effects of acupuncture tended to be greater in studies with a high risk of bias, this potential overestimation did not differ between studies published in Chinese and in other languages. In other words, the larger treatment effects in Chinese-language studies cannot be explained by a high risk of bias. Furthermore, our study found acupuncture to be more effective in Chinese populations than in other populations, which could at least partly explain the larger treatment effects observed in Chinese-language studies.
I feel that this analysis obfuscates more than it clarifies. As we have discussed often here, acupuncture studies by Chinese researchers (regardless of what language they are published in) hardly ever report negative results, and their findings are often fabricated. It, therefore, is not surprising that their effect sizes are larger than those of other trials.
The only sensible conclusion from this messy and regrettable situation, in my view, is to be very cautious and exclude them from systematic reviews.
How often do we hear that chiropractic is safe because numerous trials reported no adverse events? This systematic review tested whether there has been a change in the reporting of adverse events associated with spinal manipulation in randomized clinical trials (RCTs) since 2016.
Databases were searched from March 2016 to May 2022: MEDLINE (Ovid), Embase, CINAHL, ICL, PEDro, and Cochrane Library. Domains of interest (pertaining to adverse events) included: completeness and location of reporting; nomenclature and description; spinal location and practitioner delivering manipulation; methodological quality of the studies and details of the publishing journal. Frequencies and proportions of studies reporting on each of these domains were calculated. Univariable and multivariable logistic regression models were fitted to examine the effect of potential predictors on the likelihood of studies reporting on adverse events.
5399 records were identified by the electronic searches, of which 154 (2.9%) were included in the analysis. Of these, 94 (61.0%) reported adverse events with only 23.4% providing an explicit description of what constituted an adverse event. Reporting of adverse events in the abstract has increased (n=29, 30.9%) while reporting in the results section has decreased (n=83, 88.3%) over the past 6 years. Spinal manipulation was delivered to 7518 participants in the included studies. No serious adverse events were reported in any of these studies.
The authors concluded as follows: while the current level of reporting of adverse events associated with spinal manipulation in RCTs has increased since our 2016 publication on the same topic, the level remains low and inconsistent with established standards. As such, it is imperative for authors, journal editors and administrators of clinical trial registries to ensure there is more balanced reporting of both benefits and harms in RCTs involving spinal manipulation.
This article is clearly relevant to our discussions about adverse events after spinal manipulation. However, I find it far too uncritical. This might be due to the affiliations of some of the authors:
- Integrative Spinal Research Group, Department of Chiropractic Medicine, University Hospital Balgrist and University of Zurich, Zurich, Switzerland.
- Department of Chiropractic, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, New South Wales, Australia.
Interestingly, the authors stated that they have no conflict of interest. Also interesting is the fact that they do not cite our paper from 2012. I, therefore, take the liberty of doing it:
Objective: To systematically review the reporting of adverse effects in clinical trials of chiropractic manipulation.
Data sources: Six databases were searched from 2000 to July 2011. Randomised clinical trials (RCTs) were considered, if they tested chiropractic manipulations against any control intervention in human patients suffering from any type of clinical condition. The selection of studies, data extraction, and validation were performed independently by two reviewers.
Results: Sixty RCTs had been published. Twenty-nine RCTs did not mention adverse effects at all. Sixteen RCTs reported that no adverse effects had occurred. Complete information on incidence, severity, duration, frequency and method of reporting of adverse effects was included in only one RCT. Conflicts of interests were not mentioned by the majority of authors.
Conclusions: Adverse effects are poorly reported in recent RCTs of chiropractic manipulations.
In percentage terms the results are similar. What is very different is that the authors of the new paper merely lament that the level remains low and inconsistent with established standards, while we make it clear in the abstract that adverse effect reporting is poor and in the paper identify this deficit as a violation against research ethics and thus as a form of scientific misconduct.
In view of all this, let me re-phrase the last sentence of the authors’ conclusion:
it is imperative for authors, journal editors, and administrators of clinical trial registries to ensure that researchers adhere to accepted ethical standards and that scientific misconduct no longer gets published.
