reviewer bias
When I was still at Exeter, I used to do an average of about 4 peer reviews per week of articles that had been submitted to all sorts of journals for publication. Now I reject most of these invitations and do perhaps just one per month.
Why?
Conducting a peer review is by no means an easy task. You have to realize that the authors have usually put a lot of hard work into their paper and a lot may depend on it in terms of their future. They thus have the right to receive a fair and responsible review. To do the job properly, it took me (even with plenty of experience in reading scientific papers) between 1 and 3 hours per article. Crucially, low-quality articles typically submitted to low-quality journals are more work than papers that adhere to a certain standard.
I do not think that the journal editors who send the submissions out for review appreciate how much work they ask from the reviewers. They normally pay nothing (even if they charge exorbitant handling fees from the authors) and offer you no benefit at all. In addition, many have systems that are more than tedious asking you to register, create a pin number, etc., etc. Then you have to follow certain rules and formats that differ from journal to journal. In a word, they add an administrative burden to the task of reading, understanding, checking a paper, and composing your judgment on it.
All this can be cumbersome but it’s not the reason why I do less and less peer reviews. The true reason is that research papers on so-called alternative medicine (SCAM) are now mostly published in one of the many 3rd class SCAM journals that have recently sprung up. There are so many of them that they, of course, struggle to get enough articles to fill their pages. In turn, this means that they are far too keen to publish anything regardless of its quality or validity. As a consequence, the quality of these articles and their authors are often dismal.
Here is an example of a (rather shocking but not unusual) email I received only today; it might show you what I mean:
Dear Professor!
…
I want to publish some papers in “Areas related to your research field”. Can you help me? I can provide a thank you fee!
For example, I will give you a $2000 thank you fee for helping me write articles. For example, if you add my name to your article, I will give you a $1000 thank you fee. Or I can help you pay for APC.
I know this email is presumptuous, but my friends and I need to publish dozens of papers every year. If you can help me, we can cooperate for a long time. I’m not kidding, I’m very sincere!
If you are offended, please forgive me!
Look forward to your reply!
Warmly Wishes, …
When I do a review for a low-quality SCAM journal and find major defects in an article, my experience has been that the editor then decides to publish it nonetheless. When this happens, I feel frustrated and ask myself: WHY DID THEY ASK FOR MY OPINION IF THEY DO NOT ABIDE BY IT?
Thus I decided that these journals are just as well off without my contributions. So, if you are an editor of a SCAM journal, do me a favor and do not molest me with your invitations to conduct a peer review and
COUNT ME OUT!
Should Acupuncture-Related Therapies be Considered in Prediabetes Control?
No!
If you are pre-diabetic, consult a doctor and follow his/her advice. Do NOT do what acupuncturists or other self-appointed experts tell you. Do NOT become a victim of quackery.
But the authors of a new paper disagree with my view.
So, let’s have a look at the evidence.
Their systematic review was aimed at evaluating the effects and safety of acupuncture-related therapy (AT) interventions on glycemic control for prediabetes. The Chinese researchers searched 14 databases and 5 clinical registry platforms from inception to December 2020. Randomized controlled trials involving AT interventions for managing prediabetes were included.
Of the 855 identified trials, 34 articles were included for qualitative synthesis, 31 of which were included in the final meta-analysis. Compared with usual care, sham intervention, or conventional medicine, AT treatments yielded greater reductions in the primary outcomes, including fasting plasma glucose (FPG) (standard mean difference [SMD] = -0.83; 95% confidence interval [CI], -1.06, -0.61; P < .00001), 2-hour plasma glucose (2hPG) (SMD = -0.88; 95% CI, -1.20, -0.57; P < .00001), and glycated hemoglobin (HbA1c) levels (SMD = -0.91; 95% CI, -1.31, -0.51; P < .00001), as well as a greater decline in the secondary outcome, which is the incidence of prediabetes (RR = 1.43; 95% CI, 1.26, 1.63; P < .00001).
The authors concluded that AT is a potential strategy that can contribute to better glycemic control in the management of prediabetes. Because of the substantial clinical heterogeneity, the effect estimates should be interpreted with caution. More research is required for different ethnic groups and long-term effectiveness.
But this is clearly a positive result!
Why do I not believe it?
There are several reasons:
- There is no conceivable mechanism by which AT prevents diabetes.
- The findings heavily rely on Chinese RCTs which are known to be of poor quality and often even fabricated. To trust such research would be a dangerous mistake.
- Many of the primary studies were designed such that they failed to control for non-specific effects of AT. This means that a causal link between AT and the outcome is doubtful.
- The review was published in a 3rd class journal of no impact. Its peer-review system evidently failed.
So, let’s just forget about this rubbish paper?
If only it were so easy!
Journalists always have a keen interest in exotic treatments that contradict established wisdom. Predictably, they have been reporting about the new review thus confusing or misleading the public. One journalist, for instance, stated:
Acupuncture has been used for thousands of years to treat a variety of illnesses — and now it could also help fight one of the 21st century’s biggest health challenges.
New research from Edith Cowan University has found acupuncture therapy may be a useful tool in avoiding type 2 diabetes.
The team of scientists investigated dozens of studies covering the effects of acupuncture on more than 3600 people with prediabetes. This is a condition marked by higher-than-normal blood glucose levels without being high enough to be diagnosed as diabetes.
According to the findings, acupuncture therapy significantly improved key markers, such as fasting plasma glucose, two-hour plasma glucose, and glycated hemoglobin. Additionally, acupuncture therapy resulted in a greater decline in the incidence of prediabetes.
The review can thus serve as a prime example for demonstrating how irresponsible research has the power to mislead millions. This is why I have often said that poor research is a danger to public health.
And what can be done about this more and more prevalent problem?
The answer is easy: people need to behave more responsibly; this includes:
- trialists,
- review authors,
- editors,
- peer-reviewers,
- journalists.
Yes, the answer is easy in theory – but the practice is far from it!
The Lancet is a top medical journal, no doubt. But even such journals can make mistakes, even big ones, as the Wakefield story illustrates. But sometimes, the mistakes are seemingly minor and so well hidden that the casual reader is unlikely to find them. Such mistakes can nevertheless be equally pernicious, as they might propagate untruths or misunderstandings that have far-reaching consequences.
A recent Lancet paper might be an example of this phenomenon. It is entitled “Management of common clinical problems experienced by survivors of cancer“, unquestionably an important subject. Its abstract reads as follows:
_______________________
Improvements in early detection and treatment have led to a growing prevalence of survivors of cancer worldwide.
Models of care fail to address adequately the breadth of physical, psychosocial, and supportive care needs of those who survive cancer. In this Series paper, we summarise the evidence around the management of common clinical problems experienced by survivors of adult cancers and how to cover these issues in a consultation. Reviewing the patient’s history of cancer and treatments highlights potential long-term or late effects to consider, and recommended surveillance for recurrence. Physical consequences of specific treatments to identify include cardiac dysfunction, metabolic syndrome, lymphoedema, peripheral neuropathy, and osteoporosis. Immunotherapies can cause specific immune-related effects most commonly in the gastrointestinal tract, endocrine system, skin, and liver. Pain should be screened for and requires assessment of potential causes and non-pharmacological and pharmacological approaches to management. Common psychosocial issues, for which there are effective psychological therapies, include fear of recurrence, fatigue, altered sleep and cognition, and effects on sex and intimacy, finances, and employment. Review of lifestyle factors including smoking, obesity, and alcohol is necessary to reduce the risk of recurrence and second cancers. Exercise can improve quality of life and might improve cancer survival; it can also contribute to the management of fatigue, pain, metabolic syndrome, osteoporosis, and cognitive impairment. Using a supportive care screening tool, such as the Distress Thermometer, can identify specific areas of concern and help prioritise areas to cover in a consultation.
_____________________________
You can see nothing wrong? Me neither! We need to dig deeper into the paper to find what concerns me.
In the actual article, the authors state that “there is good evidence of benefit for … acupuncture …”[1]; the same message was conveyed in one of the tables. In support of these categorical statements, the authors quote the current Cochrane review entitled “Acupuncture for cancer pain in adults”. Its abstract reads as follows:
Background: Forty per cent of individuals with early or intermediate stage cancer and 90% with advanced cancer have moderate to severe pain and up to 70% of patients with cancer pain do not receive adequate pain relief. It has been claimed that acupuncture has a role in management of cancer pain and guidelines exist for treatment of cancer pain with acupuncture. This is an updated version of a Cochrane Review published in Issue 1, 2011, on acupuncture for cancer pain in adults.
Objectives: To evaluate efficacy of acupuncture for relief of cancer-related pain in adults.
Search methods: For this update CENTRAL, MEDLINE, EMBASE, PsycINFO, AMED, and SPORTDiscus were searched up to July 2015 including non-English language papers.
Selection criteria: Randomised controlled trials (RCTs) that evaluated any type of invasive acupuncture for pain directly related to cancer in adults aged 18 years or over.
Data collection and analysis: We planned to pool data to provide an overall measure of effect and to calculate the number needed to treat to benefit, but this was not possible due to heterogeneity. Two review authors (CP, OT) independently extracted data adding it to data extraction sheets. Data sheets were compared and discussed with a third review author (MJ) who acted as arbiter. Data analysis was conducted by CP, OT and MJ.
Main results: We included five RCTs (285 participants). Three studies were included in the original review and two more in the update. The authors of the included studies reported benefits of acupuncture in managing pancreatic cancer pain; no difference between real and sham electroacupuncture for pain associated with ovarian cancer; benefits of acupuncture over conventional medication for late stage unspecified cancer; benefits for auricular (ear) acupuncture over placebo for chronic neuropathic pain related to cancer; and no differences between conventional analgesia and acupuncture within the first 10 days of treatment for stomach carcinoma. All studies had a high risk of bias from inadequate sample size and a low risk of bias associated with random sequence generation. Only three studies had low risk of bias associated with incomplete outcome data, while two studies had low risk of bias associated with allocation concealment and one study had low risk of bias associated with inadequate blinding. The heterogeneity of methodologies, cancer populations and techniques used in the included studies precluded pooling of data and therefore meta-analysis was not carried out. A subgroup analysis on acupuncture for cancer-induced bone pain was not conducted because none of the studies made any reference to bone pain. Studies either reported that there were no adverse events as a result of treatment, or did not report adverse events at all.
Authors’ conclusions: There is insufficient evidence to judge whether acupuncture is effective in treating cancer pain in adults.
This conclusion is undoubtedly in stark contrast to the categorical statement of the Lancet authors: “there is good evidence of benefit for … acupuncture …“
What should be done to prevent people from getting misled in this way?
- The Lancet should correct the error. It might be tempting to do this by simply exchanging the term ‘good’ with ‘some’. However, this would still be misleading, as there is some evidence for almost any type of bogus therapy.
- Authors, reviewers, and editors should do their job properly and check the original sources of their quotes.
PS
In case someone argued that the Cochrane review is just one of many, here is the conclusion of an overview of 15 systematic reviews on the subject: The … findings emphasized that acupuncture and related therapies alone did not have clinically significant effects at cancer-related pain reduction as compared with analgesic administration alone.
The state of acupuncture research has long puzzled me. The first thing that would strike who looks at it is its phenomenal increase:
- Until around the year 2000, Medline listed about 200 papers per year on the subject.
- From 2005, there was a steep, near-linear increase.
- It peaked in 2020 when we had a record-breaking 20515 acupuncture papers currently listed in Medline.
Which this amount of research, one would expect to get somewhere. In particular, one would hope to slowly know whether acupuncture works and, if so, for which conditions. But this is not the case.
On the contrary, the acupuncture literature is a complete mess in which it gets more and more difficult to differentiate the reliable from the unreliable, the useful from the redundant, and the truth from the lies. Because of this profound confusion, acupuncture fans are able to claim that their pet-therapy is demonstrably effective for a wide range of conditions, while skeptics insist it is a theatrical placebo. The consumer might listen in bewilderment.
Yesterday (18/1/2021), I had a quick (actually, it was not that quick after all) look into what Medline currently lists in terms of new acupuncture research published in 2021 and found a few other things that are remarkable:
- There were already 100 papers dated 2021 (today, there were even 118); that corresponds to about 5 new articles per day and makes acupuncture one of the most research-active areas of so-called alternative medicine (SCAM).
- Of these 100 papers, only 7 were clinical trials (CTs). In my view, clinical trials would be more important than any other type of research on acupuncture. To see that they amount to just 7% of the total is therefore disappointing.
- Twelve papers were systematic reviews (SRs). It is odd, I find, to see almost twice the amount of SRs than CTs.
- Eighteen papers referred to protocols of studies of SRs. In particular protocols of SRs are useless in my view. It seems to me that the explanation for this plethora of published protocols might be the fact that Chinese researchers are extremely keen to get papers into Western journals; it is an essential boost to their careers.
- Seven papers were surveys. This multitude of survey research is typical for all types of SCAM.
- Twenty-four articles were on basic research. I find basic research into an ancient therapy of questionable clinical use more than a bit strange.
- The rest of the articles were other types of publications and a few were misclassified.
- The vast majority (n = 81) of the 100 papers were authored exclusively by Chinese researchers (and a few Korean). In view of the fact that it has been shown repeatedly that practically all acupuncture studies from China report positive results and that data fabrication seems rife in China, this dominance of China could be concerning indeed.
Yes, I find all this quite concerning. I feel that we are swamped with plenty of pseudo-research on acupuncture that is of doubtful (in many cases very doubtful) reliability. Eventually, this will create an overall picture for the public that is misleading to the extreme (to check the validity of the original research is a monster task and way beyond what even an interested layperson can do).
And what might be the solution? I am not sure I have one. But for starters, I think, that journal editors should get a lot more discerning when it comes to article submissions from (Chinese) acupuncture researchers. My advice to them and everyone else:
if it sounds too good to be true, it probably is!
The medical literature is currently swamped with reviews of acupuncture (and other forms of TCM) trials originating from China. Here is the latest example (but, trust me, there are hundreds more of the same ilk).
The aim of this review was to evaluate the effectiveness of scalp, tongue, and Jin’s 3-needle acupuncture for the improvement of post-apoplectic aphasia. PubMed, Cochrane, Embase databases were searched using index words to identify qualifying randomized controlled trials (RCTs). Meta-analyses of odds ratios (OR) or standardized mean differences (SMD) were performed to evaluate the outcomes between investigational (scalp / tongue / Jin’s 3-needle acupuncture) and control (traditional acupuncture; TA and/or rehabilitation training; RT) groups.
Thirty-two RCTs (1310 participants in investigational group and 1270 in control group) were included. Compared to TA, (OR 3.05 [95% CI: 1.77, 5.28]; p<0.00001), tongue acupuncture (OR 3.49 [1.99, 6.11]; p<0.00001), and Jin’s 3-needle therapy (OR 2.47 [1.10, 5.53]; p = 0.03) had significantly better total effective rate. Compared to RT, scalp acupuncture (OR 4.24 [95% CI: 1.68, 10.74]; p = 0.002) and scalp acupuncture with tongue acupuncture (OR 7.36 [3.33, 16.23]; p<0.00001) had significantly better total effective rate. In comparison with TA/RT, scalp acupuncture, tongue acupuncture, scalp acupuncture with tongue acupuncture, and Jin’s three-needling significantly improved ABC, oral expression, comprehension, writing and reading scores.
The authors concluded that compared to traditional acupuncture and/or rehabilitation training, scalp acupuncture, tongue acupuncture, and Jin’ 3-needle acupuncture can better improve post-apoplectic aphasia as depicted by the total effective rate, the ABC score, and comprehension, oral expression, repetition, denomination, reading and writing scores. However, quality of the included studies was inadequate and therefore further high-quality studies with lager samples and longer follow-up times and with patient outcomes are necessary to verify the results presented herein. In future studies, researchers should also explore the efficacy and differences between scalp acupuncture, tongue acupuncture and Jin’s 3-needling in the treatment of post-apoplectic aphasia.
I’ll be frank: I find it hard to believe that sticking needles in a patient’s tongue restores her ability to speak. What is more, I do not believe a word of this review and its conclusion. And now I better explain why.
- All the primary studies originate from China, and we have often discussed how untrustworthy such studies are.
- All the primary studies were published in Chinese and cannot therefore be checked by most readers of the review.
- The review authors fail to provide the detail about a formal assessment of the rigour of the included studies; they merely state that their methodological quality was low.
- Only 6 of the 32 studies can be retrieved at all via the links provided in the articles.
- As far as I can find out, some studies do not even exist at all.
- Many of the studies compare acupuncture to unproven therapies such as bloodletting.
- Many do not control for placebo effects.
- Not one of the 32 studies reports findings that are remotely convincing.
I conclude that such reviews are little more than pseudo-scientific propaganda. They seem aim at promoting acupuncture in the West and thus serve the interest of the People’s Republic of China. They pollute our medical literature and undermine the trust in science.
I seriously ask myself, are the editors and reviewers all fast asleep?
The journal ‘BMC Complement Altern Med‘ has, in its 18 years of existence, published almost 4 000 Medline-listed papers. They currently charge £1690 for handling one paper. This would amount to about £6.5 million! But BMC are not alone; as I have pointed out repeatedly, EBCAM is arguably even worse.
And this is, in my view, the real scandal. We are being led up the garden path by people who make a very tidy profit doing so. BMC (and EBCAM) must put an end to this nonsense. Alternatively, PubMed should de-list these publications.
This has been going on for far too long; urgent action is required!
Before a scientific paper gets published in a journal, it is submitted to the process of peer-review. Essentially, this means that the editor sends it to 2 or 3 experts in the field asking them to review the submission. Reviewers usually do not get any reward for this, yet the task they are asked to do can be tedious, difficult and time-consuming. Therefore, most reviewers think carefully before accepting it.
My friend Timothy Caulfield was recently invited by a medical journal to review a study of homeopathy. Here is his response to the editor as posted on Twitter:
I find myself regularly in similar situations. Yet, I have never responded in this way. Here is what I normally do:
- I have a look at the journal itself. If it is one of those SCAM publications, I tend to politely reject the invitation because, in my experience, their review process is farcical and not worth the effort. All too often it has happened that I reviewed a paper that was of very poor quality and thus recommended rejecting it. Yet the editor ignored my expert opinion and published the article nevertheless. This is why, several years ago, I decided enough is enough and no longer consider investing my time is such frustrating work.
- If the journal is of decent standing, I would have a look at the submission the editor sent me. If it makes any sense at all I would consider reviewing it (obviously depending on whether I have the time and the expertise).
- If a decent journal invites me to review a nonsensical paper (I assume that was the case Timothy referred to), I find myself in the same position as my friend Timothy. But, contrary to Timothy, I normally take the trouble to write a critical review of a nonsensical submission. Why? The reason is simple: if I don’t do it, the editor will simply send it to another reviewer. Many journals allow authors to suggest reviewers of their choice. Thus, the editor might send the submission next to the person suggested by the author who most likely will write a favourable review, thus hugely increasing the chances that the paper will be published in a decent journal.
On this blog, we have seen repeatedly that even top journal occasionally publish rubbish papers. Perhaps they do so because well-intentioned experts react in the way my friend Timothy did above (as he failed to tell us what journal invited him, I might be wrong).
If we want pseudoscience to disappear, we are fighting a lost battle. It will always rear its ugly head in third class journals. This is lamentable, but perhaps not so disastrous: by publishing little else than rubbish, these SCAM journals discredit themselves and will eventually be read only by pseudoscientists.
But we can do our bit to get rid of pseudoscience in decent journals. For this to happen, I think, rational thinkers need to accept invitations from such journals and do a proper review. And, of course, they can add to it a sentence or two about the futility of reviewing nonsense.
I am sure Timothy and I both want to eliminate pseudoscience as much as possible. In other words, we are in agreement about the aim, yet we differ in our approach. The question is: which is more effective?
I remember reading this paper entitled ‘Comparison of acupuncture and other drugs for chronic constipation: A network meta-analysis’ when it first came out. I considered discussing it on my blog, but then decided against it for a range of reasons which I shall explain below. The abstract of the original meta-analysis is copied below:
The objective of this study was to compare the efficacy and side effects of acupuncture, sham acupuncture and drugs in the treatment of chronic constipation. Randomized controlled trials (RCTs) assessing the effects of acupuncture and drugs for chronic constipation were comprehensively retrieved from electronic databases (such as PubMed, Cochrane Library, Embase, CNKI, Wanfang Database, VIP Database and CBM) up to December 2017. Additional references were obtained from review articles. With quality evaluations and data extraction, a network meta-analysis (NMA) was performed using a random-effects model under a frequentist framework. A total of 40 studies (n = 11032) were included: 39 were high-quality studies and 1 was a low-quality study. NMA showed that (1) acupuncture improved the symptoms of chronic constipation more effectively than drugs; (2) the ranking of treatments in terms of efficacy in diarrhoea-predominant irritable bowel syndrome was acupuncture, polyethylene glycol, lactulose, linaclotide, lubiprostone, bisacodyl, prucalopride, sham acupuncture, tegaserod, and placebo; (3) the ranking of side effects were as follows: lactulose, lubiprostone, bisacodyl, polyethylene glycol, prucalopride, linaclotide, placebo and tegaserod; and (4) the most commonly used acupuncture point for chronic constipation was ST25. Acupuncture is more effective than drugs in improving chronic constipation and has the least side effects. In the future, large-scale randomized controlled trials are needed to prove this. Sham acupuncture may have curative effects that are greater than the placebo effect. In the future, it is necessary to perform high-quality studies to support this finding. Polyethylene glycol also has acceptable curative effects with fewer side effects than other drugs.
END OF 1st QUOTE
This meta-analysis has now been retracted. Here is what the journal editors have to say about the retraction:
After publication of this article [1], concerns were raised about the scientific validity of the meta-analysis and whether it provided a rigorous and accurate assessment of published clinical studies on the efficacy of acupuncture or drug-based interventions for improving chronic constipation. The PLOS ONE Editors re-assessed the article in collaboration with a member of our Editorial Board and noted several concerns including the following:
- Acupuncture and related terms are not mentioned in the literature search terms, there are no listed inclusion or exclusion criteria related to acupuncture, and the outcome measures were not clearly defined in terms of reproducible clinical measures.
- The study included acupuncture and electroacupuncture studies, though this was not clearly discussed or reported in the Title, Methods, or Results.
- In the “Routine paired meta-analysis” section, both acupuncture and sham acupuncture groups were reported as showing improvement in symptoms compared with placebo. This finding and its implications for the conclusions of the article were not discussed clearly.
- Several included studies did not meet the reported inclusion criteria requiring that studies use adult participants and assess treatments of >2 weeks in duration.
- Data extraction errors were identified by comparing the dataset used in the meta-analysis (S1 Table) with details reported in the original research articles. Errors included aspects of the study design such as the experimental groups included in the study, the number of study arms in the trial, number of participants, and treatment duration. There are also several errors in the Reference list.
- With regard to side effects, 22 out of 40 studies were noted as having reported side effects. It was not made clear whether side effects were assessed as outcome measures for the other 18 studies, i.e. did the authors collect data clarifying that there were no side effects or was this outcome measure not assessed or reported in the original article. Without this clarification the conclusion comparing side effect frequencies is not well supported.
- The network geometry presented in Fig 5 is not correct and misrepresents some of the study designs, for example showing two-arm studies as three-arm studies.
- The overall results of the meta-analysis are strongly reliant on the evidence comparing acupuncture versus lactulose treatment. Several of the trials that assessed this comparison were poorly reported, and the meta-analysis dataset pertaining to these trials contained data extraction errors. Furthermore, potential bias in studies assessing lactulose efficacy in acupuncture trials versus lactulose efficacy in other trials was not sufficiently addressed.
While some of the above issues could be addressed with additional clarifications and corrections to the text, the concerns about study inclusion, the accuracy with which the primary studies’ research designs and data were represented in the meta-analysis, and the reporting quality of included studies directly impact the validity and accuracy of the dataset underlying the meta-analysis. As a consequence, we consider that the overall conclusions of the study are not reliable. In light of these issues, the PLOS ONE Editors retract the article. We apologize that these issues were not adequately addressed during pre-publication peer review.
LZ disagreed with the retraction. YM and XD did not respond.
END OF 2nd QUOTE
Let me start by explaining why I initially decided not to discuss this paper on my blog. Already the first sentence of the abstract put me off, and an entire chorus of alarm-bells started ringing once I read further.
- A meta-analysis is not a ‘study’ in my book, and I am somewhat weary of researchers who employ odd or unprecise language.
- We all know (and I have discussed it repeatedly) that studies of acupuncture frequently fail to report adverse effects (in doing this, their authors violate research ethics!). So, how can it be a credible aim of a meta-analysis to compare side-effects in the absence of adequate reporting?
- The methodology of a network meta-analysis is complex and I know not a lot about it.
- Several things seemed ‘too good to be true’, for instance, the funnel-plot and the overall finding that acupuncture is the best of all therapeutic options.
- Looking at the references, I quickly confirmed my suspicion that most of the primary studies were in Chinese.
In retrospect, I am glad I did not tackle the task of criticising this paper; I would probably have made not nearly such a good job of it as PLOS ONE eventually did. But it was only after someone raised concerns that the paper was re-reviewed and all the defects outlined above came to light.
While some of my concerns listed above may have been trivial, my last point is the one that troubles me a lot. As it also related to dozens of Cochrane reviews which currently come out of China, it is worth our attention, I think. The problem, as I see it, is as follows:
- Chinese (acupuncture, TCM and perhaps also other) trials are almost invariably reporting positive findings, as we have discussed ad nauseam on this blog.
- Data fabrication seems to be rife in China.
- This means that there is good reason to be suspicious of such trials.
- Many of the reviews that currently flood the literature are based predominantly on primary studies published in Chinese.
- Unless one is able to read Chinese, there is no way of evaluating these papers.
- Therefore reviewers of journal submissions tend to rely on what the Chinese review authors write about the primary studies.
- As data fabrication seems to be rife in China, this trust might often not be justified.
- At the same time, Chinese researchers are VERY keen to publish in top Western journals (this is considered a great boost to their career).
- The consequence of all this is that reviews of this nature might be misleading, even if they are published in top journals.
I have been struggling with this problem for many years and have tried my best to alert people to it. However, it does not seem that my efforts had even the slightest success. The stream of such reviews has only increased and is now a true worry (at least for me). My suspicion – and I stress that it is merely that – is that, if one would rigorously re-evaluate these reviews, their majority would need to be retracted just as the above paper. That would mean that hundreds of papers would disappear because they are misleading, a thought that should give everyone interested in reliable evidence sleepless nights!
So, what can be done?
Personally, I now distrust all of these papers, but I admit, that is not a good, constructive solution. It would be better if Journal editors (including, of course, those at the Cochrane Collaboration) would allocate such submissions to reviewers who:
- are demonstrably able to conduct a CRITICAL analysis of the paper in question,
- can read Chinese,
- have no conflicts of interest.
In the case of an acupuncture review, this would narrow it down to perhaps just a handful of experts worldwide. This probably means that my suggestion is simply not feasible.
But what other choice do we have?
One could oblige the authors of all submissions to include full and authorised English translations of non-English articles. I think this might work, but it is, of course, tedious and expensive. In view of the size of the problem (I estimate that there must be around 1 000 reviews out there to which the problem applies), I do not see a better solution.
(I would truly be thankful, if someone had a better one and would tell us)
Did you know that I falsified my qualifications?
Neither did I!
But this is exactly what has been posted on Amazon as a review of my book HOMEOPATHY, THE UNDILUTED FACTS. The Amazon review in question is dated 7 August 2018 and authored by ‘Paul’. As it might not be there for long (because it is clearly abusive) I copied it for you:
Edzard Ernst falsified his qualifications to get a job as a professor. When the university found out they fired him. This book is as false as the Mr Ernst
Over the years, I have received so many insults that I stared to collect them and began to quite like them. I even posted selections on this blog (see for instance here and here). Some are really funny and others are enlightening because they reflect on the mind-set of the authors. All of them show that the author has run out of arguments; thus they really are little tiny victories over unreason, I think.
But, somehow, this new one is different. It is actionable, no doubt, and contains an unusual amount of untruths in so few words.
- I never falsified anything and certainly not my qualification (which is that of a doctor of medicine). If I had, I would be writing these lines from behind bars.
- And if I had done such a thing, I would not have done it ‘to get a job as a professor’ – I had twice been appointed to professorships before I came to the UK (Hannover and Vienna).
- My university did not find out, mainly because there was nothing to find out.
- They did not fire me, but I went into early retirement. Subsequently, they even re-appointed me for several months.
- My book is not false; I don’t even know what a ‘false book’ is (is it a book that is not really a book but something else?).
- And finally, for Paul, I am not Mr Ernst, but Prof Ernst.
I don’t know who Paul is. And I don’t know whether he has even read the book he pretends to be commenting on (from what I see, I think this is very unlikely). I am sure, however, that he did not read my memoir where all these things are explained in full detail. And I certainly do not hope he ever reads it – if he did, he might claim:
This book is as false as the Mr Ernst
Distant healing is one of the most bizarre yet popular forms of alternative medicine. Healers claim they can transmit ‘healing energy’ towards patients to enable them to heal themselves. There have been many trials testing the effectiveness of the method, and the general consensus amongst critical thinkers is that all variations of ‘energy healing’ rely entirely on a placebo response. A recent and widely publicised paper seems to challenge this view.
This article has, according to its authors, two aims. Firstly it reviews healing studies that involved biological systems other than ‘whole’ humans (e.g., studies of plants or cell cultures) that were less susceptible to placebo-like effects. Secondly, it presents a systematic review of clinical trials on human patients receiving distant healing.
All the included studies examined the effects upon a biological system of the explicit intention to improve the wellbeing of that target; 49 non-whole human studies and 57 whole human studies were included.
The combined weighted effect size for non-whole human studies yielded a highly significant (r = 0.258) result in favour of distant healing. However, outcomes were heterogeneous and correlated with blind ratings of study quality; 22 studies that met minimum quality thresholds gave a reduced but still significant weighted r of 0.115.
Whole human studies yielded a small but significant effect size of r = .203. Outcomes were again heterogeneous, and correlated with methodological quality ratings; 27 studies that met threshold quality levels gave an r = .224.
From these findings, the authors drew the following conclusions: Results suggest that subjects in the active condition exhibit a significant improvement in wellbeing relative to control subjects under circumstances that do not seem to be susceptible to placebo and expectancy effects. Findings with the whole human database suggests that the effect is not dependent upon the previous inclusion of suspect studies and is robust enough to accommodate some high profile failures to replicate. Both databases show problems with heterogeneity and with study quality and recommendations are made for necessary standards for future replication attempts.
In a press release, the authors warned: the data need to be treated with some caution in view of the poor quality of many studies and the negative publishing bias; however, our results do show a significant effect of healing intention on both human and non-human living systems (where expectation and placebo effects cannot be the cause), indicating that healing intention can be of value.
My thoughts on this article are not very complimentary, I am afraid. The problems are, it seems to me, too numerous to discuss in detail:
- The article is written such that it is exceedingly difficult to make sense of it.
- It was published in a journal which is not exactly known for its cutting edge science; this may seem a petty point but I think it is nevertheless important: if distant healing works, we are confronted with a revolution in the understanding of nature – and surely such a finding should not be buried in a journal that hardly anyone reads.
- The authors seem embarrassingly inexperienced in conducting and publishing systematic reviews.
- There is very little (self-) critical input in the write-up.
- A critical attitude is necessary, as the primary studies tend to be by evangelic believers in and amateur enthusiasts of healing.
- The article has no data table where the reader might learn the details about the primary studies included in the review.
- It also has no table to inform us in sufficient detail about the quality assessment of the included trials.
- It seems to me that some published studies of distant healing are missing.
- The authors ignored all studies that were not published in English.
- The method section lacks detail, and it would therefore be impossible to conduct an independent replication.
- Even if one ignored all the above problems, the effect sizes are small and would not be clinically important.
- The research was sponsored by the ‘Confederation of Healing Organisations’ and some of the comments look as though the sponsor had a strong influence on the phraseology of the article.
Given these reservations, my conclusion from an analysis of the primary studies of distant healing would be dramatically different from the one published by the authors: DESPITE A SIZABLE AMOUNT OF PRIMARY STUDIES ON THE SUBJECT, THE EFFECTIVENESS OF DISTANT HEALING REMAINS UNPROVEN. AS THIS THERAPY IS BAR OF ANY BIOLOGICAL PLAUSIBILITY, FURTHER RESEARCH IN THIS AREA SEEMS NOT WARRANTED.
When someone has completed a scientific project, it is customary to publish it [‘unpublished science is no science’, someone once told me many years ago]. To do so, he needs to write it up and submit it to a scientific journal. The editor of this journal will then submit it to a process called ‘peer review’.
What does ‘peer review’ entail? Well, it means that 2-3 experts are asked to critically assess the paper in question, make suggestions as to how it can be improved and submit a recommendation as to whether or not the article deserves to be published.
Peer review has many pitfalls but, so far, nobody has come up with a solution that is convincingly better. Many scientists are under pressure to publish [‘publish or perish’], and therefore some people resort to cheating. A most spectacular case of fraudulent peer review has been reported recently in this press release:
SAGE statement on Journal of Vibration and Control
London, UK (08 July 2014) – SAGE announces the retraction of 60 articles implicated in a peer review and citation ring at the Journal of Vibration and Control (JVC). The full extent of the peer review ring has been uncovered following a 14 month SAGE-led investigation, and centres on the strongly suspected misconduct of Peter Chen, formerly of National Pingtung University of Education, Taiwan (NPUE) and possibly other authors at this institution.
In 2013 the then Editor-in-Chief of JVC, Professor Ali H. Nayfeh,and SAGE became aware of a potential peer review ring involving assumed and fabricated identities used to manipulate the online submission system SAGE Track powered by ScholarOne Manuscripts™. Immediate action was taken to prevent JVC from being exploited further, and a complex investigation throughout 2013 and 2014 was undertaken with the full cooperation of Professor Nayfeh and subsequently NPUE.
In total 60 articles have been retracted from JVC after evidence led to at least one author or reviewer being implicated in the peer review ring. Now that the investigation is complete, and the authors have been notified of the findings, we are in a position to make this statement.
While investigating the JVC papers submitted and reviewed by Peter Chen, it was discovered that the author had created various aliases on SAGE Track, providing different email addresses to set up more than one account. Consequently, SAGE scrutinised further the co-authors of and reviewers selected for Peter Chen’s papers, these names appeared to form part of a peer review ring. The investigation also revealed that on at least one occasion, the author Peter Chen reviewed his own paper under one of the aliases he had created.
Unbelievable? Perhaps, but sadly it is true; some scientists seem to be criminally ingenious when it comes to getting their dodgy articles into peer-reviewed journals.
And what does this have to do with ALTERNATIVE MEDICINE, you may well ask. The Journal of Vibration and Control is not even medical and certainly would never consider publishing articles on alternative medicine. Such papers go to one of the many [I estimate more that 1000] journals that cover either alternative medicine in general or any of the modalities that fall under this wide umbrella. Most of these journals, of course, pride themselves with being peer-reviewed – and, at least nominally, that is correct.
I have been on the editorial board of most of the more important journals in alternative medicine, and I cannot help thinking that their peer review process is not all that dissimilar from the fraudulent scheme set up by Peter Chen and disclosed above. What happens in alternative medicine is roughly as follows:
- a researcher submits a paper for publication,
- the editor sends it out for peer review,
- the peer reviewers are either those suggested by the original author or members of the editorial board of the journal,
- in either case, the reviewers are more than likely to be uncritical and recommend publication,
- in the end, peer review turns out to be a farcical window dressing exercise with no consequence,
- thus even very poor research and pseudo-research are being published abundantly.
The editorial boards of journals of alternative medicine tend to be devoid of experts who are critical about the subject at hand. If you think that I am exaggerating, have a look at the editorial board members of ‘HOMEOPATHY’ (or any other journal of alternative medicine) and tell me who might qualify as a critic of homeopathy. When the editor, Peter Fisher, recently fired me from his board because he felt I had tarnished the image of homeopathy, this panel lost the only person who understood the subject matter and, at the same time, was critical about it (the fact that the website still lists me as an editorial board member is merely a reflection of how slow things are in the world of homeopathy: Fisher fired me more than a year ago).
The point I am trying to make is simple: peer review is never a perfect method but when it is set up to be deliberately uncritical, it cannot possibly fulfil its function to prevent the publication of dodgy research. In this case, the quality of the science will be inadequate and generate false-positive messages that mislead the public.
