conflict of interest
When tested rigorously, the evidence for so-called alternatives medicine (SCAM) is usually weak or even negative. This fact has prompted many SCAM enthusiasts to become utterly disenchanted with rigorous tests such as the randomised clinical trial (RCT). They seem to think that, if the RCT fails to generate the findings we want, let’s use different methodologies instead. In other words, they are in favour of observational studies which often yield positive results.
This line of thinking is prevalent in all forms of SCAM, but probably nowhere more so that in the realm of homeopathy. Homeopaths see that rigorous RCTs tend not to confirm their belief and, to avoid cognitive dissonance, they focus on observational studies which are much more likely to confirm their belief.
In this context, it is worth mentioning a recent article where well-known homeopathy enthusiasts have addressed the issue of observational studies. Here is their abstract:
Background: Randomized placebo-controlled trials are considered to be the gold standard in clinical research and have the highest importance in the hierarchical system of evidence-based medicine. However, from the viewpoint of decision makers, due to lower external validity, practical results of efficacy research are often not in line with the huge investments made over decades.
Method: We conducted a narrative review. With a special focus on homeopathy, we give an overview on cohort, comparative cohort, case-control and cross-sectional study designs and explain guidelines and tools that help to improve the quality of observational studies, such as the STROBE Statement, RECORD, GRACE and ENCePP Guide.
Results: Within the conventional medical research field, two types of arguments have been employed in favor of observational studies. First, observational studies allow for a more generalizable and robust estimation of effects in clinical practice, and if cohorts are large enough, there is no over-estimation of effect sizes, as is often feared. We argue that observational research is needed to balance the current over-emphasis on internal validity at the expense of external validity. Thus, observational research can be considered an important research tool to describe “real-world” care settings and can assist with the design and inform the results of randomised controlled trails.
Conclusions: We present recommendations for designing, conducting and reporting observational studies in homeopathy and provide recommendations to complement the STROBE Statement for homeopathic observational studies.
In their paper, the authors state this:
It is important to realize three areas where observational research can be valuable. For one, as already mentioned, it can be valuable as a preparatory type of research for designing good randomized studies. Second, it can be valuable as a stand-alone type of research, where pragmatic or ethical reasons stand against conducting a randomized study. Additionally, it can be valuable as the only adequate method where choices are involved: for instance, in any type of lifestyle research or where patients have very strong preferences, such as in homeopathy and other CAM. This might also lead to a diversification of research efforts and a broader, more realistic, picture of the effects of therapeutic interventions.
My comments to this are as follows:
- Observational research can be valuable as a preparatory type of research for designing good randomized studies. This purpose is better fulfilled by pilot studies (which are often abused in SCAM).
- Observational research can be valuable as a stand-alone type of research, where pragmatic or ethical reasons stand against conducting a randomized study. Such situations rarely arise in the realm of SCAM.
- Observational research can be valuable as the only adequate method where choices are involved: for instance, in any type of lifestyle research or where patients have very strong preferences, such as in homeopathy and other CAM. I fail to see that this is true.
- Observational research leads to a diversification of research efforts and a broader, more realistic, picture of the effects of therapeutic interventions. The main aim of research into the effectiveness of SCAM should be, in my view, to determine whether the treatment per se works or not. Observational studies are likely to obscure the truth on this issue.
Don’t get me wrong, I am not saying that observational studies are useless; quite to the contrary, they can provide very important information. But what I am trying to express is this:
- We should not allow double standards in medical research. The standards and issues of observational research as they exist in conventional medicine must also apply to SCAM.
- Observational studies cannot easily determine cause and effect between the therapy and the outcome.
- Observational studies cannot be a substitute for RCTs.
- Depending on their exact design, observational studies measure the outcome caused by a whole range of factors, including the therapy per se, the placebo-effect, the natural history of the disease, the regression towards the mean.
- Observational studies are particularly useful in effectiveness research, AFTER the efficacy of a therapy has been established by RCTs.
- If RCT fail to show that a therapy is effective and observational studies seem to indicate that they work, the therapy in question is probably a placebo.
- SCAM-enthusiasts’ preference for observational studies is transparently due to motivated reasoning.
Michael Dixon LVO, OBE, MA, FRCGP has been a regular feature of this blog (and elsewhere). He used to be a friend and colleague until … well, that’s a long story. Recently, I came across his (rather impressive) Wikipedia page. To my surprise, it mentions that Dixon
“has been criticised by professor of complementary medicine and alternative medicine campaigner Edzard Ernst for advocating the use of complementary medicine. Ernst said that the stance of the NHS Alliance on complementary medicine was “misleading to the degree of being irresponsible.” Ernst had previously been sympathetic to building a bridge between complementary and mainstream medicine, co-writing an article with Michael Dixon in 1997 on the benefits of such an approach. Ernst and Dixon write “missed diagnoses by complementary therapists giving patients long term treatments are often cited but in the experience of one of the authors (MD) are extremely rare. It can also cut both ways. A patient was recently referred back to her general practitioner by an osteopath, who was questioning, as it turned out quite correctly, whether her pain was caused by metastates. Good communication between general practitioner and complementary therapist can reduce conflicts and contradictions, which otherwise have the potential to put orthodox medicine and complementary therapy in an either/or situation.”
31) February 2009, 24. “Academics and NHS Alliance clash over complementary medicine”. Pulse Today.
32) ^ Update – the journal of continuing education for General Practitioners, 7th May 1997
I have little recollection of the paper that I seem to have published with my then friend Michael, and it is not listed in Medline, nor can I find it in my (usually well-kept) files; the journal ‘Update’ does not exist anymore and was obviously not a journal good enough for keeping a copy. But I do not doubt that Wiki is correct.
In fact, it is true that, in 1997, I was still hopeful that bridges could be built between conventional medicine and so-called alternative medicine (SCAM). But I had always insisted that they must be bridges built on solid ground and with robust materials.
Put simply, my strategy was to test SCAM as rigorously as I could and to review the totality of the evidence for and against it. Subsequently, one could consider introducing those SCAMs into routine care that had passed the tests of science.
Dixon’s strategy differed significantly from mine. He had no real interest in science and wanted to use SCAM regardless of the evidence. Since the publication of our paper in 1997, he has pursued this aim tirelessly. On this blog, we find several examples of his activity.
And what happened to the bridges?
I’m glad you ask!
As it turns out, very few SCAMs have so far passed the test of science and hardly any SCAM has been demonstrated to generate more good than harm. The material to build bridges is therefore quite scarce, hardly enough for solid constructions. Dixon does still not seem to be worried about this indisputable fact. He thinks that INTEGRATED MEDICINE is sound enough for providing a way to the future. I disagree and still think it is ‘misleading to the degree of being irresponsible’.
Who is right?
Dixon or Ernst?
Opinions about this differ hugely.
Time will tell, I suppose.
Yesterday, I received a tweet from a guy called Bart Huisman (“teacher beekeeping, nature, biology, classical homeopathy, agriculture, health science, social science”). I don’t know him and cannot remember whether I had previous contact with him. His tweet read as follows:
“Why should anyone believe what Professor Edzard Ernst says, after he put his name to a BBC programme, he now describes as “deception”.”
This refers to a story that I had almost forgotten. It’s a nice one with a ‘happy ending’, so let me recount it here briefly.
In 2005, the BBC had hired me as an advisor for their 4-part TV series on alternative medicine.
The first part of the series was on acupuncture, and Prof Kathy Sykes presented the opening scene taking place in a Chinese operation theatre. In it a Chinese women was having open heart surgery with the aid of acupuncture. Kathy’s was visibly impressed and said on camera that the patient was having the surgery “with only needles to control the pain.” However, the images clearly revealed that the patient was receiving all sorts of other treatments given through intra-venous lines. So, Prof Sykes was telling the UK public a bunch of porkies. This was bound to confuse many viewers.
One of them was Simon Singh. At the time, I did not know Simon (to be honest, I did not even know of him) and was surprised to receive a phone call from him. He politely asked me to confirm that I had been the adviser of the BBC on this production. I was happy to confirm this fact. Then he asked why I had missed such a grave error. I replied that I could not possibly have spotted it, because all I had been asked to do was to review and correct the text of the programme which the BBC had sent to me by email. Before it was broadcast, I had not seen a single passage of the film.
Correcting the text had already led to several problems (not so much regarding the acupuncture part but mostly the other sections), because the BBC was reluctant to change several of the mistakes I had identified. When I told them that, in this case, I would quit, they finally found a way to alter them. But the cooperation had been far from easy. I explained all this to Simon and eventually he asked me whether I would be willing to support the official complaint he was about to file with the BBC. I agreed. This is probably where I used the term ‘deception’ that Mr Huisman mentioned in his tweet.
So, Simon submitted his complaint and eventually won the case.
But this is not the happy ending I was referring to.
During the course of the complaint, Simon and I realised that we were thinking alike and were getting on well. A few months later, he suggested that the two of us write a book together about alternative medicine. At first, I was hesitant. Simon said, “let’s try just one chapter, and see how it works out.” So we did. It turned out to be fun and instructive for both of us. So we did the other chapters as well. The book was published in 2008 and is called TRICK OR TREATMENT. It was published in about 20 different languages and the German version became ‘science book of the year in 2011 (I think).
And that’s not the happy ending either (in fact, it caused a lot of hardship for Simon who was sued by the BCA; luckily, he won that case too).
The real happy ending is the fact that Simon and I became friends for life.
Thank you Bart Huisman for reminding me of this rather lovely story.
‘HOMEOPATHY RESOURCE’ claim they are the online web site for accurate information on homeopathy, homeopaths and homeopathic organizations. On 18 June, they published an article entitled “Another Remarkable Demonstration of Homeopathy’s Effectiveness in Covid-19: ONLY 19 Deaths out of 35 million in Kerala, India”. Here it is in its full beauty:
The State of Kerela India has shown that homeopathy and grassroots health care works dramatically well in epidemic and pandemic situations.. The state, in spite of areas of great density, has reported only 19 deaths. This compares to the UK which has a population of 66 million people but a tragic 41,698 deaths.
The region’s Health Minister Shailaja Teacher has been called the “Coronavirus Slayer” after introducing homeopathy as a primary means of dealing with the epidemic. Kerala India has already come through two Nipah virus epidemics under her watch.
Tactics used by Shailaja included encouraging the use of homeopathy. In a recent media meeting, she talked about the need “to improve the immunity and resistance power of each individual not yet positive to Coronavirus, with the help of Homeopathic/ Ayurvedic medicines. That will help them resist the Coronavirus infection, help them to tide over the infection well, if at all they contract it…… everyone should take Homeopathic & Ayurvedic preventative treatments available.”
According to Dr E.S.Rajendran who practices in Kerala “The total number of people who received the homeopathic preventive medicine Ars alb 30 through Kerala government as on June 1st was 10 million. An equal number of people have also received the same through voluntary organisations. The remaining population is expected to receive the preventive medicine in another one month.” This distribution was carried out in each district with the help of resident associations and was achievable because there is a huge demand for homeopathy from the people.
Do ‘HOMEOPATHY RESOURCE’ really think that this is ‘accurate information’?
Do they feel that it amounts to evidence that homeopathy prevents COVID-19 infections?
Do they believe that it is responsible to promote such a message?
If so, they are more deluded than their homeopathic remedies are diluted!
There is a plethora of factors that might have contributed to the low infection rate in Kerela. Here are some that spring into my mind (in descending order of probability):
- Social distancing measures might have been put in place at the right time.
- Hand-washing might have been better accepted than in the UK.
- Face-masks might have been more common.
- The government might have been less incompetent than the one in the UK.
- The pandemic might be arriving with some delay in Kerela (in the last 2 days the COVID cases have more than doubled in Kerela).
- The hot weather might have inhibited the spread of the infection.
- The Ayurvedic medicine mentioned by the health minister might have worked.
- There might be many more cases due to under-diagnosis and poor testing.
- The holy cows might have prevented infections.
- Homeopathy works dramatically well in epidemic and pandemic situations.
Call me a sceptic, but – whatever turns out to be true (and I sincerely hope that the case numbers stay low in Kerela) – I do not think that ‘HOMEOPATHY RESOURCE’ is providing accurate information with their recent post.
And call me a pessimist, but I fail to see any good evidence to assume that homeopathic remedies have any effect in treating or preventing viral infections. In fact,
HIGHLY DILUTE HOMEOPATHIC REMEDIES ARE PURE PLACEBOS!
Siddha medicine is based on a combination of ancient medicinal practices and spiritual disciplines as well as alchemy and mysticism. It is thought to be one of the oldest system of healthcare of India that developed during the Indus civilization, which flourished between 2500 and 1700 BCE.
It has been reported that the Indian ‘National Institute of Siddha’ (NIS) claim to have successfully treated 160 COVID-19 positive patients. Subsequently, they have requested the government to hand over all COVID-19 Care Centres in Chennai and let Siddha doctors treat all COVID-patients. They say they are confident of flattening the curve in Chennai and convert it into a safe zone in just matter of days.
The NIS claim to have three potent combinations of Siddha drugs. “Depending upon the availability and quantity required for treating Coronavirus positive patients, we have after thorough research, come out with three different effective combinations of the Siddha preparations,” Dr R Meenakumari, Director of NIS, said. The treatment low-cost compared to the prohibitive cost in corporate hospitals and all the Siddha medicines are locally available, she added. “We have requested the state government to hand over all the COVID-19 Care Centres to us and allow us to treat all the patients. Our Siddha drug combination is potent enough to convert a positive patient into Coronavirus negative in three days’ time,” she claimed.
Her confidence stems from the fact that the Siddha doctors here have “successfully” treated 160 patients besides
23 inmates of the Puzhal Central Prison. “Initially, we treated 85 patients with SRM Medical College and Hospital and another 75 at the Greater Chennai Corporation… They all recovered and tested negative after five days of successful treatment,” she claimed. “The combination that we have prepared will help to cure the infected patient within three days… Siddha medicine has huge potential to treat the patients and there are possibilities to use the medicine to save precious human lives”.
The combination drug in question seems to be similar to or identical with Kabasura Kudineer. Also termed ‘Nilavembu Kudineer‘, this drug is a powder form of medicine mainly used in the treatment of respiratory problems such as fever, cold, severe phlegm and flu. This polyherbal Siddha medicine is also widely used as a prophylactic during times of viral epidemics. To get the proper benefits, it should be made into a decoction and then consumed.
Kabasura Kudineer is made up of 15 different ingredients:
- kali mirch
- malabar nut,
In 2009, it allegedly helped containing the spread of swine flu and, in 2012, the then Chief Minister Jayalalithaa had requested public to use Nilavembu Kudineer prepared by the Institute to prevent dengue.
Meanwhile, the ‘Central Council for Research in Siddha’ has sent a proposal to the state government to include the
traditional medicine in the treatment protocols at the state-run CCCs. “We have also urged the state government to include the Brahmananda Bhairava Mathirai a herbo-mineral preparation, which has already been approved by the AYUSH ministry to treat persons with COVID-19 related fever, at all the COVID-19 wards,” a senior doctor at the CCRS said.
Of course, we all wish that an effective treatment against COVID-19 will be found soon. However, what the NIS calls THOROUGH RESEARCH looks like a flimsy bit of pseudo-research. And their assertion that their herbal mixture turns positive into negative patients within three days is a claim that sounds far too good to be true.
I have no reason to doubt that the NIS is full of good intentions. But I am reminded of Bert Brecht’s bon mot: ‘the opposite of good is not evil, but good intentions’.
Am I the only one who is tired of hearing that, in India, homeopathy is doing wonders for the current pandemic? All of the reports that I have seen are based on little more than hearsay, anecdotes or pseudo-science. If anyone really wanted to find out whether homeopathy works, they would need more than that; in fact, they would need to conduct a clinical trial.
As it happens, there are already ~500 clinical trials of homeopathy. Many show positive effects, but the reliable ones usually don’t. Crucially, the totality of the evidence fails to be positive. So, running further studies is hardly a promising exercise. In fact, considering how utterly implausible homeopathy is, it even seems like an unethical waste of resources.
But many homeopaths disagree, particularly those in India. And it has been reported that several trials have been given the go-ahead in India and are now up and running. This regrettable fact is being heavily exploited for swaying public opinion in favour of homeopathy. The way I see it, the situation is roughly this:
- a few trials of homeopathy are being set up;
- they are designed by enthusiasts of homeopathy who lack research expertise;
- therefore their methodology is weak and biased towards generating a false-positive result;
- while this is going on, the homeopathic propaganda machine is running overtime;
- when the results will finally emerge, they will get published in a 3rd rate journal;
- homeopaths worldwide will celebrate them as a triumph for homeopathy;
- critical thinkers will be dismayed at their quality and will declare that the conclusions drawn by over-enthusiastic homeopaths are not valid;
- in the end, we will be exactly where we were before: quasi-religious believers in homeopathy will feel vexed because their findings are not accepted in science, and everyone else will be baffled by the waste of time, opportunity and resources as well as by the tenacity of homeopaths to make fools of themselves.
But criticising is easy; doing it properly is often more difficult.
So, how should it be done?
The way I see it, one should do the following:
- carefully consider the implausibility of homeopathy;
- thoroughly study the existing evidence on homeopathy;
- abandon all plans to study homeopathy in the light of the above.
But this hardly is inconceivable considering the current situation in India. If further studies of homeopathy are unavoidable, the following procedure might therefore be reasonable:
- assemble a team of experts including trial methodologists, statisticians, epidemiologists and homeopaths;
- ask them to design a rigorous protocol of one or two studies that would provide a definitive answer to the research question posed;
- make sure that, once everyone is happy with the protocol, all parties commit to abiding by the findings that will emerge from these trials;
- conduct the studies under adequately strict supervision;
- evaluate the results according to the protocol;
- publish them in a top journal;
- do the usual press-releases, interviews etc.
In India, it seems that the last point in this agenda came far too early. This is because, in this and several other countries, homeopathy has become more a belief system than a medicine. And because it is about belief, the believers will avert any truly meaningful and rigorous test of homeopathy’s efficacy.
This was essentially the question raised in a correspondence with a sceptic friend. His suspicion was that statistical methods might produce false-positive overall findings, if the research is done by enthusiasts of the so-called alternative medicine (SCAM) in question (or other areas of inquiry which I will omit because they are outside my area of expertise). Consciously or inadvertently, such researchers might introduce a pro-SCAM bias into their work. As the research is done mostly by such enthusiasts; the totality of the evidence would turn out to be heavily skewed in favour of the SCAM under investigation. The end-result would then be a false-positive overall impression about the SCAM which is less based on reality than on the wishful thinking of the investigators.
How can one deal with this problem?
How to minimise the risk of being overwhelmed by false-positive research?
Today, we have several mechanisms and initiatives that are at least partly aimed at achieving just this. For instance, there are guidelines on how to conduct the primary research so that bias is minimised. The CONSORT statements are an example. As many studies pre-date CONSORT, we need a different approach for reviews of clinical trials. The PRISMA guideline or the COCHRANE handbook are attempts to make sure systematic reviews are transparent and rigorous. These methods can work quite well in finding the truth, but one needs to be aware, of course, that some researchers do their very best to obscure it. I have also tried to go one step further and shown that the direction of the conclusion correlates with the rigour of the study (btw: this was the paper that prompted Prof Hahn’s criticism and slander of my work and person).
So, problem sorted?
The trouble is that over-enthusiastic researchers may not always adhere to these guidelines, they may pretend to adhere but cut corners, or they may be dishonest and cheat. And what makes this even more tricky is the possibility that they do all this inadvertently; their enthusiasm could get the better of them, and they are doing research not to TEST WHETHER a treatment works but to PROVE THAT it works.
In the realm of SCAM we have a lot of this – trust me, I have seen it often with my own eyes, regrettably sometimes even within my own team of co-workers. The reason for this is that SCAM is loaded with emotion and quasi-religious beliefs; and these provide a much stronger conflict of interest than money could ever do, in my experience.
And how might we tackle this thorny issue?
After thinking long and hard about it, I came up in 2012 with my TRUSTWORTHYNESS INDEX:
If we calculated the percentage of a researcher’s papers arriving at positive conclusions and divided this by the percentage of his papers drawing negative conclusions, we might have a useful measure. A realistic example might be the case of a clinical researcher who has published a total of 100 original articles. If 50% had positive and 50% negative conclusions about the efficacy of the therapy tested, his TI would be 1.
Depending on what area of clinical medicine this person is working in, 1 might be a figure that is just about acceptable in terms of the trustworthiness of the author. If the TI goes beyond 1, we might get concerned; if it reaches 4 or more, we should get worried.
An example would be a researcher who has published 100 papers of which 80 are positive and 20 arrive at negative conclusions. His TI would consequently amount to 4. Most of us equipped with a healthy scepticism would consider this figure highly suspect.
Of course, this is all a bit simplistic, and, like all other citation metrics, my TI provides us not with any level of proof; it merely is a vague indicator that something might be amiss. And, as stressed already, the cut-off point for any scientist’s TI very much depends on the area of clinical research we are dealing with. The lower the plausibility and the higher the uncertainty associated with the efficacy of the experimental treatments, the lower the point where the TI might suggest something to be fishy.
Based on this concept, I later created the ALTERNATIVE MEDICINE HALL OF FAME. This is a list of researchers who manage to go through life researching their particular SCAM without ever publishing a negative conclusion about it. In terms of TI, these people have astronomically high values. The current list is not yet long, but it is growing:
John Weeks (editor of JCAM)
Deepak Chopra (US entrepreneur)
Cheryl Hawk (US chiropractor)
David Peters (osteopathy, homeopathy, UK)
Nicola Robinson (TCM, UK)
Peter Fisher (homeopathy, UK)
Simon Mills (herbal medicine, UK)
Gustav Dobos (various, Germany)
Claudia Witt (homeopathy, Germany and Switzerland)
George Lewith (acupuncture, UK)
John Licciardone (osteopathy, US)
The logical consequence of a high TI would be that researchers of that nature are banned from obtaining research funds and publishing papers, because their contribution is merely to confuse us and make science less reliable.
I am sure there are other ways of addressing the problem of being mislead by false-positive research. If you can think of one, I’d be pleased to hear about it.
In the wake of both the NEJM and the LANCET withdrawing two potentially influential papers due unanswered questions about the source and reliability of the data, one has to ask how good or bad the process of peer review is.
Peer review is the evaluation of work by one or more people with similar competences as the producers of the work (peers). It functions as a form of self-regulation by qualified members of a profession within the relevant field. It normally involves multiple steps:
- Authors send their manuscript to a journal of their choice for publication.
- The journal editor has a look at it and decides whether to reject it straight away (for instance, because the subject area is not of interest) or whether to send it out to referees for examination (often to experts suggested by the authors of the submission).
- The referees (usually 2 or 3) have the opportunity to reject or accept the invitation to review the submission.
- If they accept, they review the paper and send their report to the editor (usually following a deadline).
- The editor tries to come to a decision about publication; often the referees are not in agreement, and a further referee has to be recruited.
- Even if the submission is potentially publishable, the referees will have raised several points that need addressing. In such cases, the editor sends the submission back to the original authors asking them to revise the article.
- The authors do their revision (often following a deadline) and re-submit their paper.
- Now the editor can decide to either publish it or send it back to the referees asking them whether they feel their criticisms have been adequately addressed.
- Depending on the referees’ verdicts, the editor makes the final decision and informs all the involved parties accordingly.
- If the paper was accepted, it then goes into production.
- When this process is finished, the authors receive the proofs for final a check.
- Eventually, the paper is published and the readers of the journal may scrutinise it.
- Often this prompts comment which may get published.
- In this case, the authors of the original paper may get invited to write a reply.
- Finally the comments and the reply are published in the journal side by side.
The whole process takes time, sometimes lots of time. I have had papers that took almost two years from submissions to publications. This delay seems tedious and, if the paper is important, unacceptable (if it is not important, it should arguably not be published at all). Equally unacceptable is the fact that referees are expected to do their reviewing for free. The consequence is that many referees do their reviewing less than well.
When I was still at Exeter, I had plenty of opportunity to see the problems of peer review from the reviewers perspective. At a time, I accepted about 5 reviews per week, and in total I surely have reviewed over 1000 papers. I often recommended inviting a statistician to do a specialist review of the stats. Only rarely were such suggestions accepted by the journal editors. Very often I recommended rejecting a submission because it was rubbish, and occasionally, I told the editor that there was a strong suspicion of the paper being fraudulent. The editors very often (I estimate in about 50% of cases) ignored my suggestions and comments and published the papers nonetheless. If the editor did follow my advice to reject a paper, I regularly saw it published elsewhere later (usually in a less well-respected journal). Several times, an author of a submission contacted me directly after seeing my criticism of his paper. Occasionally this resulted in unpleasantness, once or twice even in threats. Eventually I realised that improving the publications in the realm of SCAM was a Sisyphean task, became quite disenchanted with all this and accepted less and less reviews. Today, I do only very few.
I had even more opportunity to see the peer review process from the author’s perspective. All authors must have suffered from unfair or incompetent reviews and most will have experienced the frustrations of the endless delays. Once (before my time in alternative medicine) a reviewer rejected my paper and soon after published results that were uncannily similar to mine. In alternative medicine, researchers tend to be rather emotional about their subject. Imagine, for instance, the review you might get from Dana Ullmann of a trial of homeopathy that fails to show what he believes in.
Finally, since 40 years, I have also had the displeasure of experiencing peer review as an editor. This often seemed like trying to sail between the devil and the deep blue sea. Editors want to fill their journals with the best science they can find. But all too often, they receive the worst science they can imagine. They are constantly torn by tensions pulling them in opposite directions. And they have to cope not just with poor quality submissions but also with reviewers who miss deadlines and do their work badly.
So, peer review is fraught with problems! The trouble is that there are few solutions that would keep a better check on the reliability of science. Peer review, it often seemed to me, is the worst idea, except for all others. If peer review is to survive (and I think it probably will), there are a few things that could, from my point of view, be done to improve it:
- Make it much more attractive for the referees. Payment would be the obvious thing – and by Jove, the big journals like the LANCET and NEJM could afford it. But recognising refereeing academically would be even more important. At present, academic careers depend largely of publications; if they also depended on reviewing, experts would queue up to do it.
- The reports of the referees should get independently evaluated according to sensible criteria. These data could be conflated an published as a criterion of academic standing. Referees who fail to to a good job would spoil their chances to get re-invited for this task.
- Speed up the entire process. Waiting months on months is hugely counter-productive for all concerned.
- Today many journals ask authors for the details of experts who are potential reviewers of their submission and then send the paper in question to them for review. I find this ridiculous! No author I know of has ever resisted the temptation to name people who are friends or owe a favour. Journals should afford the extra work to find who the best independent experts on any particular subject are.
None of this is simple or fool-proof or even sure to work well, of course. But surely it is worth trying to get peer-review right. The quality of future science depends on it.
By guest blogger Loretta Marron
Although assumed to be traditional, what we know today as ‘Traditional Chinese Medicine’ (TCM) was invented in the 1950s for political reasons by then Chairman Mao. It has since been proclaimed by Xi Jinping, now life-President of the People’s Republic of China, as the “jewel” in the millennia of Chinese civilization.
In May this year, Xi “announced plans to criminalise criticism of traditional Chinese medicine”. Speaking out against TCM could land you years in prison, prosecuted for “picking fights to disturb public order” and “defaming” the practice.
With the industry expected to earn $420 billion by the end of 2020, covid-19 has provided Xi with a platform to promote unproven, potentially harmful TCM. To keep these profits filling Chinese coffers, the World Health Organization (WHO) remains silent and those challenging TCM are silenced.
In January, the late Dr Li Wenliang was arrested and gaoled for warning China about covid-19. Li was one of up to nine people who were disciplined for spreading rumours about it. As the virus silently spread around the world, Beijing told the WHO that there was ‘no clear evidence’ of spread between humans.
As their death toll passed 1,000, Beijing’s response was to remove senior officials and to sack hundreds over their handling of the outbreak. With the support of the WHO, claims continue to be made that TCM “has been proved effective in improving the cure rate”, denying the simple fact that “patients would have recovered even if they hadn’t taken the Chinese medicine”.
With cases now heading for 8 million, and over four hundred thousand people confirmed dead world-wide and with economies in free-fall, Beijing continues, “to protect its interests and people overseas; to gain leadership of international governance”,for financial gain, to aggressively use its national power. Under the guise of ‘International Aid’, during the pandemic, Beijing promoted treatments based on unproven traditional medicine, sending TCM practitioners to countries including Italy, France and Iran.
Back in 2016, the Chinese State Council released a “Strategic Development Plan for Chinese Medicine (2016-2030)”, seeking to spread ‘knowledge’ into campuses, homes and abroad.
In July 2017, a law promising equal status for TCM and western medicine came into effect. Provisions included encouragement to China’s hospitals to set up TCM centres. “The new law on traditional Chinese medicine will improve global TCM influence, and give a boost to China’s soft power”.
In 2019, after strong lobbying by the Chinese Communist Party (CCP), WHO added a chapter on TCM to their official International Classification of Diseases (ICD-11). In China, doctors are now instructed to prescribe traditional medicine to most patients.
While Chinese herbs might have exotic names, they are, once translated, often the same as western herbs, many of which might have significant interactions. WHO fails to acknowledge any drug interactions.
In 1967, Mao launched Project 523 to find a cure for chloroquine-resistant malaria. Over 240,000 compounds had already been tested and none had worked. Trained in pharmacology and modern western methods, Tu Youyou used the scientific method to test sweet wormwood, a herb traditionally used in China for fever, where she developed a useful artemisinin derivative for resistant malaria. The drug has saved millions of lives. In 2015 she won the Nobel Prize for her work. However, Tu’s work is not a blanket endorsement of TCM: without the years of research, she would not have been successful.
TCM is commercially driven. Criticism of remedies is often blocked on the Internet in China, and critics have been jailed. The majority of TCM’s are not tested for efficacy in randomized clinical trials. Clinical trials are usually of poor quality and serious side effects are underreported. China has even rolled back regulations as Beijing forcefully promotes TCM’s as an alternative to proven western medicine. An increasing number of prestigious research hospitals now prescribe and dispense herbs that may cause drug interactions alongside western medicine for major illness patients.
TCM’s are not safe. Most systematic reviews suggest that there is no good or consistent evidence for effectiveness, negative results aren’t published, research data are fabricated and TCM-exports are of dubious quality.
If the benefits of herbal remedies are to be realised, good clinical studies must be encouraged.
TCM is not medicine. It’s little more than a philosophy or a set of traditional beliefs, about various concoctions and interventions and their alleged effect on health and diseases.
To stop misleading the world with what Mao himself saw as nonsense, and to mitigate future pandemics, WHO can and should remove all mention of TCM other than to state that it is unproven and could be dangerous.
Cochrane reviews have the reputation to be the most reliable evidence available anywhere. They are supposed to be independent, rigorous, transparent and up-to-date. Usually, this reputation is justified, in my view. But do the 54 Cochrane reviews of acupuncture quoted in my previous post live up to it?
If one had to put the entire body of evidence in a nutshell, it would probably look something like this:
TOTAL NUMBER OF SYSTEMATIC REVIEWS = 54
POSITIVE CONCLUSIONS BASED ON MORE THAN ONE HIGH QUALITY STUDY = 2
FAILURE TO REACH CONVINCINGLY POSITIVE CONCLUSIONS = 52
The two positive reviews are on:
1) prevention of migraine
2) prevention of tension-type headache
Both of the positive reviews are by Linde et al.
Allow me to raise just a few further critical points:
- If I counted correctly, 19 of the 54 reviews are authored entirely by Chinese authors. Why could this be a problem? One reason could be that many Chinese authors seem to be biased in favour of acupuncture. Another reason could be that data fabrication is rife in China.
- Many if not most of the primary studies are published in Chinese. This means that it is impossible for most non-Chinese co-authors of the review as well as for the referees of the paper to check the accuracy of the data extraction.
- I counted a total of 15 reviews which were by authors who one could categorise as outspoken enthusiasts of acupuncture. In these cases, one might be concerned about the trustworthiness of the review’s conclusion.
- Many (some would say most) of the reviews cover subject areas which are frankly bizarre. Who would, for instance, consider acupuncture a plausible treatment for Glaucoma, Mumps or chronic hepatitis B?
- Despite almost all of the reviews demonstrating that there is no good reason to recommend acupuncture for the condition in question, hardly any of them draw a transparent, helpful and clear conclusion. One example might suffice: the review of acupuncture for hordeolum concluded that “Low‐certainty evidence suggests that acupuncture with or without conventional treatments may provide short‐term benefits for treating acute hordeolum…” Its Chinese authors reached this conclusion on the basis of 6 primary studies (all from China) which were all of lousy quality. In such a case, the only justified conclusion would be, in my view, something like this: THERE IS NO RELIABLE EVIDENCE …
Despite these serious limitations and avoidable confusions, the totality of the evidence from these 54 Cochrane reviews does send an important message: there is hardly a single condition for which acupuncture is clearly, convincingly and indisputably effective. What I find most regrettable, however, is that the Cochrane Collaboration allowed the often biased review authors to obscure this crucial message so thoroughly. One needs a healthy portion of critical thinking to get through to the truth here – and how many fans of acupuncture possess such a thing?