conflict of interest
When someone has completed a scientific project, it is customary to publish it [‘unpublished science is no science’, someone once told me many years ago]. To do so, he needs to write it up and submit it to a scientific journal. The editor of this journal will then submit it to a process called ‘peer review’.
What does ‘peer review’ entail? Well, it means that 2-3 experts are asked to critically assess the paper in question, make suggestions as to how it can be improved and submit a recommendation as to whether or not the article deserves to be published.
Peer review has many pitfalls but, so far, nobody has come up with a solution that is convincingly better. Many scientists are under pressure to publish [‘publish or perish’], and therefore some people resort to cheating. A most spectacular case of fraudulent peer review has been reported recently in this press release:
London, UK (08 July 2014) – SAGE announces the retraction of 60 articles implicated in a peer review and citation ring at the Journal of Vibration and Control (JVC). The full extent of the peer review ring has been uncovered following a 14 month SAGE-led investigation, and centres on the strongly suspected misconduct of Peter Chen, formerly of National Pingtung University of Education, Taiwan (NPUE) and possibly other authors at this institution.
In 2013 the then Editor-in-Chief of JVC, Professor Ali H. Nayfeh,and SAGE became aware of a potential peer review ring involving assumed and fabricated identities used to manipulate the online submission system SAGE Track powered by ScholarOne Manuscripts™. Immediate action was taken to prevent JVC from being exploited further, and a complex investigation throughout 2013 and 2014 was undertaken with the full cooperation of Professor Nayfeh and subsequently NPUE.
In total 60 articles have been retracted from JVC after evidence led to at least one author or reviewer being implicated in the peer review ring. Now that the investigation is complete, and the authors have been notified of the findings, we are in a position to make this statement.
While investigating the JVC papers submitted and reviewed by Peter Chen, it was discovered that the author had created various aliases on SAGE Track, providing different email addresses to set up more than one account. Consequently, SAGE scrutinised further the co-authors of and reviewers selected for Peter Chen’s papers, these names appeared to form part of a peer review ring. The investigation also revealed that on at least one occasion, the author Peter Chen reviewed his own paper under one of the aliases he had created.
Unbelievable? Perhaps, but sadly it is true; some scientists seem to be criminally ingenious when it comes to getting their dodgy articles into peer-reviewed journals.
And what does this have to do with ALTERNATIVE MEDICINE, you may well ask. The Journal of Vibration and Control is not even medical and certainly would never consider publishing articles on alternative medicine. Such papers go to one of the many [I estimate more that 1000] journals that cover either alternative medicine in general or any of the modalities that fall under this wide umbrella. Most of these journals, of course, pride themselves with being peer-reviewed – and, at least nominally, that is correct.
I have been on the editorial board of most of the more important journals in alternative medicine, and I cannot help thinking that their peer review process is not all that dissimilar from the fraudulent scheme set up by Peter Chen and disclosed above. What happens in alternative medicine is roughly as follows:
- a researcher submits a paper for publication,
- the editor sends it out for peer review,
- the peer reviewers are either those suggested by the original author or members of the editorial board of the journal,
- in either case, the reviewers are more than likely to be uncritical and recommend publication,
- in the end, peer review turns out to be a farcical window dressing exercise with no consequence,
- thus even very poor research and pseudo-research are being published abundantly.
The editorial boards of journals of alternative medicine tend to be devoid of experts who are critical about the subject at hand. If you think that I am exaggerating, have a look at the editorial board members of ‘HOMEOPATHY’ (or any other journal of alternative medicine) and tell me who might qualify as a critic of homeopathy. When the editor, Peter Fisher, recently fired me from his board because he felt I had tarnished the image of homeopathy, this panel lost the only person who understood the subject matter and, at the same time, was critical about it (the fact that the website still lists me as an editorial board member is merely a reflection of how slow things are in the world of homeopathy: Fisher fired me more than a year ago).
The point I am trying to make is simple: peer review is never a perfect method but when it is set up to be deliberately uncritical, it cannot possibly fulfil its function to prevent the publication of dodgy research. In this case, the quality of the science will be inadequate and generate false-positive messages that mislead the public.
Times are hard, also in the strange world of chiropractic, I guess. What is therefore more understandable than the attempt of chiropractors to earn a bit of money from people who want to lose weight? If just some of the millions of obese individuals could be fooled into believing that chiropractic is the solution for their problem, chiropractors across the world could be laughing all the way to the bank.
But how does one get to this point? Easy: one only needs to produce some evidence suggesting that chiropractic care is effective in reducing body weight. An extreme option is the advice by one chiropractor to take 10 drops of a homeopathic human chorionic gonadotropin product under the tongue 5 times daily. But, for many chiropractors, this might be one step too far. It would be preferable to show that their hallmark therapy, spinal adjustment, leads to weight loss.
With this in mind, a team of chiropractors performed a retrospective file analysis of patient files attending their 13-week weight loss program. The program consisted of “chiropractic adjustments/spinal manipulative therapy augmented with diet/nutritional intervention, exercise and one-on-one counselling.”
Sixteen of 30 people enrolled completed the program. At its conclusion, statistically and clinically significant changes were noted in weight and BMI measures based on pre-treatment (average weight = 190.46 lbs. and BMI = 30.94 kg/m(2)) and comparative measurements (average weight = 174.94 lbs. and BMI = 28.50 kg/m(2)).
According to the authors of this paper, “this provides supporting evidence on the effectiveness of a multi-modal approach to weight loss implemented in a chiropractic clinic.”
They do not say so, but we all know it, of course: one could just as well combine knitting or crossword puzzles with diet/nutritional intervention, exercise and one-on-one counselling to create a multi-modal program for weight loss showing that knitting or crossword puzzles are effective.
With this paper, chiropractors are not far from their aim of being able to mislead the public by claiming that CHIROPRACTIC CARE IS A NATURAL, SAFE, DRUG-FREE AND EFFECTIVE OPTION IN THE MANAGEMENT OF OBESITY.
Am I exaggerating? No, of course not. There must be thousands of chiropractors who have already jumped on the ‘weight loss band-waggon’. If you don’t believe me, go on the Internet and have a look for yourself. One of the worst sites I have seen might be ‘DOCTORS GOLDMINE’ (yes, most chiropractors call themselves ‘doctor these days!) where a chiropractor promises his colleagues up to $100 000 per month extra income, if they subscribe to his wonderful weight-loss scheme.
It would be nice to be able to believe those who insist that these money-grabbing chiropractors are but a few rotten apples in a vast basket of honest practitioners. But I have problems with this argument – there seem to be far too many rotten apples and virtually no activity or even ambition to get rid of them.
Dr. Oz, famous through his TV show promoting all types of quackery, recently testified before a US Senate subcommittee hearing on protecting consumers from false and deceptive advertising of weight loss products. This event turned out to be less than flattering for Dr Oz. One journalist commented that he “might as well be a cowardly lion — sent home with his tail between his legs after being accused at a congressional hearing of lying on his show about weight-loss claims.”
“I don’t get why you need to say this stuff, because you know it’s not true,” said Senator Claire McCaskill, who led the commerce subcommittee hearing. “The scientific community is almost monolithically against you in terms of the efficacy of the products you called ‘miracles,’ ” the Democratic senator from Missouri told Oz. “It’s a major problem when people are spending more and more money and they’re gaining more and more weight,” said Senator Amy Klobuchar.“Either you don’t talk about these things at all, or you’re going to have to be more specific because right now . . . this is not working.”
A source close to Dr Oz said he was perplexed: “We were invited down to Washington to testify at a hearing about scams and instead it became all about how much we hate your show.” Oz himself testified that he “heard the message…I do personally believe in the items that I talk about.”
“I intensively study them. I have given my family these products. . . . If you can lose a pound a week more than you would have lost by using them, it jump-starts you and gets you going. I think it makes sense.” “I’m surprised you’re defending this,” McCaskill replied. “It’s something that gives people false hope. I don’t see why you need to go there.”
Another journalist commented that the Senators repeatedly placed him on the defense over his weight loss products: “I know you know how much power you have. I know you know that. You are very powerful and [with] power comes a great deal of responsibility,” Senator Claire McCaskill , who led the Senate’s consumer protection hearing titled “Protecting Consumers from False and Deceptive Advertising of Weight-Loss Products…You are being made an example of today because of the power you have in this space…We didn’t call this hearing to beat up on you but we did call this hearing to talk about a real crisis in consumer protection. You can either be part of the police here or you can be part of the problem.”
Oz insisted he was no huckster but admitted the products promoted on his show don’t always have “the scientific muster” to present their benefits as “fact…I actually do personally believe in the items that I talk about in the show. I passionately studied them. I recognize that oftentimes they don’t have the scientific muster to present as fact but nevertheless I would give my audience the advice I give my family all the time. And I have given my family these products,” he said.
Dr Oz also said that some alternative treatments, such as prayer, cannot be tested scientifically. “I don’t think this ought to be a referendum on the use of alternative medical therapies. Because if that’s the case, listen, I’ve been criticized for having folks coming on my show talking about the power of prayer,” he said. “I can’t prove that prayer helps people survive an illness.”
No, Dr Oz! I know you are mistaken! I have done the research – both on alternative slimming aids and on spiritual healing. The results quite clearly show that these methods are not more effective than a placebo.
In the early 1920s, a French physician thought he had discovered the virus that caused the Spanish flu. It oscillated under his microscope, and he thus called it oscillococcus. Not only did it cause the flu, in the opinion of his discoverer, but it was also responsible for a whole host of other diseases, including cancer. In fact, the virus does not exist, or at least nobody ever confirmed it existed, but that fact did not stop our good doctor to make a homeopathic remedy from it which he thought would cure all these diseases. His remedy, Oscillococcinum, is made from the liver and heart of a duck because the imaginative inventor believed that the fictitious virus was present in these organs of this animal.
To understand all this fully, one needs to know that the duck organs are so highly diluted that no molecule of the duck is present in the remedy. It is sold in the C200 potency. This means that one part of organ extract is diluted 1: 10 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 (a note to Boiron’s legal team: I had a hell of a time getting all these zeros right; in case, I got it wrong after all, it is an honest error – please do not sue me for it!). The dilution is so extreme that it amounts to a single molecule per a multitude of universes.
Given these facts it seems unlikely that the remedy has any effects on human health which go beyond those of a placebo. Let’s see what the current Cochrane review says about its effectiveness: There is insufficient good evidence to enable robust conclusions to be made about Oscillococcinum(®) in the prevention or treatment of influenza and influenza-like illness. Our findings do not rule out the possibility that Oscillococcinum(®) could have a clinically useful treatment effect but, given the low quality of the eligible studies, the evidence is not compelling. There was no evidence of clinically important harms due to Oscillococcinum(®).
Considering that the first author of this review works for the British Homeopathic Association and the senior author is the homeopath of the Queen, this seems a pretty clear statement, don’t you think?
Regardless of the scientific evidence, Oscillococcinum made of ‘Anas Barbariae Hepatis et Cordis Extractum‘, as it is officially called, became a homeopathic best-seller. In the US alone Boiron, the manufacturer, is said to sell US$ 15 m per year of this product. Not only that, in France, where the remedy is a popular medicine sold in virtually all pharmacies and often recommended as soon as you walk into a pharmacy, it is hard to find anyone who does not swear by the ‘potentized‘ duck or is willing to discuss its merits critically.
The amazing duck, it seems, has turned into a ‘holy cow’.
If we search on ‘Medline’ for ‘complementary alternative medicine’ (CAM), we currently get about 13000 hits. A little graph on the side of the page demonstrates that, during the last 4 years, the number of articles on this subject has grown exponentially.
Surely, this must be very good news: such intense research activity will soon tell us exactly which alternative treatments work for which conditions and which don’t.
I beg to differ. Let me explain why.
The same ‘Medline’ search informs us that the majority of the recent articles were published in an open access journal called ‘Evidence-Based Complementary and Alternative Medicine’ (eCAM). For example, of the 80 most recent articles listed in Medline (on 26/5/2014), 53 came from that journal. The publication frequency of eCAM and its increase in recent years beggars belief: in 2011, they published just over 500 articles which is already a high number, but, in 2012, the figure had risen to >800, and in 2013 it was >1300 (the equivalent 2013 figure for the BMJ/BMJ Open by comparison is 4, and that for another alt med journal, e.g. Forsch Komplement, is 10)
How do they do it? How can eCAM be so dominant in publishing alt med research? The trick seems to be fairly simple.
Let’s assume you are an alt med researcher and you have an article that you would like to see published. Once you submit it to eCAM, your paper is sent to one of the ~150 members of the editorial board. These people are almost all strong proponents of alternative medicine; critics are a true rarity in this group. At this stage, you are able to suggest the peer reviewers for your submission (all who ever accepted this task are listed on the website; they amount to several thousand!), and it seems that, with the vast majority of submissions, the authors’ suggestions are being followed.
It goes without saying that most researchers suggest colleagues for peer reviewing who are not going to reject their work (the motto seems to be “if you pass my paper, I will pass yours). Therefore even faily flimsy bits of research pass this peer review process and get quickly published online in eCAM.
This process explains a lot, I think: 1) the extraordinarily high number of articles published 2) why currently more than 50% of all alt med research originate from eCAM 3) why so much of it is utter rubbish.
Even the mere titles of some of the articles might demonstrate my point. A few examples have to suffice:
- Color distribution differences in the tongue in sleep disorder
- Wen-dan decoction improves negative emotions in sleep-deprived rats by regulating orexin-a and leptin expression.
- Yiqi Huoxue Recipe Improves Heart Function through Inhibiting Apoptosis Related to Endoplasmic Reticulum Stress in Myocardial Infarction Model of Rats.
- Protective Effects of Bu-Shen-Huo-Xue Formula against 5/6 Nephrectomy-Induced Chronic Renal Failure in Rats
- Effects and Mechanisms of Complementary and Alternative Medicine during the Reproductive Process
- Evidence-based medicinal plants for modern chronic diseases
- Transforming Pain into Beauty: On Art, Healing, and Care for the Spirit
This system of uncritical peer review and fast online publication seems to suit many of the people involved in this process: the journal’s owners are laughing all the way to the bank; there is a publication charge of US$ 2000 per article, and, in 2013, the income of eCAM must therefore have been well over US$2 000 000. The researchers are equally delighted; they get even their flimsiest papers published (remember: ‘publish or perish’!). And the evangelic believers in alternative medicine are pleased because they can now claim that their field is highly research-active and that there is plenty of evidence to support the use of this or that therapy.
But there are others who are not served well by eCAM habit of publishing irrelevant, low quality articles:
- professionals who would like to advance health care and want to see reliable evidence as to which treatments work and which don’t,
- the public who, in one way or another, pay for all this and might assume that published research tends to be relevant and reliable,
- the patients who have given their time to researchers in the hope that their gift will improve health care,
- ill individuals who hope that alternative treatments might relieve their suffering,
- politicians who rely on research to be reliable in order to arrive at the right decisions.
Come to think of it, the vast majority of people should be less than enchanted with eCAM and similar journals.
Boiron is the world’s biggest producer of homeopathic remedies. It also is a firm that is relatively active in research into homeopathy. Here is one of their investigations which I find most remarkable.
This study was designed to describe the sociodemographic and clinical characteristics of patients recommended allopathic and/or homeopathic medicines for influenza-like illness (ILI) or ear nose and throat ENT disorders by pharmacists in France and to investigate the effectiveness of these treatments.
The introduction of the article includes interesting information; it informs us that, although homeopathy is more popular in Europe than in the Unites States, sales of homeopathic medicines in the United States grew by more than 1,000% in the late 1970s and early 1980s and continue to grow. In parallel, the number of physicians specializing in homeopathy doubled between 1980 and 1982. In 2003, sales of homeopathic medicines in the United States were estimated to be between $300 and $450 million, with an average growth rate of approximately 8% per year. Homeopathic drugs are among the top 10 nonprescription products sold in the category of analgesics to treat coughs, colds, and flu. The sale of homeopathic medicines in the United States is controlled by the Food, Drug, and Cosmetic Act and regulations issued by the U.S Food and Drug Administration (FDA). Homeopathic medicines in the United States are subject to well-controlled regulatory processes that closely resemble those used for allopathic medicines. FDA regulations for the sale of homeopathic medicines in the United States state that they can only be sold without prescription if they are for self-limiting conditions such as the common cold…
Am I mistaken, or does that paragraph read a bit like a text written by the marketing team of Boiron wanting to establish their products in the US?
Anyway, the methodology and results of the study are described in the abstract as follows:
A prospective, observational, multicenter study was carried out in randomly selected pharmacies across the 8 IDREM medical regions of France. Pharmacies that agreed to participate recruited male or female patients who responded to the following inclusion criteria: age ≥ 12 years presenting with the first symptoms of an ILI or ENT disorder that were present for less than 36 hours prior to the pharmacy visit. All medicines recorded in the study were recommended by the pharmacists. The following data were recorded at inclusion and after 3 days of treatment: the intensity of 13 symptoms, global symptom score, and disease impact on daily activities and sleep. Two groups of patients were compared: those recommended allopathic medicine only (AT group) and those recommended homeopathic medicine with or without allopathic medicine (HAT group). The number and severity of symptoms, change in global symptom score, and disease impact on daily activities and sleep were compared in the 2 treatment groups after 3 days of treatment. Independent predictors of recommendations for homeopathic medicine were identified by multi- factorial logistic regression analysis.
A total of 242 pharmacies out of 4,809 (5.0%) contacted agreed to participate in the study, and 133 (2.8%) included at least 1 patient; 573 patients were analyzed (mean age: 42.5 ± 16.2 years; 61.9% female). Of these, 428 received allopathic medicines only (74.7%; AT group), and 145 (25.3%) received homeopathic medicines (HAT group) alone (9/145, 1.6%) or associated with allopathy (136/145, 23.7%). At inclusion, HAT patients were significantly younger (39.6 ± 14.8 vs. 43.4 ± 16.1 years; P less than 0.05), had a higher mean number of symptoms (5.2 ± 2.5 vs. 4.4 ± 2.5; P less than 0.01), and more severe symptoms (mean global symptom score: 24.3 ± 5.5 vs. 22.3 ± 5.8; P = 0.0019) than AT patients. After 3 days, the improvement in symptoms and disease impact on daily activities and sleep was comparable in both groups of patients.
From these findings, the authors draw the following conclusions: Patients recommended homeopathic medicine by pharmacists were younger and had more severe symptoms than those recommended allopathic medicine. After 3 days of treatment, clinical improvement was comparable in both treatment groups. Pharmacists have an important role to play in the effective management of ILI and ENT disorders.
And, to make perfectly clear what all this is about, the first sentence of the ‘discussion’ puts it to the point by stating that homeopathic medicine, with or without allopathic medicine, appears to be effective at alleviating the symptoms of ILI or ENT disorders.
As I have heard it said that Boiron seems to have the nasty habit of threatening their critics with legal action, I ought to be quite cautious in my assessment of this ‘masterpiece of promotion’. Yet a few comments must surely be permitted.
‘To describe the sociodemographic and clinical characteristics of patients recommended allopathic and/or homeopathic medicines’ is not what I personally find an interesting subject of research, nor is it anything that will affect health care meaningfully, I think. Yet ‘to investigate the effectiveness of these treatments’ is certainly interesting and important. I will therefore focus on this second aim of the study.
Hold on, was this really a ‘study’? On closer inspection, it seemed much more like a survey. People who felt that they were suffering from ILI and ENT disorders and thus went to a pharmacy to buy something for their problem were offered either homeopathic or conventional medicines. Those who accepted either of the recommendations were asked to fill out some self-assessment forms and received a phone call three days later to check their symptoms. 94% of all patients in the homeopathy group took homeopathic medicine in combination with ‘allopathic’ medicine (it is interesting, perhaps even telling, that this term used by the authors was invented by Hahnemann as an insult to conventional medicine!). There was no examination by a doctor to verify what condition the survey-participants were truly suffering from, and there was no verification that the information provided during the follow-up telephone call was in any way real. The most frequently recommended homeopathic medicine was Anas barbariae 200C (Oscillococcinum) which is Boiron’s famous homeopathically diluted (about one molecule per universe, I guess) duck-liver heavily promoted in France against colds and similar conditions.
As it turns out, those survey-participants who accepted the homeopathic recommendation were significantly younger than those who accepted the recommendation for a conventional treatment (many surveys confirm that younger people are more prone to trying alternative medicine than older ones). It stands to reason, that the younger (and therefore fitter) patients were in better general health and therefore might recover quicker than the older ones. But, in fact, they did not!
Could this be due to the homeopathic remedies actually delaying recovery? Of course not! Who would be silly enough to claim that homeopathy could have this (or any other) effect? According to the authors, it is due to the fact that this group ‘had more severe symptoms than those recommended allopathic medicine’. But, as I said, we have to take their word for it; there is no independent verification of this. It would, of course, be quite ridiculous to postulate that those survey-participants accepting homeopathy were also a little more introspective or concerned about their own health (perhaps even more gullible) and thus claimed more severe symptoms!
And what about the authors’ conclusion that clinical improvement was comparable in both treatment groups? Well, this is more than a little problematic, in my view: first, we have no independent verification of the ‘improvement’ in either group. Second, we don’t know that the conventional treatments actually worked, and it could well be that both approaches were similarly ineffective, and that the observed outcomes are merely a reflection of the natural history of the condition. And third, one might expect the homeopathic (younger) group to do not similarly well but slightly better, simply because the natural history of the illness would tend to be more benign in younger people.
Before I finish,I should make a brief comment about the authors’ courageous statement that homeopathic medicine, with or without allopathic medicine, appears to be effective at alleviating the symptoms of ILI or ENT disorders. I think, for the reasons I already provided, this is extremely doubtful. In my view, more critical scientists would have phrased the conclusions differently:
THIS SURVEY SHOWS THAT EVALUATING THE EFFECTIVENESS OF MEDICAL INTERVENTIONS REQUIRES A MORE RIGOROUS METHODOLOGY THAN THAT OF A SURVEY.
But perhaps this would be asking a little too much of the authors; after all, at the end of the article, we find this telling footnote: Laboratoires Boiron provided financial support for the study. Cognet-Dementhon, Thevenard, Duru, and Allaert received consulting fees from Laboratoires Boiron for this study. Danno and Bordet are employees of Laboratoires Boiron.
The question whether infant colic can be effectively treated with manipulative therapies might seem rather trivial – after all, this is a benign condition which the infant quickly grows out of. However, the issue becomes a little more tricky, if we consider that it was one of the 6 paediatric illnesses which were at the centre of the famous libel case of the BCA against my friend and co-author Simon Singh. At the time, Simon had claimed that there was ‘not a jot of evidence’ for claiming that chiropractic was an effective treatment of infant colic, and my systematic review of the evidence strongly supported his statement. The BCA eventually lost their libel case and with it the reputation of chiropractic. Now a new article on this intriguing topic has become available; do we have to reverse our judgements?
The aim of this new systematic review was to evaluate the efficacy or effectiveness of manipulative therapies for infantile colic. Six RCTs of chiropractic, osteopathy or cranial osteopathy alone or in conjunction with other interventions were included with a total of 325 infants. Of the 6 included studies, 5 were “suggestive of a beneficial effect” and one found no evidence of benefit. Combining all the RCTs suggested that manipulative therapies had a significant effect. The average crying time was reduced by an average of 72 minutes per day. This effect was sustained for studies with a low risk of selection bias and attrition bias. When analysing only those studies with a low risk of performance bias (i.e. parental blinding) the improvement in daily crying hours was no longer statistically significant.
The quality of the studies was variable. There was a generally low risk of selection bias but a high risk of performance bias. Only one of the studies recorded adverse events and none were encountered.
From these data, the authors drew the following conclusion: Parents of infants receiving manipulative therapies reported fewer hours crying per day than parents whose infants did not and this difference was statistically significant. Most studies had a high risk of performance bias due to the fact that the assessors (parents) were not blind to who had received the intervention. When combining only those trials with a low risk of such performance bias the results did not reach statistical significance.
Does that mean that chiropractic does work for infant colic? No, it does not!
The first thing to point out is that the new systematic review included not just RCTs of chiropractic but also osteopathy and cranio-sacral therapy.
The second important issue is that the effects disappear, once performance bias is being accounted for which clearly shows that the result is false positive.
The third relevant fact is that the majority of the RCTs were of poor quality. The methodologically best studies were negative.
And the fourth thing to note is that only one study mentioned adverse effects, which means that the other 5 trials were in breach of one of rather elementary research ethics.
What makes all of this even more fascinating is the fact that the senior author of the new publication, George Lewith, is the very expert who advised the BCA in their libel case against Simon Singh. He seems so fond of his work that he even decided to re-publish it using even more misleading language than before. It is, of course, far from me to suggest that his review was an attempt to white-wash the issue of chiropractic ‘bogus’ claims. However, based on the available evidence, I would have formulated conclusions which are more than just a little different from his; something like this perhaps:
The current best evidence suggests that the small effects that emerge when we pool the data from mostly unreliable studies are due to bias and therefore not real. This systematic review therefore fails to show that manipulative therapies are effective. It furthermore points to a serious breach of research ethics by the majority of researchers in this field.
Imagine an area of therapeutics where 100% of all findings of hypothesis-testing research is positive, i.e. comes to the conclusion that the treatment in question is effective. Theoretically, this could mean that the therapy is a miracle cure which is useful for every single condition in every single setting. But sadly, there are no miracle cures. Therefore something must be badly and worryingly amiss with the research in an area that generates 100% positive results.
Acupuncture is such an area; we and others have shown that Chinese trials of acupuncture hardly ever produce a negative finding. In other words, one does not need to read the paper, one already knows that it is positive – even more extreme: one does not need to conduct the study, one already knows the result before the research has started. But you might not believe my research nor that of others. We might be chauvinist bastards who want to discredit Chinese science. In this case, you might perhaps believe Chinese researchers.
In this systematic review, all randomized controlled trials (RCTs) of acupuncture published in Chinese journals were identified by a team of Chinese scientists. A total of 840 RCTs were found, including 727 RCTs comparing acupuncture with conventional treatment, 51 RCTs with no treatment controls, and 62 RCTs with sham-acupuncture controls. Among theses 840 RCTs, 838 studies (99.8%) reported positive results from primary outcomes and two trials (0.2%) reported negative results. The percentages of RCTs concealment of the information on withdraws or sample size calculations were 43.7%, 5.9%, 4.9%, 9.9%, and 1.7% respectively.
The authors concluded that publication bias might be major issue in RCTs on acupuncture published in Chinese journals reported, which is related to high risk of bias. We suggest that all trials should be prospectively registered in international trial registry in future.
I applaud the authors’ courageous efforts to conduct this analysis, but I do not agree with their conclusion. The question why all Chinese acupuncture trials are positive has puzzled me since many years, and I have quizzed numerous Chinese colleagues why this might be so. The answer I received was uniformly that it would be very offensive for Chinese researchers to conceive a study that does not confirm the views held by their peers. In other words, acupuncture research in China is conducted to confirm the prior assumption that this treatment is effective. It seems obvious that this is an abuse of science which must cause confusion.
Whatever the reasons for the phenomenon, and we can only speculate about them, the fact has been independently confirmed several times and is now quite undeniable: acupuncture trials from China – and these constitute the majority of the evidence-base in this area – cannot be trusted. The only way to adequately deal with this problem that I can think of is to discard them outright.
Today, there are several dozens of journals publishing articles on alternative medicine. ‘The Journal of Alternative and Complementary Medicine’ is one of the best known, and it has one of the highest impact factors of them all. The current issue holds a few ‘gems’ which might be worthy of a comment or two. Here I have selected three articles reporting clinical studies, and I reproduce their abstracts (almost) in full (in italics) and add my comments (for clarity in bold). All the articles are available electronically, and I have provided the links for those who want to investigate beyond the abstracts.
STUDY No 1
The first ‘pilot study‘ was aimed to demonstrate the potential of auricular acupuncture (AAT) for insomnia in maintenance haemodialysis (MHD) patients and to prepare for a future randomized controlled trial.
Eligible patients were enrolled into this descriptive pilot study and received AAT designed to manage insomnia for 4 weeks. Questionnaires that used the Pittsburgh sleep quality index (PSQI) were completed at baseline, after a 4-week intervention, and 1 month after completion of treatment. Sleep quality and other clinical characteristics, including sleeping pills taken, were statistically compared between different time points.
A total of 22 patients were selected as eligible participants and completed the treatment and questionnaires. The mean global PSQI score was significantly decreased after AAT intervention (p<0.05). Participants reported improved sleep quality (p<0.01), shorter sleep latency (p<0.05), less sleep disturbance (p<0.01), and less daytime dysfunction (p=0.01). They also exhibited less dependency on sleep medications, indicated by the reduction in weekly estazolam consumption from 6.98±4.44 pills to 4.23±2.66 pills (p<0.01). However, these improvements were not preserved 1 month after treatment.
Conclusions: In this single-center pilot study, complementary AAT for MHD patients with severe insomnia was feasible and well tolerated and showed encouraging results for sleep quality.
In alternative medicine research, it has become far too common (almost generally accepted) to call a flimsy trial a ‘pilot study’. The authors give their game away by stating that, by conducting this trial, they want to ‘demonstrate the potential of AAT’. This is not a legitimate aim of research; science is for TESTING hypotheses, not for PROVING them!
The results of this trial show that patients experienced improvements after receiving AAT which, however, did not last. As there was no placebo control group, the most likely explanation for these outcomes would be that AAT generated a short-lasting placebo effect.
A sample size of 22 is, of course, far to small to allow any conclusions about the safety of the intervention. Despite these obvious facts, the authors seem convinced that AAT is both safe and effective.
STUDY No 2
The aim of the second study was to compare the therapeutic effect of Yamamoto new scalp acupuncture (YNSA), a recently developed microcupuncture system, with traditional acupuncture (TCA) for the prophylaxis and treatment of migraine headache.
In a randomized clinical trial, 80 patients with migraine headache were assigned to receive YNSA or TCA. A pain visual analogue scale (VAS) and migraine therapy assessment questionnaire (MTAQ) were completed before treatment, after 6 and 18 sections of treatment, and 1 month after completion of therapy.
All the recruited patients completed the study. Baseline characteristics were similar between the two groups. Frequency and severity of migraine attacks, nausea, the need for rescue treatment, and work absence rate decreased similarly in both groups. Recovery from headache and ability to continue daily activities 2 hours after medical treatment showed similar improvement in both groups (p>0.05).
Conclusions: Classic acupuncture and YNSA are similarly effective in the prophylaxis and treatment of migraine headache and may be considered as alternatives to pharmacotherapy.
This is what is technically called an ‘equivalence trial’, i.e. a study that compares an experimental treatment (YNSA) to one that is (assumed to be) effective. To demonstrate equivalence, such trials need to have large sample sizes, and this study is woefully underpowered. As it stands, the results show nothing meaningful at all; if anything, they suggest that both interventions were similarly useless.
STUDY No 3
The third study determined whether injection with hypertonic dextrose and morrhuate sodium (prolotherapy) using a pragmatic, clinically determined injection schedule for knee osteoarthritis (KOA) results in improved knee pain, function, and stiffness compared to baseline status.
The participants were 38 adults who had at least 3 months of symptomatic KOA and who were in the control groups of a prior prolotherapy randomized controlled trial (RCT) (Prior-Control), were ineligible for the RCT (Prior-Ineligible), or were eligible but declined the RCT (Prior-Declined).
The injection sessions at occurred at 1, 5, and 9 weeks with as-needed treatment at weeks 13 and 17. Extra-articular injections of 15% dextrose and 5% morrhuate sodium were done at peri-articular tendon and ligament insertions. A single intra-articular injection of 6 mL 25% dextrose was performed through an inferomedial approach.
The primary outcome measure was the validated Western Ontario McMaster University Osteoarthritis Index (WOMAC). The secondary outcome measure was the Knee Pain Scale and postprocedure opioid medication use and participant satisfaction.
The Prior-Declined group reported the most severe baseline WOMAC score (p=0.02). Compared to baseline status, participants in the Prior-Control group reported a score change of 12.4±3.5 points (19.5%, p=0.002). Prior-Decline and Prior-Ineligible groups improved by 19.4±7.0 (42.9%, p=0.05) and 17.8±3.9 (28.4%, p=0.008) points, respectively; 55.6% of Prior-Control, 75% of Prior-Decline, and 50% of Prior-Ineligible participants reported score improvement in excess of the 12-point minimal clinical important difference on the WOMAC measure. Postprocedure opioid medication resulted in rapid diminution of prolotherapy injection pain. Satisfaction was high and there were no adverse events.
Conclusions: Prolotherapy using dextrose and morrhuate sodium injections for participants with mild-to-severe KOA resulted in safe, significant, sustained improvement of WOMAC-based knee pain, function, and stiffness scores compared to baseline status.
This study had nothing that one might call a proper control group: all the three groups mentioned were treated with the experimental treatment. No attempt was made to control for even the most obvious biases: the observed effects could have been due to placebo or any other non-specific effects. The authors conclusions imply a causal relationship between the treatment and the outcome which is wrong. The notion that the experimental treatment is ‘safe’ is based on just 38 patients and therefore not reasonable.
All of this might seem rather trivial, and my comments could be viewed as a deliberate and vicious attempt to discredit one of the most respected journals of alternative medicine. Yet, considering that articles of this nature are more the rule than the exception in alternative medicine, I do think that this flagrant lack of scientific rigour is a relevant issue and has important implications.
As long as research in this area continues to be deeply flawed, as long as reviewers turn a blind eye to (or are not smart enough to detect) even the most obvious mistakes, as long as journal editors accept any rubbish in order to fill their pages, there is a great danger that we are being continuously being mislead about the supposed therapeutic value of alternative therapies.
Many who read this blog will, of course, have the capacity to think critically and might therefore not fall into the trap of accepting the conclusions of fatally flawed research. But many other people, including politicians, journalists and consumers, might not have the necessary appraisal skills and will thus not be able to tell that such studies can serve only one purpose: to popularise bogus treatments and thereby render health care less effective and more dangerous. Enthusiasts of alternative medicine are usually fully convinced that such studies amount to evidence and ram this pseudo-information down the throat of health care decision makers – the effects of such lobbying on public health can be disastrous.
And there is another downside to the publication of such dismal drivel: assuming (as I do) that not all of alternative medicine is completely useless, such embarrassingly poor research will inevitably have detrimental effects on the discipline of alternative medicine. After being exposed to a seemingly endless stream of pseudo-research, critics will eventually give up taking any of it seriously and might claim that none of it is worth the bother. In other words, those who conduct, accept or publish such nonsensical papers are not only endangering medical progress in general, they are also harming the very cause they try so desperately hard to advance.
An article in the ‘Huffpost Healthy Living’ recently discussed “the top three things that surprise people about acupuncture”. On closer inspection, they turn out to be the top three untruths about acupuncture. Here is (in italics and slightly abbreviated) what the article said.
Acupuncture is not just for pain
…It’s true that acupuncture can work wonders on pain conditions…However, acupuncture can alleviate a wide variety of ailments that have nothing to do with physical pain. Whether you have digestive issues, gynecological conditions, emotional concerns such as anxiety and depression, asthma, seasonal allergies, you name it, acupuncture can help address your symptoms.
Acupuncturists go to school for a long time
People tend to be unaware of the extent to which acupuncturists train to become licensed in their profession. Many assume becoming an acupuncturist is similar to becoming a massage therapist or Reiki practitioner or yoga instructor… At minimum, a licensed acupuncturist in the United States has been to three years of graduate school. Four years is more common. They hold master’s degrees. Some acupuncturists with doctorates have studied at the graduate level for five-plus years. Upon graduating from an accredited school, all acupuncturists must pass multiple board exams to become licensed in their state. In addition to the academic and state requirements for practicing acupuncture, many acupuncturists seek hands-on training and mentorship in the form of apprenticeships and continuing education seminars.
Acupuncture is relaxing
Acupuncture needles are surprisingly thin. They do not bear any resemblance to needles that are used for injections or to draw blood… In most cases, the insertion of acupuncture needles does not hurt…Once the needles are in, they start working their magic, which is where the relaxation part comes in. Acupuncture helps shift your body out of sympathetic mode (fight or flight) and into parasympathetic mode (rest and digest). It mellows out the nervous system, decreases muscular tension, and helps quiet internal chatter…
AND NOW THE FACTS:
1) There is not a single condition for which the evidence is truly compelling demonstrating that acupuncture is more than a placebo. Certainly there is no good evidence that acupuncture works for digestive issues, gynecological conditions, emotional concerns such as anxiety and depression, asthma or seasonal allergies.
2) In most countries, anyone can call themselves an acupuncturist, regardless of background or training.
3) The relaxing element of an acupuncture session is foremost the fact that patients lie down and have to keep still for 20 minutes or so. The insertion of needles does cause mild pain in many patients, and the claim about parasympathetic mode is mostly phantasy.
I despair about the nonsense that is published about alternative medicine on a daily basis – not because I have an axe to grind, but because it misleads patients into making wrong therapeutic decisions.