The FDA just made the following significant announcement:
The Food and Drug Administration (FDA) is announcing a public hearing to obtain information and comments from stakeholders about the current use of human drug and biological products labeled as homeopathic, as well as the Agency’s regulatory framework for such products. These products include prescription drugs and biological products labeled as homeopathic and over-the-counter (OTC) drugs labeled as homeopathic. FDA is seeking participants for the public hearing and written comments from all interested parties, including, but not limited to, consumers, patients, caregivers, health care professionals, patient groups, and industry. FDA is seeking input on a number of specific questions, but is interested in any other pertinent information participants would like to share.
April 20-21, 2015
9:00 am to 4:00 pm
FDA White Oak Campus
10903 New Hampshire Avenue
Bldg. 31, Room 1503A (Great Room)
Silver Spring, Maryland 20993
Attendance, Registration, and Oral Presentations
Registration is free and available on a first-come, first-served basis. If you wish to attend or make an oral presentation, please reference section III of the forthcoming Federal Register Notice (Attendance and/or Participation in the Public Hearing) for information on how to register and the deadline for registration.
If you cannot attend in person, information about how you can access a live Webcast will be located at Homeopathic Product Regulation
The agenda will be posted soon
And this is what Reuters reported about the planned event:
The hearing, scheduled for April 20-21, will discuss prescription drugs, biological products, and over-the-counter drugs labeled homeopathic, a market that has expanded to become a multimillion dollar industry in the United States. The agency is set to evaluate its regulatory framework for homeopathic products after a quarter century. (http://1.usa.gov/1Hxwup3) An Australian government study released this month concluded that homeopathy does not work. (http://bit.ly/1BheAmR) The FDA issued a warning earlier this month asking consumers not to rely on asthma products labeled homeopathic that are sold over the counter. (http://1.usa.gov/1EEuKrC) Homeopathic medicines include pellets placed under the tongue, tablets, liquids, ointments, sprays and creams. The basic principles of homeopathy, formulated by German physician Samuel Hahnemann in the late 18th century, are based on a theory that a disease can be treated using small doses of natural substances that in a healthy person would produce symptoms of the disease. The agenda for the hearing will be posted soon, the FDA said on Tuesday.
In my view, this is an important occasion for experts believing in evidence to make their position regarding homeopathy heard. I therefore encourage all my readers who have an evidence-based opinion on homeopathy to submit it to the hearing.
Homeopathy has a long history in Canada. In 1842, James Lilli was probably the first Canadian homeopath to begin practicing in Toronto. Joseph J. Lancaster, who had studied in New York, began practicing sometime in the 1840s in Ontario. The ‘Homeopathic Medical Society of Canada’ was established in 1854 in Hamilton, Ontario.
Since these early days much has changed. At present, all health care professions in Ontario are governed by the ‘Regulated Health Professions Act’ which allows all health-care professions the same right to practice. This law upholds the belief that the public has the right to choose what health care it wishes, and that the government should only intervene to regulate where a profession poses a significant risk of harm to the public. Only allopathic professions are currently regulated, and, in Ontario, doctors are censured if they practice homeopathy.
All schools of homeopathy in Canada offer ‘diplomate status’ and all offer three-year, part-time courses (one or two weekends per month plus perhaps one or two evenings per week). There are no legal doctorate or university degree programs for homeopathy in Canada. A doctorate in any field other than allopathic medicine cannot legally be used while practicing homeopathy.
I have been reliably informed that the regulation of homeopathy in Ontario is about to change. A transitional council of the ‘College of Homeopaths of Ontario’ had already been appointed in September 2009. The next step in the regulatory process is now imminent. On April 1 this year, Ontario will proclaim the ‘Homeopathy Act’. The bill will further empower the ‘College of Homeopaths of Ontario’. This regulatory body will hence forth have control over who gets to call themselves a homeopath. In addition, it will also have a complaint tracking system.
This moves comes only days after the ‘Australian National Health and Medical Research Council’ has published the most thorough and independent assessment of homeopathy in the history of this form of alternative therapy. It concluded that homeopathy should not be used to treat health conditions that are chronic, serious, or could become serious. People who choose homeopathy may put their health at risk if they reject or delay treatments for which there is good evidence for safety and effectiveness.
In view of the fact that homeopathy has been disproven as a treatment that fails to have a positive risk/benefit balance, the move of the regulators in Ontario seems pure madness to me. It sends the wrong signal to consumers and gives credibility to a form of quackery.
In a nutshell: EVEN THE BEST REGULATION OF NONSENSE WILL RESULT IN NONSENSE!
The other day, I received a request from THE GUARDIAN: could I write a piece on homeopathy in relation to the Australian report which had just come out; they gave me ~700 words and all of 3 hours to do it. I had an extremely busy day, but accepted the challenge nevertheless.
My article was published the next day and the ‘headliner’ at THE GUARDIAN had elected to call it There is no scientific case for homeopathy: the debate is over.
What followed was a flurry of debate – well over 2200 comments – which was more than a little ironic, considering the headline.
Essentially, my article had repeated the well-rehearsed arguments which have so often been made on this blog and elsewhere:
• Our trials failed to show that homeopathy is more than a placebo.
• Our reviews demonstrated that the most reliable of the 230 or so trials of homeopathy ever published are also not positive.
• Studies with animals confirmed the results obtained on humans.
• Surveys and case reports suggested that homeopathy can be dangerous.
• The claims made by homeopaths to cure conditions like cancer, asthma or even Ebola were bogus.
• The promotion of homeopathy is not ethical.
The comments that followed were mixed, of course; those that disagreed with me used a range of counter-arguments; in no specific order, these were the following:
- For several reasons, I cannot be trusted.
- I even once stated that I have treated my wife homeopathically.
- The Australian report was neither thorough nor reliable.
- The Australian expert panel were bought by Big Pharma.
- Homeopathic treatment must be individualised and can therefore not be tested in RCTs.
- Just because we don’t understand how homeopathy works, we should not conclude that it is ineffective.
- 200 years of positive experience with homeopathy clearly prove that it works.
- The huge popularity of homeopathy worldwide demonstrated its effectiveness.
- The fact that some very clever people support homeopathy shows that it works.
- Homeopathy works in animals and little children, therefore it cannot be just a placebo.
- The Queen and my aunt Doris use homeopathy.
- Placebos work.
- Patients must be able to choose; patient choice is an important principle in all health care.
- There’s more to evidence than just RCTs.
- Homeopathy works like vaccines.
With such an abundance of counter-arguments, the debate is clearly NOT over! Or is it? Let’s see how solid the arguments really are.
1) I cannot be trusted
Ad hominem attacks are no arguments at all; they are merely a sign that the person using them has no real arguments left.
2) I treated my wife homeopathically
This is true. At one stage in my life, I treated anyone who couldn’t run fast enough to escape me with homeopathy. What does that show? It simply shows that I can make mistakes too.
3) The Australian report was flawed
Perhaps it was not entirely faultless (no report ever is), but it certainly was rigorous – more so than any previous document in the entire history of homeopathy. If it excluded certain types of evidence, like the observational studies (which are so much loved by homeopaths), it did so because such data are wide open to bias.
4) The panel was not independent
Yes, it was! It even included a homeopath. The Australian National Health and Medical Research Council is internationally highly respected, and to defame it without evidence is, in a way, just another ad hominem attack.
5) Homeopathy must be individualised
This is a half-truth: classical homeopathy is mostly individualised, but lots of homeopathic prescribing is not individualised. And in any case, we have recently seen how totally unconvincing the results of strictly individualised trials of homeopathy are. This argument turns out to be a red herring.
6) We currently don’t understand how homeopathy works
What we do understand perfectly well, however, is the fact that no explanation exists which would not require throwing over board big chunks of the laws of nature. But even if we accepted that the mode of action is unknown, this would not change the lack of homeopathy’s clinical effectiveness. Lots of treatments work without us understanding how.
7) Experience shows it works
Experience is a very unreliable indicator of effectiveness; there are simply far too many confounders such as placebo effects, regression towards the mean or natural history of the disease. This is why we need evidence to be sure, and historically medicine finally started making progress when this lesson had been learnt.
8) The amazing popularity of homeopathy is proof of its effectiveness
This is the ‘argumentum ad populum’ fallacy. Think of the popularity of blood-letting to see how wrong this argument can be.
9) Homeopathy is backed by some very clever people
So what? Clever people are not always correct – look at me (just joking!)
10) Homeopathy works in animals and little children which proves that it is more than a placebo
First, animals and children do also show placebo-responses.
Second, the animal owner/parent might respond to placebo and thus mimic a placebo-response in the patient.
Third, the evidence for homeopathy is not positive neither in animals nor in children.
11) The Queen swears by homeopathy
Yes, so much so that, as soon as she is really ill, she makes use of what the very best of conventional medicine has to offer.
12) Placebos work
For sure! But that does not mean that we should prescribe placebos. If an effective treatment is given with compassion and empathy, the patient will also profit from a placebo effect – in addition to the effect of the treatment. Merely administering placebos means withholding the latter and is thus not in the best interest of the patient.
13) Patient choice
Yes, patient choice is important. However, it only applies to the choice between treatments that are demonstrably effective – if not choice becomes arbitrariness.
14) Evidence is more than just RCTs
True, there are many study designs other than RCTs. They all have their place in research – but when the research question is to test whether a treatment is effective beyond placebo, they are all open to different types of bias. The one that minimises bias best and thus produces more reliable findings than any other study design is the placebo-controlled, double-blind RCT.
15) Homeopathy works like vaccines
No! The ‘like cures like principle’ appears to be similar to the principles of vaccination, but this appearance is misleading. Vaccines contain small amounts of active material, while the typical homeopathic remedy doesn’t. Vaccines use the substance that causes the illness, e. g. (parts of) a virus, while homeopathy doesn’t.
So, is there still a debate? Obviously there is – the Guardian headliner was wrong – but it is a debate without reasonable arguments. And in the public domain, the debate is dominated by enthusiasts who endlessly repeat nonsensical notions which have been shown to be wrong over and over again.
In a nutshell:
Yes, there continues to be a debate.
No, there is no reasonable debate.
I have argued since many years that pharmacists should not be selling or promoting homeopathic and other remedies for which there is no proof of efficacy – the last time I published my view on this matter is even less than a week ago: Personally, I would go another step further and remind pharmacists who sell homeopathic remedies to the unsuspecting public that it is unethical to pretend they are more than placebos.
Despite my insistence and despite the fact that many agree with me (at least privately), there are precious few pharmacists who actually do something meaningful about the current situation. And there is very little visible change: in the UK, it is currently hard to find a pharmacy where homeopathic remedies are not on the shelves, and certainly all the major chains seem to put money before health care ethics.
I am, of course, speaking about the situation in the UK, France, Germany and some other European countries. Perhaps elsewhere things are different?
A NZ website seems to indicate that ‘down under’ the pharmacists are getting more active. Some strongly argue against unproven or disproven remedies in pharmacies:
Firstly, …it’s not a case that “pharmacists ‘should’ only be selling health products for which there is credible evidence of efficacy” (alterations mine, emboldened) but that they are obliged to—but choose not to. Their ethical guidelines state -
[PHARMACISTS] MUST:… Only purchase, supply or promote any medicine, complementary therapy, herbal remedy or other healthcare product where there is no reason to doubt its quality or safety and when there is credible evidence of efficacy.
…Secondly, the argument that ‘other businesses sell junk remedies therefore we shall’ is unsound. One of the key points about the ethical regulations for pharmacies is that customers should be able to walk into a store and have an expectation that the remedies within the store are basically sound. If other businesses elect to be unsound, that’s poor health practice, but no justification to do likewise. On the face of it, it would seem that the profit motive is ruling over sound and ethical practice.
Thirdly, that some GPs subscribe placebos should have no standing in this. There is some arguments for GPs to prescribe placebo remedies in some cases; others would argue that education is a better response in most cases. Either way—and just my opinion—it seems to me that GPs prescribing homeopathic remedies encourages people to think these have real remedial effects. I don’t work within the industry, but I am sure are ways of offering placebos that avoid using off-the-shelf commercial products. One might be that patients only get placebo ‘treatments’ via prescription.
…Fourthly, Pharmacy Today encourages that “pharmacies need to reconsider their stance in the light of this report”***. While this is an excellent idea, and one I thoroughly support, I suspect the underlying driver isn’t the report, but media presence on the topic. There is a long trail of evidence over many years showing that homeopathic remedies are not effective for anything.
The Australian study*** that prompted the latest round of interest drew this statement,
Based on the assessment of the evidence of effectiveness of homeopathy, NHMRC concludes that there are no health conditions for which there is reliable evidence that homeopathy is effective.
Homeopathy should not be used to treat health conditions that are chronic, serious, or could become serious. People who choose homeopathy may put their health at risk if they reject or delay treatments for which there is good evidence for safety and effectiveness. People who are considering whether to use homeopathy should first get advice from a registered health practitioner.* Those who use homeopathy should tell their health practitioner and should keep taking any prescribed treatments.
The National Health and Medical Research Council expects that the Australian public will be offered treatments and therapies based on the best available evidence.
…Why were the relevant professional bodies not onto this evidence sooner?…
I might add another one: why are the European professional bodies of pharmacy doing so little about this ongoing breach of their own ethical codes?
(*** the report that the author refers to is the one by the Australian National Health and Medical Research Council we discussed on this blog a few days ago.)
According to Bloomberg Markets, A Nelson & Co Ltd. manufactures and markets natural healthcare products. The company offers arnica creams that provide natural first aid for bruises; plant and flower based remedies that help in managing the emotional demands of everyday life; and over-the-counter homeopathic medicines for everyday ailments, such as relief from travel sickness and relief for the symptoms of hay fever. It also provides hemorrhoid relief creams and soothing hygienic wipes; anti-blemish range products for various skin types and age groups; multi-purpose cream that helps to soothe and restore skin; iron supplements; teething granules that provide relief from the symptoms and discomfort of teething; a range of creams, ointments, and sprays for a range of common skin conditions/complaints; and a range of commonly used herbal remedies. The company offers products for ailments, including aches and pains, mild anxiety, babies and children, colds and minor infections, digestion, emotional health, energy, everyday stresses, first aid, getting older, pets, quit smoking, skin, sleep, travel, and women’s health. It also operates a clinic; and a pharmacy that offers homeopathy and complementary healthcare products. The company offers its products through its pharmacy in the United Kingdom; and distributors in Europe, Latin America, and internationally. It also serves customers online. The company was formerly known as Armbrecht, Nelson & Co. The company was founded in 1860 and is based in London, United Kingdom with subsidiary offices in Boston, Massachusetts; and Hamburg, Germany. A Nelson & Co Ltd. operates as a subsidiary of Nelson and Russell Holdings Ltd.
In the journal ‘Chemist and Druggist’ we find an article informing us that, in 1930, Nelsons Homeopathic Pharmacy was approached by Dr Edward Bach who wanted help making and selling his products. He had created 38 flower remedies to rebalance emotions and later created an emergency remedy, a combination of five flower remedies that became Rescue. The relationship between Nelsons and the Dr Edward Bach Centre, based at Dr Bach’s former home at Mount Vernon in Oxfordshire, continues to this day and both the Bach Original Flower Remedies and Rescue are key ranges for Nelsons.
Nelson’s homeopathic pharmacy has a proud history:
Ernst Louis Armbrecht, a German pharmacist and disciple of Samuel Hahnemann, came to London and founded Nelsonsin 1860. Since then, Nelsons has been supplying homeopathic medicines. “Our wish today” they state “is the same as 152 years ago: to make homeopathy accessible and to provide the highest standards of medicine and advice.”
The highest standards of medicine and advice? It seems that the Advertising Standards Authority (ASA) disagrees. A recent ASA Adjudication on A Nelson & Co Ltd deals with an advertisement by Nelsons for ‘Bach Rescue Night’ which stated “I CAN’T SWITCH OFF…The RESCUE NIGHT range helps your mind switch off, so you can enjoy a natural night’s sleep”
A freelance health writer had challenged whether the claims “I can’t switch off … Rescue Night range helps your mind switch off, so you can enjoy a natural night’s sleep” was an authorised health claim in the EU Register of Nutrition and Health Claims for Foods (the EU Register).
The ASA noted that, according to EC Regulation 1924/2006 on Nutrition and Health Claims made on Foods (the Regulation), which was reflected in the CAP Code, only health claims which appeared on the list of authorised health claims (the Register) could be made in ads promoting foods, including food supplements. Health claims were defined as those that stated, suggested or implied that a relationship existed between a food category, a food or one of its constituents and health.
The ASA furthermore stated: We acknowledged Rescue Remedy’s assertion that their ad had not made specific claims to aid sleep or that it improved sleep. However, we considered that the use of visuals such as a crescent moon and stars on a dark background, that the letter ‘O’ in the word “OFF” resembled a simple on/ off light switch image, the text “… you can enjoy a natural night’s sleep” and the name of the product “Rescue Night” was likely to give the impression to consumers that it was a product that would aid sleep or that it would help consumers fall asleep easily. We understood that ‘unwanted thoughts’ was one reason why consumers might find it difficult to get to sleep and, again, considered this added to the impression that the product would contribute positively to sleep. We therefore considered that the ad made a health claim related to sleep involving a food item.
We understood that some Bach Flower Remedies contained levels of alcohol which would preclude them from bearing health claims altogether, however, we noted that Bach Rescue Night was alcohol free. We acknowledged Rescue Remedy’s points regarding EFSA and ‘on hold’ claims for botanicals. We understood that ‘on hold’ claims for such botanicals could be used in marketing, provided such use had the same meaning as the proposed claim and they were used in compliance with applicable existing national provisions (in this case the CAP Code). However, Rescue Remedy did not provide evidence that relevant proposed claims for white chestnut, or any of the other product ingredients were ‘on hold’. Nevertheless, we understood that there were no ‘on hold’ claims entered onto the Register for white chestnut or the other product ingredients. Furthermore, ‘on hold’ claims should also be supported with adequate substantiation which we did not receive.
Because the ad made health claims relating to Bach Rescue Night as a sleep aid and we had not seen evidence that relevant claims for the botanical ingredients contained in the product were ‘on hold’, we concluded that the ad breached the Code.
The ad breached CAP Code (Edition 12) rules 15.1, 15.1.1 and 15.7 (Food, food supplements and associated health or nutritional claims).
The ASA ruled that the ad must not appear again in its current form. We told A Nelson & Co Ltd t/a rescueremedy.co.uk not to make health claims for botanical ingredients if they did not comply with the requirements of the Regulation.
I am afraid that such a ruling will have very little effect on the sale of Bach Flower Remedies. In case you have any doubt, I should mention that these inventions of Dr Bach are not supported by good evidence. Here is the abstract of my systematic review on the subject:
Bach flower remedies continue to be popular and its proponents make a range of medicinal claims for them. The aim of this systematic review was to critically evaluate the evidence for these claims. Five electronic databases were searched without restrictions on time or language. All randomised clinical trials of flower remedies were included. Seven such studies were located. All but one were placebo-controlled. All placebo-controlled trials failed to demonstrate efficacy. It is concluded that the most reliable clinical trials do not show any differences between flower remedies and placebos.
Bach Flower Remedies have no effect whatsoever!
Come to think of it, this is not entirely true: they obviously keep the ASA busy, they exploit the gullible public, and they are clearly good for the cash flow at Nelson’s.
I reported previously on this blog that the Australian National Health and Medical Research Council (NH&MRC) has made the most thorough and independent assessment of homeopathy in the history of this form of alternative therapy. The report published at the time was preliminary, and further evidence from parties that were critical of it needed to be taken into account.
Today (11/3/2015), the NH&MRC has released its final report on homeopathy. In essence, it concluded that there is no scientific basis for homeopathy and no quality evidence of its efficacy.
The report is similar in its conclusions to the UK Government Science and Technology 2010 Report on Homeopathy.
The summary of the report sates the following:
Homeopathy should not be used to treat health conditions that are chronic, serious, or could become serious. People who choose homeopathy may put their health at risk if they reject or delay treatments for which there is good evidence for safety and effectiveness. People who are considering whether to use homeopathy should first get advice from a registered health practitioner. Those who use homeopathy should tell their health practitioner and should keep taking any prescribed treatments.
Professor Warwick Anderson, CEO of the Council, commented: “All medical treatments and interventions should be underpinned by reliable evidence. NH&MRC’s review shows that there is no good quality evidence to support the claim that homeopathy works better than a placebo.”
The report confirms what many scientific and public health experts have maintained since about 200 years, that not only does homeopathy not work, if it did, chemistry, physics and physiology could not.
“In short, the continued promotion of homeopathy would represent a commercial scam preying on the gullible,” said Professor John Dwyer AO, President of Friends of Science in Medicine (FSM), a non-industry aligned advocacy group for quality evidence in health. “Worse still, as a placebo therapy, harm may occur when it replaces effective therapies, as those given homeopathic treatments may wrongly think that they and their children are protected from serious infections,” he added.
The report emphasises that ‘People who choose homeopathy may put their health at risk, if they reject or delay treatments for which there is good evidence for safety and effectiveness’.
“With so many pressures on the Australian health dollar the Government must stop subsidising unproven alternative therapies such as homeopathy through student loans for homeopathy and similar courses,” said Professor Rob Morrison OAM, Co-Vice President of FSM, “The government should also not subsidise health funds offering cover for pseudoscientific electives of this kind,” said Professor Morrison. “The government should at least mandate that health funds offer a cheaper option that does not include ‘belief based’ therapies,” he said.
“It is completely unprofessional and unethical now for pharmacies to sell homeopathic products,” said Emeritus Professor Alastair MacLennan AO, Co-Vice President of FSM, “These products should now be removed from pharmacies and retail outlets”.
Highly diluted homeopathic remedies are pure placebos; at least this is what sceptics have been saying for about 200 years. This assumption is based on the fact that homeopathy’s plausibility is close to zero and that the totality of the reliable evidence fails to demonstrate that it works beyond placebo for any condition.
But, if this is true, why do so many patients swear by homeopathy and experience benefit from it? This question has been answered many times: THE BENEFIT IS NOT DUE TO THE REMEDY BUT TO NON-SPECIFIC EFFECTS OF THE CONSULTATION.
More confirmation for this conclusion comes from an unexpected source.
Indian homeopaths recently published a trial of individualized homeopathy in osteoarthritis. To be more precise, it was a prospective, parallel-arm, double-blind, randomized, placebo-controlled pilot study which was conducted from January to October 2014 involving 60 patients (homeopathy, n = 30; placebo, n = 30). All patients were suffering from acute painful episodes of knee osteoarthritis and visiting the outpatient clinic of Mahesh Bhattacharyya Homeopathic Medical College and Hospital, West Bengal, India.
The results show statistically significant reduction in 3 visual analogue scales (measuring pain, stiffness, and loss of function) and Osteoarthritis Research Society International scores in both groups over 2 weeks (P < .05). However, group differences were not significant (P > .05).
The authors conclude that, overall, homeopathy did not appear to be superior to placebo; still, further rigorous evaluation in this design involving a larger sample size seems feasible in future.
Considering what I wrote above, I would alter these conclusion to something much more reasonable: further studies of homeopathy are certainly feasible. However, they are neither necessary nor desirable.
TO PUT IT DIFFERENTLY: HOMEOPATHY BELONGS IN THE BOOKS OF MEDICAL HISTORY.
Homeopathy has many critics who claim that there is no good evidence for this type of therapy. Homeopaths invariably find this most unfair and point to a plethora of studies that show an effect. They are, of course, correct! There are plenty of trials that suggest that homeopathic remedies do work. The question, however, is HOW RELIABLE ARE THESE STUDIES?
Here is a brand new one which might stand for dozens of others.
In this study, homeopaths treated 50 multimorbid patients with homeopathic remedies identifies by a method called ‘polarity analysis’ (PA) and prospectively followed them over one year (PA enables homeopaths to calculate a relative healing probability, based on Boenninghausen’s grading of polar symptoms).
The 43 patients (86%) who completed the observation period experienced an average improvement of 91% in their initial symptoms. Six patients dropped out, and one did not achieve an improvement of 80%, and was therefore also counted as a treatment failure. The cost of homeopathic treatment was 41% of projected equivalent conventional treatment.
Good news then for enthusiasts of homeopathy? 91% improvement!
Yet, I am afraid that critics might not be bowled over. They might smell a whiff of selection bias, lament the lack of a control group or regret the absence of objective outcome measures. But I was prepared to go as far as stating that such results might be quite interesting… until I read the authors’ conclusions that is:
Polarity Analysis is an effective method for treating multimorbidity. The multitude of symptoms does not prevent the method from achieving good results. Homeopathy may be capable of taking over a considerable proportion of the treatment of multimorbid patients, at lower costs than conventional medicine.
Virtually nothing in these conclusions is based on the data provided. They are pure extrapolation and wild assumptions. Two questions seem to emerge from this:
- How on earth can we take this and so many other articles on homeopathy seriously?
- When does this sort of article cross the line between wishful thinking and scientific misconduct?
During the next few weeks, I will post several short excerpts from my new book ‘A SCIENTIST IN WONDERLAND‘. Its subtitle already discloses much of what it is all about: ‘A MEMOIRE OF SEARCHING THE TRUTH AND FINDING TROUBLE’.
Some of my critics are likely to claim that I engage in this form of ‘promotion’ because I want to maximise my income by enticing my readers to buy the book. This is partly true, of course: after having worked very hard on this book for about 5 years, I want it to be read (but, at the same time, my critics would be mistaken: I do not expect to get rich on my new book – I am not that naïve; this ‘memoire’ will never be found in any best-seller list, I am sure). So, I suggest (if you do not want me to profit in any way) that you read my memoire after you got it from your library (which obviously would not affect my cash-flow all that much).
So here it is: with much trepidation and even more excitement I present to you the very first, short excerpt (as I said, there will be more).
There are some people, a fortunate few, who seem to know from an early age where they want to go in life, and have no trouble getting there.
I was not one of them. I was born in Germany in the years immediately following the end of World War II and, like many German children of that era, I was acutely aware of the awkwardness and unease that my elders displayed when it came to discussions that touched on the country’s recent history. Even as a young boy, I was conscious that there was a large and restive skeleton in the nation’s closet, and that it belonged to all of us – even to those of us who had not been alive during the Nazi era were somehow nevertheless its legatees, inextricably bound to it simply by the awareness of its existence.
With time, the growing realization that so many of our peers – teachers, uncles, aunts; perhaps even our own parents – had lent their assent, or worse, their enthusiastic assistance to the Nazi regime robbed their generation of its moral authority and left us, their children, unmoored and adrift.
In a profound sense I felt homeless. An accident of fate had landed me on the planet with a German passport, and with German as my mother tongue, but where did I really belong? Where would I go? What would I do with my life?
There had been physicians in my family for generations and there was always an expectation that I, too, would enter that profession. Yet I felt no strong pull towards medicine. As a young man my only real passion was music, particularly jazz, with its anarchic improvisations and disobedient rhythms; and the fact that it had been banned by the Nazis only made it all the more appealing to me. I would have been perfectly happy to linger indefinitely in the world of music, but eventually, like a debt come due, medicine summoned me, and I surrendered myself to the profession of my forebears.
In hindsight I am glad that my mother nudged me gently yet insistently in the direction of medical school. While music has delighted and comforted me throughout my life, it has been medicine that has truly defined me, stretching, challenging and nourishing me intellectually, even as it tested me on a personal level almost to the limits of my endurance.
Certainly, I had never anticipated that asking basic and necessary questions as a scientist might prove so fiercely controversial, and that as a result of my research I might become involved in ideological wrangling and political intrigue emanating from the highest level.
If I had known the difficulties I would face, the stark choices, the conflicts and machinations that awaited me, would I have chosen to spend my life in medicine? Yes, I would. Becoming a physician and pursuing the career of a scientist has afforded me not only the opportunity to speak out against the dangerous and growing influence of pseudoscience in medicine, but also, paradoxically, has given me both the reason and the courage to look back steadily at the unbearable past.
This is the story of how I finally found where I belong.
As promised, I will try with this post to explain my reservations regarding the new meta-analysis suggesting that individualised homeopathic remedies are superior to placebos. Before I start, however, I want to thank all those who have commented on various issues; it is well worth reading the numerous and diverse comments.
To remind us of the actual meta-analysis, it might be useful to re-publish its abstract (the full article is also available online):
A rigorous and focused systematic review and meta-analysis of randomised controlled trials (RCTs) of individualised homeopathic treatment has not previously been undertaken. We tested the hypothesis that the outcome of an individualised homeopathic treatment approach using homeopathic medicines is distinguishable from that of placebos.
The review’s methods, including literature search strategy, data extraction, assessment of risk of bias and statistical analysis, were strictly protocol-based. Judgment in seven assessment domains enabled a trial’s risk of bias to be designated as low, unclear or high. A trial was judged to comprise ‘reliable evidence’ if its risk of bias was low or was unclear in one specified domain. ‘Effect size’ was reported as odds ratio (OR), with arithmetic transformation for continuous data carried out as required; OR > 1 signified an effect favouring homeopathy.
Thirty-two eligible RCTs studied 24 different medical conditions in total. Twelve trials were classed ‘uncertain risk of bias’, three of which displayed relatively minor uncertainty and were designated reliable evidence; 20 trials were classed ‘high risk of bias’. Twenty-two trials had extractable data and were subjected to meta-analysis; OR = 1.53 (95% confidence interval (CI) 1.22 to 1.91). For the three trials with reliable evidence, sensitivity analysis revealed OR = 1.98 (95% CI 1.16 to 3.38).
Medicines prescribed in individualised homeopathy may have small, specific treatment effects. Findings are consistent with sub-group data available in a previous ‘global’ systematic review. The low or unclear overall quality of the evidence prompts caution in interpreting the findings. New high-quality RCT research is necessary to enable more decisive interpretation.
Since my team had published an RCTs of individualised homeopathy, it seems only natural that my interest focussed on why the study (even though identified by Mathie et al) had not been included in the meta-analysis. Our study had provided no evidence that adjunctive homeopathic remedies, as prescribed by experienced homeopathic practitioners, are superior to placebo in improving the quality of life of children with mild to moderate asthma in addition to conventional treatment in primary care.
I was convinced that this trial had been rigorous and thus puzzled why, despite receiving ‘full marks’ from the reviewers, they had not included it in their meta-analysis. I thus wrote to Mathie, the lead author of the meta-analysis, and he explained: For your trial (White et al. 2003), under domain V of assessment, we were unable to extract data for meta-analysis, and so it was attributed high risk of bias, as specified by the Cochrane judgmental criteria. Our designated main outcome was the CAQ, for which we needed to know (or could at least estimate) a mean and SD for both the baseline and the end-point of the study. Since your paper reported only the change from baseline in Table 3 or in the main text, it is not possible to derive the necessary end-point for analysis.
It took a while and several further emails until I understood: our study did report both the primary (Table 2 quality of life) and secondary outcome measure (Table 3 severity of symptoms). The primary outcome measure was reported in full detail such that a meta-analysis would have been possible. The secondary outcome measure was also reported but not in full detail, and the data provided by us would not lend themselves to meta-analyses. By electing not our primary but our secondary outcome measure for their meta-analysis, Mathie et al were able to claim that they were unable to use our study and reject it for their meta-analysis.
Why did they do that?
The answer is simple: in their methods section, they specify that they used outcome measures “based on a pre-specified hierarchical list in order of greatest to least importance, recommended by the WHO“. This, I would argue is deeply flawed: the most important outcome measure of a study is usually the one for which the study was designed, not the one that some guys at the WHO feel might be important (incidentally, the WHO list was never meant to be applied to meta-analyses in that way).
By following rigidly their published protocol, the authors of the meta-analysis managed to exclude our negative trial. Thus they did everything right – or did they?
Well, I think they committed several serious mistakes.
- Firstly, they wrote the protocol, which forced them to exclude our study. Following a protocol is not a virtue in itself; if the protocol is nonsensical it even is the opposite. Had they proceeded as is normal in such cases and used our primary outcome measure in their meta-analyses, it is most likely that their overall results would not have been in favour of homeopathy.
- Secondly, they awarded our study a malus point for the criterium ‘selective outcome reporting’. This is clearly a wrong decision: we did report the severity-outcome, albeit not in sufficient detail for their meta-analysis. Had they not committed this misjudgment, our RCT would have been the only one with an ‘A’ rating. This would have very clearly highlighted the nonsense of excluding the best-rated trial from meta-analysis.
There are several other oddities as well. For instance, Mathie et al judge our study to be NOT free of vested interest. I asked Mathie why they had done this and was told it is because we accepted free trial medication from a homeopathic pharmacy. I would argue that my team was far less plagued by vested interest than the authors of their three best (and of course positive) trials who, as I happen to know, are consultants for homeopathic manufacturers.
And all of this is just in relation to our own study. Norbert Aust has uncovered similar irregularities with other trials and I take the liberty of quoting his comments posted previously again here:
I have reason to believe that this review and metaanalysis in biased in favor of homeopathy. To check this, I compared two studies (1) Jacobs 1994 about the treatment of childhood diarrhea in Nicaragua, (2) Walach 1997 about homeopathic threatment of headaches. The Jacobs study is one of the three that provided ‘reliable evidence’, Walach’s study earned a poor C2.2 rating and was not included in the meta-analyses. Jacobs’ results were in favour of homeopathy, Walach’s not.
For the domains where the rating of Walach’s study was less than that of the Jacobs study, please find citations from the original studies or my short summaries for the point in question.
Domain I: Sequence generation:
“The remedy selected was then mailed to a notary public who held a stock of placebos. The notary threw a dice and mailed either the homeopathic remedy or an appropriate placebo. The notary was provided with a blank randomisation list.”
Rating: UNCLEAR (Medium risk of bias)
“For each of these medications, there was a box of tubes in sequentially numbered order which had been previously randomized into treatment or control medication using a random numbers table in blocks of four”
Rating: YES (Low risk of bias)
Domain IIIb: Blinding of outcome assessor
“The notary was provided with a blank randomization list which was an absolutely unique document. It was only handed out after the biometrician (WG) had deposited all coded original data as a printout at the notary’s office. (…) Data entry was performed blindly by personnel not involved in the study. ”
Rating: UNCLEAR (Medium risk of bias)
“All statistical analyses were done before breaking the randomisation code, using the program …”
Rating: YES (Low risk of bias)
Domain V: Selective outcome reporting
Study protocol was published in 1991 prior to enrollment of participants, all primary outcome variables were reported with respect to all participants and the endpoints.
Rating: NO (high risk of bias)
No prior publication of protocol, but a pilot study exists. However this was published in 1993 only after the trial was performed in 1991. Primary outcome defined (duration of diarrhea), reported but table and graph do not match, secondary outcome (number of unformed stools on day 3) seems defined post hoc, for this is the only one point in time, this outcome yielded a significant result.
Rating: YES (low risk of bias)
Domain VI: Other sources of bias:
Rating: NO (high risk of bias), no details given
Imbalance of group properties (size, weight and age of children), that might have some impact on course of disease, high impact of parallel therapy (rehydration) by far exceeding effect size of homeopathic treatment
Rating: YES (low risk of bias), no details given
In a nutshell: I fail to see the basis for the different ratings in the studies themselves. I assume bias of the authors of the review.
So, what about the question posed in the title of this article? The meta-analysis is clearly not a ‘proof of concept’. But is it proof for misconduct? I asked Mathie and he answered as follows: No, your statement does not reflect the situation at all. As for each and every paper, we selected the main outcome measure for your trial using the objective WHO classification approach (in which quality of life is clearly of lower rank than severity). This is all clearly described in our prospective protocol. Under no circumstances did we approach this matter retrospectively, in the way you are implying.
Some nasty sceptics might have assumed that the handful of rigorous studies with negative results were well-known to most researchers of homeopathy. In this situation, it would have been hugely tempting to write the protocol such that these studies must be excluded. I am thrilled to be told that the authors of the current new meta-analysis (who declared all sorts of vested interests at the end of the article) resisted this temptation.