A “null field” is a scientific field where there is nothing to discover and where observed associations are thus expected to simply reflect the magnitude of bias.
This analysis aimed to characterize a null field using a known example, homeopathy (a pseudoscientific medical approach based on using highly diluted substances), as a prototype. The researchers identified 50 randomized placebo-controlled trials of homeopathy interventions from highly cited meta-analyses. The primary outcome variable was the observed effect size in the studies. Variables related to study quality or impact were also extracted.
The mean effect size for homeopathy was 0.36 standard deviations (Hedges’ g; 95% confidence interval: 0.21, 0.51) better than placebo, which corresponds to an odds ratio of 1.94 (95% CI: 1.69, 2.23) in favor of homeopathy. 80% of studies had positive effect sizes (favoring homeopathy). The effect size was significantly correlated with citation counts from journals in the directory of open-access journals and CiteWatch. We identified common statistical errors in 25 studies.
The authors concluded that a null field like homeopathy can exhibit large effect sizes, high rates of favorable results, and high citation impact in the published scientific literature. Null fields may represent a useful negative control for the scientific process.
The paper is perhaps not the easiest to comprehend but once you got the idea, you will agree with me that it is BRILLIANT. I warmly recommend it to all fans of homeopathy – in fact, if I could I’d offer it to King Charles as a present for the coronation.
Its authors are among the most prominent medical epidemiologist of our time with affiliations that speak for themselves:
- Department of Epidemiology and Population Health, Stanford University, Stanford, CA, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA.
- 2Department of Epidemiology and Population Health, Stanford University, Stanford, CA, USA.
- 3Department of Epidemiology and Population Health, Stanford University, Stanford, CA, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA; Department of Medicine, Stanford University, Stanford, CA, USA; Department of Biomedical Data Science, Stanford University, Stanford, CA, USA; Department of Statistics, Stanford University, Stanford, CA, USA.
It is, of course, a pity that the article is behind a paywall – but fortunately, the senior author, John Ioannidis, published his email address together with the abstract: [email protected]. So, if you have trouble understanding the point of the analysis, I suggest you ask for a reprint to get your head around it. I promise it’s worth it.
14 Responses to Homeopathy can offer empirical insights on treatment effects in a null field
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
So after well over 200 years, scientists finally found a valid application for homeopathy.
I can only imagine how happy homeopaths must be to have their much-ridiculed system of belief put to good use after all. I can already see the big smile on one Mr Ullman’s face!
yes, he’ll be so happy; after such a long wait he deserves it!
Unfortunately Ioannidis went full-crank during the pandemic and his once-considerable reputation has slumped.
I know – but many of his papers – including the one here – are excellent.
Dr Healy has some good quotes in his book Pharmageddon, on the magnitude of the bias in conventional medical research:
“And companies can ensure their drug never causes anything by making sure that trials are organized so that important adverse events cannot become statistically significant.”
There can be a few better symbols of Pharmageddon then prescription only drugs becoming among the most consumed drugs in pregnancy in the face of strengthening warnings that they cause birth defects. The answer to how this could happen lie in great part in how the pharmaceutical companies have managed to capitalize on the way the very protections put in place by Senator Kefauver in his 1962 Bill and in the reforms that defeated him. Prescription only status has made doctors the targets of a marketing exercise that is far more sophisticated than placing even billions of pages of advertisements in medical journals and bribing doctors to use drugs. As outlined in chapter 1, the patent status of drugs has given companies an incentive to chase blockbuster profits-doing so regardless of patient welfare. Controlled trials have given the companies a means to persuade doctors that snake oil works so well that withholding it in pregnancy would be unethical, and also means to make problems consequent on treatments vanish. But all of these things hinge on the fact that these drugs are available by prescription only.”
And what relation does your comment have to the article discussed in my post?
Your very first sentence is referring to identifying the magnitude of the bias in medical research. The post is about how we can supposedly compare all research to bogus results, supposedly, in homeopathy research.
The entire field of medical research has been corrupted by money on the conventional side, and by what I am not sure on the alternative side – money (not much), fame (none), seeking acceptance by mainstream medicine (not going to happen in a million years).
So we just have to be clear about the magnitude of the bias in medical research wherever we look.
and I thought that bias needs to be proven on a case-by-case basis- unless, of course, one is a bit paranoid.
@Edzard
His comments are great null field examples?
This is something I have proposed on several occasions over the years: that the one good purpose homeopathy could serve is as a perfect model to explore the way in which an utterly useless intervention might present itself in the scientific literature and in wider society. It doesn’t work, it cannot work, so how do biases, errors, incompetence and corruption play their various roles in intermittently giving it the appearance of efficacy?
I’d never have had the technical statistical skills to tackle the project myself.
It is a BRILLIANT idea in that your postulate (homeopathy has no effect) can then be used to disprove double-blind placebo-controlled evidence to the contrary that you disagree with, LOL. And I thought pseudoscientists were the only ones that had embraced circular reasoning! 😉
is it a ‘postulate’ that remedies bar of active ingredient have no effect?
This is a very curious article. Ioannidis being skeptical of homeopathy showed that 80% of homeopathy trials are positive and that even removing the 25 with “common” statistical errors, on average they have a similar effect size to many conventional drugs. But, stupidly, the only thing that deduced is that the effect is “product of some artifact” because “it can’t work and is pseudomedicine”.
I find it funny that Edzard tried to validate the article because Ioannidis is an eminence in epidemiology, but when Luc Montagnier (greater eminence than Ioannidis and his group combined) validated homeopathy, it turns out that he had become a “crank”.
It is enough for me to look at the references of the “great Ioannidis”:
– “Grimes D.R. Proposed mechanisms for homeopathy are physically impossible. Focus Alternate Complement Therapies. 2012; 17: 149-15”. This review, which in the article says that it was reviewed by Ernst himself, has basic errors such as saying that Benveniste’s study was from “1987” when it was from 1988 or that Ennis’s studies were “negative” when the conclusion says the opposite. Or the funniest part about Grimes trying to eliminate homeopathy with a few simple high school calculations exaggerating everything and not taking into account the part of sequential serial dilution, because of course, instead of sticking to reality, skeptics come up with the idea that homeopathy is the same as “diluting a drop in the ocean”.
-“Cukaci C. Freissmuth M. Mann C. Marti J. Sperl V. Against all odds-the persistent popularity of homeopathy.”This article is much funnier than the previous one, the authors are really sad that homeopathy has not lost popularity (even if Ernst says otherwise and deceives himself), and their “arguments” are that an editorial published in 2005 would have magically refuted a reanalysis published in 2018! Or that a replication of Louis Rey’s study would have been “refuted by the VanWijk study” when in fact this study actually replicates the results only when the time factor is taken into account. Does the Wien Klin Wochenschr not have the minimum rigor to have detected these and other basic errors?
-“Reisman S. Balboul M. Jones T. P-curve accurately rejects evidence for homeopathic ultramolecular dilutions.
PeerJ. 2019; 7: e6318”. Oh, look, an analysis based on a questionable tool and a dataset in which the authors get confused. After seeing the tests they used, it is very sad to see that the authors are quite incompetent and discarded the Frei et al test because they believed that “it was not based on ultramolecular doses”, how is it that they could not detect that an LM potency falls into the range of the ultramolecular? And I better not follow him when they put in a Walach proving as “clinical trial of efficacy” or the lack of a strategy of how they choose the trials.
– “Shang A. Huwiler-Muntener K. Nartey L. Juni P. Dorig S.
Sterne J.A. et al. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy.
Lancet. 2005; 366: 726-732”. Are you telling me that the “great Ionannis” was so incompetent as not to quote the refutation made by Lüdtke and Rutten ironically published in the same Journal of Clinical Epidemiology?
-“https://www.en.wikipedia.org/wiki/Wikipedia:WikiProject_Academic_Journals/Journals_cited_by_Wikipedia/Questionable1 “How bad was the review that the link doesn’t even work? Well, at least there’s a backup: https://www.en.wikipedia.org/wiki/Wikipedia:WikiProject_Academic_Journals/Journals_cited_by_Wikipedia/Questionable1 The only thing that is there is “Homeopathy [Unreliable fields]”, there is no method that validates that assumption. Are Wikipedia editors experts and can they qualify as reliable what experts from Thieme, Nature and ElSevier have been doing? Are you telling me that the great Ioannidis went to Wikipedia to believe?
– “Elton D.C. Spencer P.D. Pathological water science – four examples and what they have in common. in: Gadomski A. Water in Biomechanical and Related Systems. Springer International Publishing, Cham2021: 155-169”. This article is really funny, literally Elton doesn’t deny that EZ water exists, he just says that he disagrees with Pollack’s interpretation. Elton was only able to cite Ernst’s 2002 review (refuted by Robert Hahn) and Maddox’s 1988 editorial (refuted by Ennis). The other reference is a chapter of
Yuvan and Bier that I have already read, and that only summarizes what a 2018 article “Limits on Quantum Coherent Domains in Liquid Water”. This article could only criticize a first article by Del Giudice, and nothing more, because the authors at the end say that they did not bother to check the updates. Bier does not seem to be aware that Del Giudice himself said in other works that indeed his 1988 article had serious limitations but that they were corrected in later works. Are you telling me that Bier was unable to read those works before and only focused on one article to make a cartoon?
It would be interesting to see why the “great Ioannidis” avoided quoting the reviews of Alexander Tournier and Louis Demangeat or Luc Montagnier. Or maybe because neither Ernst nor his henchmen are eminent in physics or engineering
Magical thinking enters the scene whenever our most sacred beliefs are confronted by reality. For example: if we inform a homeopathy enthusiast that from the 13CH dilution there is no longer a trace of the solute, he will reply by saying that, from there, the “immaterial healing energy” (vitalist argument) of the homeopathic substance is manifested through the “memory of water” (pseudoscientific argument). And if we clarify that Canadian and German scientists demonstrated that the memory of water exists (Cowan et al., 2005), but that it only lasts 50 femtoseconds, it will tell us that the aforementioned research must be false (denialism) or that it does not trust science or scientists (antiscience) or that the study, in addition to being erroneous, it was funded by the pharmaceutical industry to harm homeopathy (conspiracy theory). If we finally explain that Oscillococcinum® – a homeopathic compound 200CH – is prepared with the heart and liver of a poor duck, and that a single duck is enough to produce all the medicine sold in the world for a year at a value of 20 million dollars (Barrett, 2016), our believer, with all certainty, will leave furious claiming that we are misinformed. “When (believers) are faced with facts, they leave aside the evidence, but never the theory: that is the philosophy of the protagonists of this game” (Randi, 2016; p. 207).