Most of the underlying assumptions of alternative medicine (AM) lack plausibility. Whenever this is the case, so the argument put forward by an international team of researchers in a recent paper, there are difficulties involved in obtaining a valid statistical significance in clinical studies.
Using a mostly statistical approach, they argue that, since the prior probability of a research hypothesis is directly related to its scientific plausibility, the commonly used frequentist statistics, which do not account for this probability, are unsuitable for studies exploring matters in various degree disconnected from science. Any statistical significance obtained in this field should be considered with great caution and may be better applied to more plausible hypotheses (like placebo effect) than the specific efficacy of the intervention.
The researchers conclude that, since achieving meaningful statistical significance is an essential step in the validation of medical interventions, AM practices, producing only outcomes inherently resistant to statistical validation, appear not to belong to modern evidence-based medicine.
To emphasize their arguments, the researchers make the following additional points:
- It is often forgotten that frequentist statistics, commonly used in clinical trials, provides only indirect evidence in support of the hypothesis examined.
- The p-value inherently tends to exaggerate the support for the hypothesis tested, especially if the scientific plausibility of the hypothesis is low.
- When the rationale for a clinical intervention is disconnected from the basic principles of science, as in case of complementary alternative medicines, any positive result obtained in clinical studies is more reasonably ascribable to hypotheses (generally to placebo effect) other than the hypothesis on trial, which commonly is the specific efficacy of the intervention.
- Since meaningful statistical significance as a rule is an essential step to validation of a medical intervention, complementary alternative medicine cannot be considered evidence-based.
Further explanations can be found in the discussion of the article where the authors argue that the quality of the hypothesis tested should be consistent with sound logic and science and therefore have a reasonable prior probability of being correct. As a rule of thumb, assuming a “neutral” attitude towards the null hypothesis (odds = 1:1), a p-value of 0.01 or, better, 0.001 should suffice to give a satisfactory posterior probability of 0.035 and 0.005 respectively.
In the area of AM, hypotheses often are entirely inconsistent with logic and frequently fly in the face of science. Four examples can demonstrate this instantly and sufficiently, I think:
- Homeopathic remedies which contain not a single ‘active’ molecule are not likely to generate biological effects.
- Healing ‘energy’ of Reiki masters has no basis in science.
- Meridians of acupuncture are pure imagination.
- Chiropractic subluxation have never been shown to exist.
Positive results from clinical trials of implausible forms of AM are thus either due to chance, bias or must be attributed to more credible causes such as the placebo effect. Since the achievement of meaningful statistical significance is an essential step in the validation of medical interventions, unless some authentic scientific support to AM is provided, one has to conclude that AM cannot be considered as evidence-based.
Such arguments are by no means new; they have been voiced over and over again. Essentially, they amount to the old adage: IF YOU CLAIM THAT YOU HAVE A CAT IN YOUR GARDEN, A SIMPLE PICTURE MAY SUFFICE. IF YOU CLAIM THERE IS A UNICORN IN YOUR GARDEN, YOU NEED SOMETHING MORE CONVINCING. An extraordinary claim requires an extraordinary proof! Put into the context of the current discussion about AM, this means that the usual level of clinical evidence is likely to be very misleading as long as it totally neglects the biological plausibility of the prior hypothesis.
Proponents of AM do not like to hear such arguments. They usually insist on what we might call a ‘level playing field’ and fail to see why their assumptions require not only a higher level of evidence but also a reasonable scientific hypothesis. They forget that the playing field is not even to start with; to understand the situation better, they should read this excellent article. Perhaps its elegant statistical approach will convince them – but I would not hold my breath.
17 Responses to The statistical argument against alternative medicine: why it “cannot be considered evidence-based”
Leave a Reply to Pete 628 Cancel reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
If anyone wants to know more about the statistical arguments, I recently wrote a paper on the topic, available at http://arxiv.org/abs/1407.5296
Or, for a simpler version, see http://www.dcscience.net/?p=6518
Thank you very much for writing this paper and for sharing it with us, David.
You have very clearly illustrated what I’ve always (ad hoc) called “the engineering method”, which is a strict, and usually much more valuable, version of the generalized scientific method commonly used in data acquisition and analysis. In some branches of engineering, even one anomalous datum amongst trillions of data samples warrants investigation to determine if the anomaly has revealed a fundamental error in the design or implementation of the system.
If sCAM trials were mandated to use, as a minimum, the 3-sigma rule mentioned in your paper then the currently flourishing tree of sCAM quackery would rapidly wither away.
I made similar arguments some time ago that no clinical trial could ever convince a rational person of the efficacy of homeopathy.
Should Cochrane Call for More Research Into Homeopathy?
The only acceptable use of clinical evidence, as opposed to a double blind trial or full investigation, is as an indicator that perhaps a trial might be worth conducting. It gives no evidence to encourage the wider use of drugs or procedures.
I just have a quick question, why do doctors and the medical community seem to think that we should always base science on actual facts when religion and other spiritual concepts along with meditation and other mind techniques, even acupuncture and other alternative medicine methods are so widely commonly used that this Placebo concept that everyone talks about is only used when science wants to take a look at how a new drug is going to affect the populace? If we as a civilization on this planet can help our bodies heal itself isn’t that proof in and of itself weather its called a miracle or hand of God aspect. The mind is a powerful thing and using it would benefit anyone who tries which I think is more then enough to encourage the fact. Modern medicine from what I have researched and seen with my own eyes is only out to control the populace and make a profit. If people want to have an Alternative Method to the system already in place who has the right to take that away and only give what’s given in the medical community. Why not cater to those who practice Alternative medicine not only will it be profitable if you have it in the right areas but you also have a happier populace.
Please, formulate an understandable question. There is no connection between the start and the end of your question.
@ Amanda on Sunday 24 July 2016 at 21:10
“I just have a quick question,”
Always be wary of someone who says they have a quick question; invariably it isn’t, just as with this example.
“why do doctors and the medical community seem to think that we should always base science on actual facts”
Is there another sort of science not based on facts? Is that the one where bridges fall down, buildings collapse, cranes topple, TVs explode, cars burst into flames, traffic lights cause collisions, and ships sink, among many others? Do you want “science” in some ares of your life but not medicine?
” when religion and other spiritual concepts along with meditation and other mind techniques, even acupuncture and other alternative medicine methods are so widely commonly used that this Placebo concept that everyone talks about is only used when science wants to take a look at how a new drug is going to affect the populace? If we as a civilization on this planet can help our bodies heal itself isn’t that proof in and of itself weather its called a miracle or hand of God aspect. The mind is a powerful thing and using it would benefit anyone who tries which I think is more then enough to encourage the fact. Modern medicine from what I have researched and seen with my own eyes is only out to control the populace and make a profit. If people want to have an Alternative Method to the system already in place who has the right to take that away and only give what’s given in the medical community. Why not cater to those who practice Alternative medicine not only will it be profitable if you have it in the right areas but you also have a happier populace.”
Sorry, I can’t work out what the rest of this incoherent gibberish is intended to mean.
I’m reawakening an old post. Yay!
I figured on commenting on this subject since this was the first search result on google for arguments against alternative medicine.
So, I would like to add, how much do you know of basic biology and chemistry to make these assumptions? First of all, alternative medicine and nutrition go rather nicely together. These three sciences are good explanations for why many forms of alternative medicine do work.
Have you heard of Nutrigenomics?
Here’s a link to “summarize” the topic: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2137135/
Much of alternative, at least the herbal-based and food-based forms, are rooted in Nutrigenomics. Chinese, Japanese, and Korean philosophy have a much more concrete understanding than most western-based alternative medicine forms that have been diluted, ie. lost and reinterpreted, since the Dark Ages. So if you’re basing this idea of Alternative Medicine on western-based research, you’re bound to come up short.
In relation to other alternative medicinal forms, including acupuncture and yoga, those are based on a mixture of mental health wellness and regulating of the blood cyclical trials. Both of which are understood in the Medical field in the form mental health(such as the use of a therapist or psychiatrist) and blood clots, which to this day science has no real evidence to why some are caused only that they can be the result of heart problems or blood disorders but this is not always the case.
The issue I have with your argument is Western medicine is by virtue of existence an entirely different form of medical practice. Most alternative medicine deals with the symptoms and prevention. The extreme cases are few and when they do occur the alternative medicine is simply used to make the individual comfortable.
The interesting thing is what does western medicine do differently in that regard? To make the individual comfortable? Actually, I’m not quite sure it even does that every time or even consistently. Pills for the illness but cause two other illnesses in their wake. Antibiotics that cause infections. Chemotherapy that still kills the subject. Western medicine focuses of waiting until the bacteria or the virus has completely taken over before reacting. And in some cases, only delaying the inevitable. Perhaps your idea of medicine and these alternative forms are more alike than you think.
Personally, I think it depends on the individual which is better but to say there is no science based in alternative medicine because the only language you can read is English is only sad. There is clearly more of the world you need to research and learn about. No matter how old you are. There is only more and more information being found. Tomorrow, everything you think is correct could be wrong.
But I guess that’s just science 🙂
are you for real???
Both Bohr, Aristotle, and Hippocrates are considered the founders of Chemistry and Medicine, but their information and flat-out incorrect models they imagined to be real were used for centuries as the basis of understanding in how we function as humans inside and out. For you to honestly believe science has always stayed stable and consistent is easily proven to be just as incorrect.
But, that’s not something I think this discussion should veer into.
I figured this site, for all its legitimacy, would simply explain to me the relationship between Nutrigenomics and how(without bias, might I add, because data is the difference between science and any or other belief systems)that may affect our understanding of alternative medicine.
I don’t really care about wandering conversations that have nothing to do with that question. Thanks. 🙂
Quoting Wikipedia:
The trouble with alternative medicine is that it usually lays claims on highly implausible mechanisms, at times highjacks seemingly relevant, but ultimately irrelevant, mechanisms, and boldly and persistently, but all-too-often stagnantly, stands on correlation, without providing any high-quality evidence of causation. Establishing a causal link with the help of robust evidence usually upgrades alternative medicine to plain medicine.
The first thing you will necessarily have to forfeit is wishful thinking, which is the hardest part. From then on, things are straightforward. If you get increasingly confident evidence for a specific effect of a nutrient on some locus and this is accompanied by a clinically relevant manifestation (i.e. not a “silent mutation” type of thing etc.), you can check for clinical effects of whatever contains this nutrient. You might get an apparent association there. By piling evidence, a causal link may unfold and, with time, you may be approaching the next generation of blood type diets, for example. But you need mechanisms, so you must dive to the molecular level. And because these are bold claims to begin with, you must pile extraordinary and robust evidence.
Because alternative medicine researchers are not usually equipped with a strong line of defense against wishful thinking (though there are bright exceptions, such as the host of this blog you’re reading right now, and which is the primary reason you are in the right place), to answer your question about how nutrigenomics will affect the understanding of alternative medicine, I believe it will primarily provide far more ground for spurious and flimsy conclusions, a new era of highly implausible claims and profound amounts of misrepresentation to promote whatever is the next generation of dietary supplements…among other “unfortunacies”. On the other hand, a small number of herbal preparations may be found to have specific effects, or, even better, some subtle dietary details may help organize and tailor a diet for diabetes patients of varying genotypic manifestations, or for sensitive dietary cholesterol responders.
Try to be critical with whatever you view as evidence and don’t jump to conclusions. And try to spare the majestic sentimental appeals, science is about reason. If you are using your question to lay out your type of thinking, then it is not an honest question, and you are probably already equipped with your desired answer. If you dropped by to reinforce some belief of yours, you are in the wrong place.
Hello, I appreciate the effort in the explanation. I think the issue with this mainline of conversation is bias. If each experiment is meant to be proven, disprove, and/or support an already well-known theory then isn’t the following act of the process to take in new information as it applies as long as evidence proves supportive. Most of these clinical trials listed are, of course, unlikely to be proven true but that isn’t what I’m asking in relation to Nutrigenomics.
I’ll try to clarify because I don’t want to derail the conversation. Nutrigenomics is based on the idea that certain genes are constantly changing and are changed by the ingestion of foods that produce amino acids and other nutrients our body can use to do its work.
The question is: Does Nutrigenomics mean that herbal-based medicine is a possibility? Is it possible that we simply haven’t found the right mixtures or the right people to react to the medicine effectively? I only question this because we know how telomeres work in cellular aging and we know that the sudden appearances or losses of allergies are the byproducts of the natural changes our biochemistry goes through as we age.
I’m using theories and basic biochemistry as a reference so I don’t find this anything but critical. If you would like to be critical of this idea based on factual evidence or supporting evidence that contradicts the existence of Nutrigenomics then by all means do so. I’m interested.
@Yasmin
Your obvious enthusiasm for ‘nutrigenomics’ and its relevance just might be exceeding the real expectations. IOW, I get the impression you’re putting the cart a very long way ahead of the horse. (And the horse itself is probably just a foal at this stage.)
From the article you linked to in your first post (by the way, it’s a weak, unscholarly review that doesn’t even cite references properly) the bulk of the nutrigenomic field so far consists of measuring genomic changes that result from exposure of cells in vitro to ‘nutrients’ — predominantly vitamins. But in which of the ~200 cell types in the human body are the changes investigated? I guess we’re looking at tissue cultures.
Your ‘perspective’ article mentions experiments in mice, but provides no reference to a peer-reviewed article containing details. (You can stand up and say anything you like at a conference.) All this is a very long way from any medical treatment, yet you’re asking about the relevance of this work to medicine. Short answer, it’s got a helluva way to go to acquire any meaningful relevance.
The other face of nutrigenomics is apparently concerned with associations between gene sequence changes (SNPs) and diet. This is a branch of epidemiology (your linked article says “…integration of genomic science with nutrition and, when possible, other lifestyle variables such as cigarette smoking and alcohol consumption”) and you must make yourself aware of the huge minefield into which this pokes a scientific toe. Epidemiological work requires very long-term studies that take account of as many lifestyle variables as possible. And even then, they tend to fall into the classical trap of ‘correlation does not imply causation’. Take a look at this website, which is a bit of an eye-opener about correlations.
You ask “Does Nutrigenomics mean that herbal-based medicine is a possibility?” Herbal medicine is a reality! Many herbs have been found to have medicinal properties, and analysis has usually succeeded in identifying the molecule that causes the medicinal effect. The active molecule may then be modified to improve one or other of its properties, but it’s always better to treat diseases with a known molecule(s) than with a herb, where the specific type of herb, the part of the herb used and the method of preparation can all differ. The main reason herbal medicine (as currently practised by its fans) is deprecated on this blog is because so very few herbal remedies have been adequately tested to prove their efficacy in robustly scientific clinical trials.
Re-read the final para graph of James’s excellent response to you and try to understand and apply its points to your own thinking.
Maybe you’ll let us know (in outline) exactly what your PhD project entails, and you might attract some useful or at least interesting responses on this blog. But please try to suppress your eager enthusiasm for the grandiose, theoretical long-term implications of nutrigenomics for medical science and stay within the bounds of reason.
Yasmine, you are failing to communicate. You are talking to yourself. Is that all you wish to do?
Starting with an assumption and then adjusting the information to fit it, that is not how science works 😀
“Tomorrow everything you think is correct could be wrong”…
If by “you”, you mean YOU, then YOU are certainly correct.
“….Have a much more concrete understanding than most western-based alternative medicine forms that have been diluted i.e. lost and reinterpreted since the Dark ages”.
Yikes & congratulations….sentence after sentence of words and punctuation but utterly devoid of any meaning! Much like the entire history of TCM.
an important, relevant addition (http://rsos.royalsocietypublishing.org/content/4/12/171085):
We wish to answer this question: If you observe a ‘significant’ p-value after doing a single unbiased experiment, what is the probability that your result is a false positive? The weak evidence provided by p-values between 0.01 and 0.05 is explored by exact calculations of false positive risks. When you observe p = 0.05, the odds in favour of there being a real effect (given by the likelihood ratio) are about 3 : 1. This is far weaker evidence than the odds of 19 to 1 that might, wrongly, be inferred from the p-value. And if you want to limit the false positive risk to 5%, you would have to assume that you were 87% sure that there was a real effect before the experiment was done. If you observe p = 0.001 in a well-powered experiment, it gives a likelihood ratio of almost 100 : 1 odds on there being a real effect. That would usually be regarded as conclusive. But the false positive risk would still be 8% if the prior probability of a real effect were only 0.1. And, in this case, if you wanted to achieve a false positive risk of 5% you would need to observe p = 0.00045. It is recommended that the terms ‘significant’ and ‘non-significant’ should never be used. Rather, p-values should be supplemented by specifying the prior probability that would be needed to produce a specified (e.g. 5%) false positive risk. It may also be helpful to specify the minimum false positive risk associated with the observed p-value. Despite decades of warnings, many areas of science still insist on labelling a result of p < 0.05 as ‘statistically significant’. This practice must contribute to the lack of reproducibility in some areas of science. This is before you get to the many other well-known problems, like multiple comparisons, lack of randomization and p-hacking. Precise inductive inference is impossible and replication is the only way to be sure. Science is endangered by statistical misunderstanding, and by senior people who impose perverse incentives on scientists.