MD, PhD, FMedSci, FSB, FRCP, FRCPEd

Most of the underlying assumptions of alternative medicine (AM) lack plausibility. Whenever this is the case, so the argument put forward by an international team of researchers in a recent paper, there are difficulties involved in obtaining a valid statistical significance in clinical studies.

Using a mostly statistical approach, they argue that, since the prior probability of a research hypothesis is directly related to its scientific plausibility, the commonly used frequentist statistics, which do not account for this probability, are unsuitable for studies exploring matters in various degree disconnected from science. Any statistical significance obtained in this field should be considered with great caution and may be better applied to more plausible hypotheses (like placebo effect) than the specific efficacy of the intervention.

The researchers conclude that, since achieving meaningful statistical significance is an essential step in the validation of medical interventions, AM practices, producing only outcomes inherently resistant to statistical validation, appear not to belong to modern evidence-based medicine.

To emphasize their arguments, the researchers make the following additional points:

  • It is often forgotten that frequentist statistics, commonly used in clinical trials, provides only indirect evidence in support of the hypothesis examined.
  • The p-value inherently tends to exaggerate the support for the hypothesis tested, especially if the scientific plausibility of the hypothesis is low.
  • When the rationale for a clinical intervention is disconnected from the basic principles of science, as in case of complementary alternative medicines, any positive result obtained in clinical studies is more reasonably ascribable to hypotheses (generally to placebo effect) other than the hypothesis on trial, which commonly is the specific efficacy of the intervention.
  • Since meaningful statistical significance as a rule is an essential step to validation of a medical intervention, complementary alternative medicine cannot be considered evidence-based.

Further explanations can be found in the discussion of the article where the authors argue that the quality of the hypothesis tested should be consistent with sound logic and science and therefore have a reasonable prior probability of being correct. As a rule of thumb, assuming a “neutral” attitude towards the null hypothesis (odds = 1:1), a p-value of 0.01 or, better, 0.001 should suffice to give a satisfactory posterior probability of 0.035 and 0.005 respectively.

In the area of AM, hypotheses often are entirely inconsistent with logic and frequently fly in the face of science. Four examples can demonstrate this instantly and sufficiently, I think:

  • Homeopathic remedies which contain not a single ‘active’ molecule are not likely to generate biological effects.
  • Healing ‘energy’ of Reiki masters has no basis in science.
  • Meridians of acupuncture are pure imagination.
  • Chiropractic subluxation have never been shown to exist.

Positive results from clinical trials of implausible forms of AM are thus either due to chance, bias or must be attributed to more credible causes such as the placebo effect. Since the achievement of meaningful statistical significance is an essential step in the validation of medical interventions, unless some authentic scientific support to AM is provided, one has to conclude that AM cannot be considered as evidence-based.

Such arguments are by no means new; they have been voiced over and over again. Essentially, they amount to the old adage: IF YOU CLAIM THAT YOU HAVE A CAT IN YOUR GARDEN, A SIMPLE PICTURE MAY SUFFICE. IF YOU CLAIM THERE IS A UNICORN IN YOUR GARDEN, YOU NEED SOMETHING MORE CONVINCING. An extraordinary claim requires an extraordinary proof! Put into the context of the current discussion about AM, this means that the usual level of clinical evidence is likely to be very misleading as long as it totally neglects the biological plausibility of the prior hypothesis.

Proponents of AM do not like to hear such arguments. They usually insist on what we might call a ‘level playing field’ and fail to see why their assumptions require not only a higher level of evidence but also a reasonable scientific hypothesis. They forget that the playing field is not even to start with; to understand the situation better, they should read this excellent article. Perhaps its elegant statistical approach will convince them – but I would not hold my breath.

7 Responses to The statistical argument against alternative medicine: why it “cannot be considered evidence-based”

  • If anyone wants to know more about the statistical arguments, I recently wrote a paper on the topic, available at http://arxiv.org/abs/1407.5296
    Or, for a simpler version, see http://www.dcscience.net/?p=6518

    • Thank you very much for writing this paper and for sharing it with us, David.

      You have very clearly illustrated what I’ve always (ad hoc) called “the engineering method”, which is a strict, and usually much more valuable, version of the generalized scientific method commonly used in data acquisition and analysis. In some branches of engineering, even one anomalous datum amongst trillions of data samples warrants investigation to determine if the anomaly has revealed a fundamental error in the design or implementation of the system.

      If sCAM trials were mandated to use, as a minimum, the 3-sigma rule mentioned in your paper then the currently flourishing tree of sCAM quackery would rapidly wither away.

  • I made similar arguments some time ago that no clinical trial could ever convince a rational person of the efficacy of homeopathy.

    Should Cochrane Call for More Research Into Homeopathy?


    Firstly, and most importantly, to all intents and purposes, clinical trials of highly implausible treatments, such as homeopathy, can never be used as evidence of their efficacy. No matter how good the statistical result of a trial, or how much data is analysed in a meta-analysis, the probability will always be greater that we are just analysing flawed data rather than there being a real effect. Homeopaths complain that sceptics never accept that trial data is proof of the effectiveness of homeopathy. This approach shows that homeopaths are quite right in their fears, although sceptics ought to be careful to point out that it is not because there is no evidence, but rather than the available evidence falls far short of any meaningful threshold of acceptance. Without a degree of plausibility, homeopaths are asking scientists to believe in the daily occurrence of miracles, and that will not do.

    Secondly, homeopaths often accuse sceptics of double standards where low standards of evidence appear to exist for many routine hospital procedures whereas strong evidence is demanded for homeopathy. We can now see that this is not hypocrisy, but an inevitable consequence of scientific thinking. It is perfectly rational to accept treatments as effective if they have very high plausibility but little in the way of good objective evidence.

  • The only acceptable use of clinical evidence, as opposed to a double blind trial or full investigation, is as an indicator that perhaps a trial might be worth conducting. It gives no evidence to encourage the wider use of drugs or procedures.

  • I just have a quick question, why do doctors and the medical community seem to think that we should always base science on actual facts when religion and other spiritual concepts along with meditation and other mind techniques, even acupuncture and other alternative medicine methods are so widely commonly used that this Placebo concept that everyone talks about is only used when science wants to take a look at how a new drug is going to affect the populace? If we as a civilization on this planet can help our bodies heal itself isn’t that proof in and of itself weather its called a miracle or hand of God aspect. The mind is a powerful thing and using it would benefit anyone who tries which I think is more then enough to encourage the fact. Modern medicine from what I have researched and seen with my own eyes is only out to control the populace and make a profit. If people want to have an Alternative Method to the system already in place who has the right to take that away and only give what’s given in the medical community. Why not cater to those who practice Alternative medicine not only will it be profitable if you have it in the right areas but you also have a happier populace.

    • Please, formulate an understandable question. There is no connection between the start and the end of your question.

    • @ Amanda on Sunday 24 July 2016 at 21:10

      “I just have a quick question,”

      Always be wary of someone who says they have a quick question; invariably it isn’t, just as with this example.

      “why do doctors and the medical community seem to think that we should always base science on actual facts”

      Is there another sort of science not based on facts? Is that the one where bridges fall down, buildings collapse, cranes topple, TVs explode, cars burst into flames, traffic lights cause collisions, and ships sink, among many others? Do you want “science” in some ares of your life but not medicine?

      ” when religion and other spiritual concepts along with meditation and other mind techniques, even acupuncture and other alternative medicine methods are so widely commonly used that this Placebo concept that everyone talks about is only used when science wants to take a look at how a new drug is going to affect the populace? If we as a civilization on this planet can help our bodies heal itself isn’t that proof in and of itself weather its called a miracle or hand of God aspect. The mind is a powerful thing and using it would benefit anyone who tries which I think is more then enough to encourage the fact. Modern medicine from what I have researched and seen with my own eyes is only out to control the populace and make a profit. If people want to have an Alternative Method to the system already in place who has the right to take that away and only give what’s given in the medical community. Why not cater to those who practice Alternative medicine not only will it be profitable if you have it in the right areas but you also have a happier populace.”

      Sorry, I can’t work out what the rest of this incoherent gibberish is intended to mean.

Leave a Reply

Your email address will not be published. Required fields are marked *

Please answer the following: *

Recent Comments

Note that comments can be edited for up to five minutes after they are first submitted.


Click here for a comprehensive list of recent comments.

Categories