What is and what isn’t evidence, and why is the distinction important?

In the area of alternative medicine, we tend to engage in sheer endless discussions around the subject of evidence; the relatively few comments on this new blog already confirm this impression. Many practitioners claim that their very own clinical experience is at least as important and generalizable as scientific evidence. It is therefore relevant to analyse in a little more detail some of the issues related to evidence as they apply to the efficacy of alternative therapies.

To prevent the debate from instantly deteriorating into a dispute about the value of this or that specific treatment, I will abstain from mentioning any alternative therapy by name and urge all commentators to do the same. The discussion on this post should not be about the value of homeopathy or any other alternative treatment; it is about more fundamental issues which, in my view, often get confused in the usually heated arguments for or against a specific alternative treatment.

My aim here is to outline the issues more fully than would be possible in the comments section of this blog. Readers and commentators can subsequently be referred to this post whenever appropriate. My hope is that, in this way, we might avoid repeating the same arguments ad nauseam.

Clinical experience is notoriously unreliable

Clinicians often feel quite strongly that their daily experience holds important information about the efficacy of their interventions. In this assumption, alternative practitioners are usually entirely united with healthcare professionals working in conventional medicine.

When their patients get better, they assume this to be the result of their treatment, especially if the experience is repeated over and over again. As an ex-clinician, I do sympathise with this notion which might even prevent practitioners from losing faith in their own work. But is the assumption really correct?

The short answer is NO. Two events [the treatment and the improvement] that follow each other in time are not necessarily causally related; we all know that, of course. So, we ought to consider alternative explanations for a patient’s improvement after therapy.

Even the most superficial scan of the possibilities discloses several options: the natural history of the condition, regression towards the mean, the placebo-effect, concomitant treatments, social desirability to name but a few. These and other phenomena can contribute to or determine the clinical outcome such that inefficacious treatments appear to be efficacious.

What follows is simple, undeniable and plausible for scientists, yet intensely counter-intuitive for clinicians: the prescribed treatment is only one of many influences on the clinical outcome. Thus even the most impressive clinical experience of the perceived efficacy of a treatment can be totally misleading. In fact, experience might just reflect the fact that we repeat the same mistake over and over again. Put differently, the plural of anecdote is anecdotes, not evidence!

Clinicians tend to get quite miffed when anyone tries to explain to them how multifactorial the situation really is and how little their much-treasured experience tells us about therapeutic efficacy. Here are seven of the counter-arguments I hear most frequently:

1)      The improvement was so direct and prompt that it was obviously caused by my treatment [this notion is not very convincing; placebo-effects can be just as prompt and direct].

2)      I have seen it so many times that it cannot be a coincidence [some clinicians are very caring, charismatic, and empathetic; they will thus regularly generate powerful placebo-responses, even when using placebos].

3)      A study with several thousand patients shows that 75% of them improved with my treatment [such response rates are not uncommon, even for ineffective treatments, if patient-expectation was high].

4)      Surely chronic conditions don’t suddenly get better; my treatment therefore cannot be a placebo [this is incorrect, eventually many chronic conditions improve, if only temporarily].

5)      I had a patient with a serious condition, e.g. cancer, who received my treatment and was cured [if one investigates such cases, one often finds that the patient also took a conventional treatment; or, in rare instances, even cancer-patients show spontaneous remissions].

6)      I have tried the treatment myself and had a positive outcome [clinicians are not immune to the multifactorial nature of the perceived clinical response].

7)      Even children and animals respond very well to my treatment, surely they are not prone to placebo-effects [animals can be conditioned to respond; and then there is, of course, the natural history of the disease].

Is all this to say that clinical experience is useless? Clearly not! I am merely pointing out that, when it comes to therapeutic efficacy, clinical experience is no replacement for evidence. It is invaluable for a lot of other things, but it can at best provide a hint and never a proof of efficacy.

What then is reliable evidence?

As the clinical outcomes after treatments always have many determinants, we need a different approach for verifying therapeutic efficacy. Essentially, we need to know what would have happened, if our patients had not received the treatment in question.

The multifactorial nature of any clinical response requires controlling for all the factors that might determine the outcome other than the treatment per se. Ideally, we would need to create a situation or an experiment where two groups of patients are exposed to the full range of factors, and the only difference is that one group does receive the treatment, while the other one does not. And this is precisely the model of a controlled clinical trial.

Such studies are designed to minimise all possible sources of bias and confounding. By definition, they have a control group which means that we can, at the end of the treatment period, compare the effects of the treatment in question with those of another intervention, a placebo or no treatment at all.

Many different variations of the controlled trial exist so that the exact design can be adapted to the requirements of the particular treatment and the specific research question at hand. The over-riding principle is, however, always the same: we want to make sure that we can reliably determine whether or not the treatment was the cause of the clinical outcome.

Causality is the key in all of this; and here lies the crucial difference between clinical experience and scientific evidence. What clinician witness in their routine practice can have a myriad of causes; what scientists observe in a well-designed efficacy trial is, in all likelihood, caused by the treatment. The latter is evidence, while the former is not.

Don’t get me wrong; clinical trials are not perfect. They can have many flaws and have rightly been criticised for a myriad of inherent limitations. But it is important to realise that, despite all their short-commings, they are far superior than any other method for determining the efficacy of medical interventions.

There are lots of reasons why a trial can generate an incorrect, i.e. a false positive or a false negative result. We therefore should avoid relying on the findings of a single study. Independent replications are usually required before we can be reasonably sure.

Unfortunately, the findings of these replications do not always confirm the results of the previous study. Whenever we are faced with conflicting results, it is tempting to cherry-pick those studies which seem to confirm our prior belief – tempting but very wrong. In order to arrive at the most reliable conclusion about the efficacy of any treatment, we need to consider the totality of the reliable evidence. This goal is best achieved by conducting a systematic review.

In a systematic review, we assess the quality and quantity of the available evidence, try to synthesise the findings and arrive at an overall verdict about the efficacy of the treatment in question. Technically speaking, this process minimises selection and random biases. Systematic reviews and meta-analyses [these are systematic reviews that pool the data of individual studies] therefore constitute, according to a consensus of most experts, the best available evidence for or against the efficacy of any treatment.

Why is evidence important?

In a way, this question has already been answered: only with reliable evidence can we tell with any degree of certainty that it was the treatment per se – and not any of the other factors mentioned above – that caused the clinical outcome we observe in routine practice. Only if we have such evidence can we be sure about cause and effect. And only then can we make sure that patients receive the best possible treatments currently available.

There are, of course, those who say that causality does not matter all that much. What is important, they claim, is to help the patient, and if it was a placebo-effect that did the trick, who cares? However, I know of many reasons why this attitude is deeply misguided. To mention just one: we probably all might agree that the placebo-effect can benefit many patients, yet it would be a fallacy to assume that we need a placebo treatment to generate a placebo-response.

If a clinician administers an efficacious therapy [one that generates benefit beyond placebo] with compassion, time, empathy and understanding, she will generate a placebo-response PLUS a response to the therapy administered. In this case, the patient benefits twice. It follows that, merely administering a placebo is less than optimal; in fact it usually means cheating the patient of the effect of an efficacious therapy.

The frequently voiced counter-argument is that there are many patients who are ill without an exact diagnosis and who therefore cannot receive a specific treatment. This may be true, but even those patients’ symptoms can usually be alleviated with efficacious symptomatic therapy, and I fail to see how the administration of an ineffective treatment might be preferable to using an effective symptomatic therapy.


We all agree that helping the patient is the most important task of a clinician. This task is best achieved by maximising the non-specific effects [e.g. placebo], while also making sure that the patient benefits from the specific effects of what medicine has to offer. If that is our goal in clinical practice, we need reliable evidence and experience. Therefore one cannot be a substitute for the other, and scientific evidence is an essential precondition for good medicine.

35 Responses to What is (and what isn’t) clinical evidence, and why is the distinction important?

  • Are we assuming scientists and clinicians are capable of being more objective than the rest of us? When we earn a livelihood through certain method and culminate our identity from a particular belief, when someone attempts to take that away from us, with say, new evidence, we are NOT likely to embrace it, especially if it threatens our livelihood or identity. The more common response is a self-preserving visceral emotional reaction causing an outright rejection of any idea that is in conflict with our established view, thus unknowingly, we become the self-appointed guardian of the status quo. We see this behavior not just in medicine, but also religion, politics and every other area of our lives. You would hope scientists and clinicians would act and think with perfect logic uninfluenced by money and immune to personal bias, but all the “evidence” suggests they are just as fallible and vulnerable as the rest of us.

    Setting aside our biases, shouldn’t the burden of proof for a pharmaceutical drug be far higher than that of a natural remedy? My understanding is more people die each year from properly prescribed drugs than die in car accidents — in 2009 37,485 people died from prescription drugs, versus 36,284 from traffic accidents. If you intend to fly a jumbo jet caring 200 passengers you are going to be held to a much higher standard than a driver of a passenger car. How many people die each year coffee enemas (reference to Louise Lubetkin’s post yesterday), herbal tinctures, vitamin supplements, and other natural potions? When the traditional practitioner is wrong the consequence can be death or injury. When the alternative practitioner is wrong the consequence is constipation or diarrhea.

    • ” When the alternative practitioner is wrong the consequence is constipation or diarrhea.”

      Really? Have you not heard of Cameron Ayres, Gloria Thomas, Penelope Dingle to name but a few of those who died because an alternative practitioner was wrong?

      • Seriously? Are you attempting to make a moral equivalency argument?

        Yes, I have heard of those rare cases, but you either conveniently ignore or fail to grasp larger point. Dr. Ernst asked in his post, “What is (and what isn’t) clinical evidence, and why is the distinction important?” My point was, when weighing what is (and what isn’t) evidence, the bar must be higher for the traditional medical professional wielding a knife or toxic drug (Pharmacology, “poison” in classic Greek) than that of a alternative practitioner wielding a herbal remedy or natural tincture.

        Somehow lost in the efficacy and effectiveness argument over traditional vs. alternative approaches is the fact that (according to the Journal of American Medical Association) 39,000 people die each year due to unnecessary surgery and other errors in hospitals. 80,000 people die due to other infections in hospitals. 37,485 people died in 2009 from adverse drug reactions in USA. There were nearly 4.6 million drug-related visits to emergency rooms in the United States in 2009, with more than half due to adverse reactions to prescription medications – most of which were being taken exactly as prescribed.

        These deaths are not attributed vitamin supplements. It seems to me this is an important distinction worth mentioning.

        • David have you read the wkipedia article on vitamin poisoning?

          It is the dose that makes the poison as they say (well possibly not for lead, ionising radiation and a few other things for which no safe dose is known) , and some food supplements (e.g. selenium) have a relatively narrow range between beneficial and toxic.

          There are some wacky supplements out there which are simply unsafe and need tight regulation.

        • Sorry I forgot all about this conversation.

          “Seriously? Are you attempting to make a moral equivalency argument?”

          No, dear. I was merely pointing out the outrageous falsehood in your own.

        • Natural/Herbal does not necessarily mean safer than pharmaceutical drugs – you have heard of Deadly Nightshade I assume? If not the clue is in the name. Skepticat has cited some cases of death as a result of alternative therapy, also herbal/alternative remedies such as St Johns Wort can interfere with pharmacological medicine so of course they need to be regulated.

          I believe that it is important, as Dr Ernst points out, that we rely on scientific evidence to support claims of efficacy, not only as evidence of efficacy but also to investigate any possible side effects. This must be applied to pharmacological and alternative treatments, there should be no double standards – this also means that research into alternative treatments should not be discredited or denied merely because their mechanism is at present unknown, there are plenty of examples in medicine of treatments working and being used before a mechanism has been discovered such as aspirin and radiotherapy.

          • David, I feel for you….clearly the comments here show this ongoing pattern of denial…..while approved pharmaceutical drugs continue to kill and injure hundreds of thousands of people each year around the world, victims whom followed the usage and dosage instructions…forming only one part of the mass iatrogenic deaths attributed to modern western “medicine”……

            these sleepwalking, hypnotised people live in a world of denial where they continually want to point towards natural, often botanical therapies as being the bigger danger and concern.

            Ignore their delusion and perhaps one day they will break free from their denial, when they do basic research on Iatrogenic deaths….

      • OMG!!! 3 people that you can name killed supposedly by an alternative practitioner. Horrors!!

        Are you aware that conventional medical errors are now the third leading cause of death in the USA?
        That doesnt include all the other Standard Medical “Care” iatrogenic causes of death and injury.

  • David: are you seriously advocating double standareds in healthcare?

    • With respect, Dr. Ernst, are you seriously implying that we don’t already have a double standard in healthcare? What I’m advocating for is common sense.

      Here’s an example: The pharmaceutical industry and the AMA with the support of the FDA are seeking legislation requiring vitamin supplements to adhere to the same safety guidelines as prescription drugs. On the face of it this sounds reasonable, right? But what’s the point of passing legislation to protect the population from a product that causes no harm? In the last 7 years there is virtually no record of anyone dieing from taking vitamin supplements. Peel the onion and you can plainly see this is nothing but a canard. This is appears to be a direct attack by the FDA on the supplement industry and naturopathic doctors. The AMA and big pharma (which is the largest funding source to the FDA through user fees) are in bed with the FDA and they are exploiting this cozy relationship to help them eliminate their competition through legislation. The objective is to require a vitamin supplements to go through a lengthy FDA review to determine safety and efficacy, a process that costs millions of dollars and almost no vitamin supplement company could afford, thus putting them out of business. Further, the FDA would require a doctor’s prescription (just like in Germany) for an individual to receive a vitamin supplement. This legislation is seeping through the U.S. congressional sewer system as we speak. It seems perfectly reasonable (and common sense) to scrutinize a product that kills thousands every year. Does it make sense to apply the same level of scrutiny to a product that harms nobody? Common sense would suggest it makes no sense at all. Is that a double standard?

  • David: pity that your vitamins do not prevent paranoia!
    no, seriously: of course there are double standards everywhere – but our task has to be to fight them and abolish them, and not to perpetuate them.

  • Denial is a defense mechanism. Actually, I hear Kava, Valerian and Passionflower prevents denial. 🙂

    On a serious note, what am I missing here? I take it you do not believe there is a conflict of interest between the FDA and big pharma? Do you not believe there should be more stringent scrutiny and rigorous testing on products that present a far greater risk?

    There are different rules/laws for gun and knife ownership. They are both dangerous and both can kill, but gun ownership falls under much higher scrutiny (and illegal in some countries) because they pose a much greater risk to society. This is why we apply a different standard to the ownership of kitchen knifes. Is this a double standard that we should abolish?

  • What you are missing, I think, is that this post is about what constitutes evidence for the efficacy of a therapy. Once you make a claim like “my treatment works for condition XY” you need sound evidence and not wishful thinking.

  • “Are we assuming scientists and clinicians are capable of being more objective than the rest of us? ”
    We should not assume that. We should look at science as the way to objectively test claims.
    Scientists and others “use” or apply science as humans – warts an’ all.
    This does not make science bad.

    I must say it is pretty tiring to see the “but they are worse” argument rolled out. There is already a lot of science being applied to examining big pharma – studies exposing the weaknesses and mistakes. And more could be done.
    But that has NOTHING to do with the concept that the scientific method is the ONLY one to help provide us with more objective info about a method or theory.

  • In 1850, Worthington Hooker wrote a book called Lessons from the History of Medical Delusions.

    He wrote that the principle cause of medical delusion is the “too ready disposition to consider whatever follows a cause as being the result of that cause”.

    It would appear that 160 years later, there are far too many who still have failed to grasp this.

    • @ Andy Lewis

      Why do you think that people have a “too ready disposition to consider whatever follows a cause as being the result of that cause” and what can be done to change this?

  • @ Andy Lewis

    Why do you think people are “too ready disposition to consider whatever follows a cause as being the result of that cause”?

  • What can be done about it? I would have thought the answer is rather obvious: become a skeptic!

  • The current wild west lack of regulation of supplements is far from ideal. But if there is a change to be made to this regulatory structure you do have to account for the overall relative safety of supplements compared to pharmaceuticals and also the lack of patents and thus profit motive for natural supplements. Obviously pharmaceuticals are, in general, much more dangerous than supplements.

    Is NIH going to come up with $billions every year to study supplements? If not then noone will do the studies necessary to get supplements through the FDA drug approval process. Simply mandating that all supplements must meet the standards applied to pharmaceuticals doesn’t make sense at all.

    • @Justin Reilly
      “you do have to account for the overall relative safety of supplements compared to pharmaceuticals…”
      According to the FDA website, 280 warning letters have been sent to (USA) manufacturers of dietary supplements so far this year alone. (There are 28 pages of 20 letters at the link.) They mostly cover toxic content in supplements and misbranding. According to the American Cancer Society, the FDA reporting system logged 1,009 dietary supplement adverse events in 2010, 2,047 in 2011 and 2,844 in 2012. (I could not find these data directly on the FDA website, but I presume the ACS is a responsible body when it reports such numbers.) The ACS also states: “Most people who suffer unexpected side effects, illnesses, or drug interactions from dietary supplements don’t call a poison control center or the supplement manufacturer. This means that the numbers we have are likely very low estimates of actual events.”
      “…and also the lack of patents and thus profit motive for natural supplements.”
      “Producing about $32 billion in revenue for just nutritional supplements alone in 2012, [the industry] is projected to double that by topping $60 billion in 2021 according to the Nutritional Business Journal.” That’s a quote from Forbes Journal, the business magazine.
      ” Obviously pharmaceuticals are, in general, much more dangerous than supplements.”
      “… taking dietary supplements can be risky, especially for people who are getting cancer treatment.” Another quote from the ACS webpage linked above. I would suggest that what is obvious is that pharmaceuticals, in general, are capable of doing a lot of good by curing diseases that would never under any circumstances be affected by dietary supplements. What’s important in judging whether any kind of product you put in your mouth is the risk-benefit analysis (and, when a third party is paying, cost-effectiveness analysis). Your sweeping implication that nutritional supplements are inherently safe is incorrect. Your question whether the FDA will invest billions of dollars in research is incorrectly targeted: an industry worth $32 billion several years ago should be perfectly capable of doing its own research.
      The bottom line, which seems to defeat the intellect of far too many people, is that if you are going to market something that claims to benefit health, the onus is on you to produce robust evidence to demonstrate (quantitatively) its efficacy and its safety. A similar stricture applies to all manufactured goods, but in most countries regulations have been applied to ensure things that are sold work as claimed and don’t cause harm. Why healthcare seems to be the exception to this rule, where thousands of unproven remedies are allowed to be marketed without fundamental proof of efficacy and safety, is a mystery to me.


  • Thanks, I really enjoyed this article. Clinicians can certainly form strong opinions about their abilities based on their success in clinic.

  • The replies to this article, that was well stated and that posed a good question, are very disappointing; just as disappointing as Western medicine.

    People who are sick want to become well and we will seek anything that seems/hopes/promises to give a remedy. We have no other choice. Those who live by double-blind studies must also understand that people who are ill have no choice but to not live by this standard.

    Rather than demean alternative medicine and methods, Big Pharma and their adherents should look for ways to integrate both. Double-blind studies are fine but a patient could/would be made better were appropriate herbals also applied. That would satisfy those who want the patient to be the most important aspect to be focused on.

    Sadly, Big Pharma regards alternatives as a dagger to their dominance and their profits; therefore, herbals are not covered by medical insurance and Integrative Medicine is not studied in medical schools. It’s too bad that “shame” is not a factor in a balance sheet.

    There is no answer to the question posed by Dr. Ernst.
    1. Those who want the industry to pay for studies, as they should, must understand that the supplement manufacturer does not gain anything and can lose a great deal when/if the study shows that their supplement is not effective.
    2. The maker of supplements already has a huge and growing market – so why rock a boat and put profits in jeopardy? They think it is wiser to spend their monies on better marketing to gain market share.
    3. To have the government/NIH pay for the studies is an answer to some but the study will certainly be labeled as questionable as it will gore someones’ ox. And, of course, the study WILL be compromised, as it will tend to advocate for Big Pharma.
    4. Nowhere in the US Constitution does it allow for the Federal government to pay for such a study. That unconstitutional spending is done all the time is no reason to allow for more unconstitutional spending.

  • As a person uneducated in the world of medicine or science I have a question for Dr. Ernst or Dr. Money-Kyrle:
    I am somewhat confused about the process of getting medicines/cures approved. It seems to me that if someone had the cure for Cancer that worked as well as some alternative medicine people promote, it would not be difficult to get the money together to do an actual high quality study. If I was an investor and a person came to me with some sort of real evidence that their cancer cure worked I would back it in a second – the return on investment would be astounding if it worked even if I couldn’t market it in the US.

    So not having the money to do studies doesn’t seem plausible unless there is something I don’t know.

    Let’s say (XYZ) cures cancer. I’m a holistic practitioner and I used (XYZ) to cure 50 patients and have the charts to show it as well as xrays of the cancer and then monitor the patients for a certain number of years and make sure the cancer either doesn’t come back or at least doesn’t come back as much as current medical norms.

    Now what? I get some money together, take all of this data to a University and ask them to do a study? Then I can use that study to get it approved by whatever regulatory body my country uses?

    I have heard that the FDA is corrupt and wouldn’t let it go through. Although if I imagine myself holding a genuine high quality study showing that my cancer cure works, standing outside the FDA offices after getting rejected, I feel that I could figure out a way to get the information out to the world in a way that doctors would be using my method within a few years.

    Now, don’t get me wrong on this, I don’t blindly trust the FDA, doctors, pharma or alternative medicine. Each time I go to the doctor or alternative medicine place I make sure I try to understand everything that is going on and I generally do more research before and afterwards – I’ve been wrong in my own research plenty of times but I think I am getting smarter with each mistake I make.

    On the contrary to what I have been told, I find the FDA much too lax – I personally took a supplement for months that ended up having amphetamines in it and didn’t find out until years later when the FDA figured it out and shut the company down. Also one can see this in the current medical drug epidemic where people are being over-prescribed. This seems like an under-regulation problem to me.

    Sorry for being long-winded, my question is basically:
    If you found that (XYZ) cured cancer much better than what is currently available, what would be the basic steps that would have to be taken and barriers to entry that would need to be navigated in order to get (XYZ) sold for the purpose of curing cancer? Is this possible to describe in a way that a layman like me would understand?

    The impossibility of getting *real cures* approved for use is something I have heard over and over in the alternative healing space and I feel some clarity on the process and barriers would be useful to me and others who are unsure who to trust.

Leave a Reply to FrankO Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

If you want to be able to edit your comment for five minutes after you first submit it, you will need to tick the box: “Save my name, email, and website in this browser for the next time I comment.”
Recent Comments

Note that comments can be edited for up to five minutes after they are first submitted but you must tick the box: “Save my name, email, and website in this browser for the next time I comment.”

The most recent comments from all posts can be seen here.