MD, PhD, MAE, FMedSci, FRSB, FRCP, FRCPEd.

The efficacy or effectiveness of medical interventions is, of course, best tested in clinical trials. The principle of a clinical trial is fairly simple: typically, a group of patients is divided (preferably at random) into two subgroups, one (the ‘verum’ group) is treated with the experimental treatment and the other (the ‘control’ group) with another option (often a placebo), and the eventual outcomes of the two groups is compared. If done well, such studies are able to exclude biases and confounding factors such that their findings allow causal inference. In other words, they can tell us whether an outcome was caused by the intervention per se or by some other factor such as the natural history of the disease, regression towards the mean etc.

A clinical trial is a research tool for testing hypotheses; strictly speaking, it tests the ‘null-hypothesis’: “the experimental treatment generates the same outcomes as the treatment of the control group”. If the trial shows no difference between the outcomes of the two groups, the null-hypothesis is confirmed. In this case, we commonly speak of a negative result. If the experimental treatment was better than the control treatment, the null-hypothesis is rejected, and we commonly speak of a positive result. In other words, clinical trials can only generate positive or negative results, because the null-hypothesis must either be confirmed or rejected – there are no grey tones between the black of a negative and the white of a positive study.

For enthusiasts of alternative medicine, this can create a dilemma, particularly if there are lots of published studies with negative results. In this case, the totality of the available trial evidence is negative which means the treatment in question cannot be characterised as effective. It goes without saying that such an overall conclusion rubs the proponents of that therapy the wrong way. Consequently, they might look for ways to avoid this scenario.

One fairly obvious way of achieving this aim is to simply re-categorise the results. What, if we invented a new category? What, if we called some of the negative studies by a different name? What about NON-CONCLUSIVE?

That would be brilliant, wouldn’t it. We might end up with a simple statistic where the majority of the evidence is, after all, positive. And this, of course, would give the impression that the ineffective treatment in question is effective!

How exactly do we do this? We continue to call positive studies POSITIVE; we then call studies where the experimental treatment generated worst results than the control treatment (usually a placebo) NEGATIVE; and finally we call those studies where the experimental treatment created outcomes which were not different from placebo NON-CONCLUSIVE.

In the realm of alternative medicine, this ‘non-conclusive result’ method has recently become incredibly popular . Take homeopathy, for instance. The Faculty of Homeopathy proudly claim the following about clinical trials of homeopathy: Up to the end of 2011, there have been 164 peer-reviewed papers reporting randomised controlled trials (RCTs) in homeopathy. This represents research in 89 different medical conditions. Of those 164 RCT papers, 71 (43%) were positive, 9 (6%) negative and 80 (49%) non-conclusive.

This misleading nonsense was, of course, warmly received by homeopaths. The British Homeopathic Association, like many other organisations and individuals with an axe to grind lapped up the message and promptly repeated it: The body of evidence that exists shows that much more investigation is required – 43% of all the randomised controlled trials carried out have been positive, 6% negative and 49% inconclusive.

Let’s be clear what has happened here: the true percentage figures seem to show that 43% of studies (mostly of poor quality) suggest a positive result for homeopathy, while 57% of them (on average the ones of better quality) were negative. In other words, the majority of this evidence is negative. If we conducted a proper systematic review of this body of evidence, we would, of course, have to account for the quality of each study, and in this case we would have to conclude that homeopathy is not supported by sound evidence of effectiveness.

The little trick of applying the ‘NON-CONCLUSIVE’ method has thus turned this overall result upside down: black has become white! No wonder that it is so popular with proponents of all sorts of bogus treatments.

4 Responses to The alchemists of alternative medicine – part 3: the ‘NON-CONCLUSIVE’ method

  • Excellent post! In this context we must mention Bronfort &al’s article, praised by the chiropractic community for all its “favourable evidence”: Bronfort G, Haas M. Effectiveness of manual therapies: the UK evidence report. Chiropr Man Therap [Internet]. BioMed Central; 2010 [cited 2014 Feb 6];18(3):3. Available from: http://www.biomedcentral.com/1746-1340/18/3

    Here they even divide the non-conclusive (or inconclusive) evidence into no less than three subcategories: “Inconclusive but favorable evidence”, “Inconclusive and unclear direction of evidence”, and “Inconclusive but non-favorable evidence”. Believe it or not.
    On the other hand, when the evidence is clear, they lump the degrees of strength together into “High and moderate quality positive evidence” and “High and moderate quality negative evidence”, respectively. Perhaps because among the positive evidence there was only one of high quality (for low back pain).

    In my opinion, the article is well written and honest in its reporting of the examined studies and conclusions — apart from the alchemy with the terminology. However, since the article gives almost no support to chiropractic, I have been puzzled why the chiropractors so love it. But thanks to your post here, I can understand it now: Not counting in the figures, the word “negative” (of evidence) occurs 0 times in the article text or abstract, “positive” (of effect) 1 time (0 in the abstract), and the word “inconclusive” 47 times (4 in the abstract). And “inconclusive” here of course means “almost positive” for them, particularly the “Inconclusive but favorable” category.
    Of the 65 conditions studied, they deemed 19 as having “positive” evidence for chiropractic treatment, 40 as “inconclusive”, of the “inconclusive”, no less than 27 were counted as “favorable” and 6 as negative. With a clearer eye, this means 46 negative vs 19 positive. But the abstract reflects this only very vaguely, and many people don’t read the rest, perhaps. Lots of wishful thinking there.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe via email

Enter your email address to receive notifications of new blog posts by email.

Recent Comments

Note that comments can be edited for up to five minutes after they are first submitted but you must tick the box: “Save my name, email, and website in this browser for the next time I comment.”

The most recent comments from all posts can be seen here.

Archives
Categories