Why many results of alternative medicine research are wrong

Blinding patients in clinical trials is a key methodological procedure for minimizing bias and thus making sure that the results are reliable. In alternative medicine, blinding is not always straight forward, and many studies are therefore not patient-blinded. We all know that this can introduce bias into a trial, but how large is its effect on study outcomes?

This was the research question addressed by a recent systematic review of randomized clinical trials with one sub-study (i.e. experimental vs control) involving blinded patients and another, otherwise identical, sub-study involving non-blinded patients. Within each trial, the researchers compared the difference in effect sizes (i.e. standardized mean differences) between the two sub-studies. A difference <0 indicates that non-blinded patients generated a more optimistic effect estimate. The researchers then pooled the differences with random-effects inverse variance meta-analysis, and explored reasons for heterogeneity.

The main analysis included 12 trials with a total of 3869 patients. Ten of these RCTs were studies of acupuncture. The average difference in effect size for patient-reported outcomes was -0.56 (95% confidence interval -0.71 to -0.41), (I(2 )= 60%, P = 0.004), indicating that non-blinded patients exaggerated the effect size by an average of 0.56 standard deviation, but with considerable variation. Two of the 12 trials also used observer-reported outcomes, showing no indication of exaggerated effects due lack of patient blinding.

There was an even larger effect size difference in the 10 acupuncture trials [-0.63 (-0.77 to -0.49)], than in the two non-acupuncture trials [-0.17 (-0.41 to 0.07)]. Lack of patient blinding was also associated with increased attrition rates and the use of co-interventions: ratio of control group attrition risk 1.79 (1.18 to 2.70), and ratio of control group co-intervention risk 1.55 (0.99 to 2.43).

The authors conclude that this study provides empirical evidence of pronounced bias due to lack of patient blinding in complementary/alternative randomized clinical trials with patient-reported outcomes.

This is a timely, rigorous and important analysis. In alternative medicine, we currently see a proliferation of trials that are not patient-blinded. We always suspected that they are at a high risk of generating false-positive results – now we know that this is, in fact, the case.

What should we do with this insight? In my view, the following steps would be wise:

1. Take the findings from the existing trials that are devoid of patient-blinding with more than just a pinch of salt.
2. Discourage the funding of future studies that fail to include patient-blinding.
3. If patient-blinding is truly and demonstrably impossible – which is not often the case – make sure that the trialists at least include blinding of the assessors of the primary outcome measures.