Researchers tend to report only studies that are positive, while leaving negative trials unpublished. This publication bias can mislead us when looking at the totality of the published data. One solution to this problem is the p-curve. A significant p-value indicates that obtaining the result within the null distribution is improbable. The p-curve is the distribution of statistically significant p-values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves – containing more low (.01s) than high (.04s) significant p-values – only right-skewed p-curves are diagnostic of evidential value. By telling us whether we can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.
The authors of this article tested the distributions of sets of statistically significant p-values from placebo-controlled studies of homeopathic ultramolecular dilutions. Such dilute mixtures are unlikely to contain a single molecule of an active substance. The researchers tested whether p-curve accurately rejects the evidential value of significant results obtained in placebo-controlled clinical trials of homeopathic ultramolecular dilutions.
Their inclusion criteria were as follows:
- Study is accessible to the authors.
- Study is a clinical trial comparing ultramolecular dilutions to placebo.
- Study is randomized, with randomization method specified.
- Study is double-blinded.
- Study design and methodology result in interpretable findings (e.g., an appropriate statistical test is used).
- Study reports a test statistic for the hypothesis of interest.
- Study reports a discrete p-value or a test statistic from which a p-value can be derived.
- Study reports a p-value independent of other p-values in p-curve.
The first 20 studies, in the order of search output, that met the inclusion criteria were used for analysis.
The researchers found that p-curve analysis accurately rejects the evidential value of statistically significant results from placebo-controlled, homeopathic ultramolecular dilution trials (1st graph below). This result indicates that replications of the trials are not expected to replicate a statistically significant result. A subsequent p-curve analysis was performed using the second significant p-value listed in the studies, if a second p-value was reported, to examine the robustness of initial results. P-curve rejects evidential value with greater statistical significance (2nd graph below). In essence, this seems to indicate that those studies of highly diluted homeopathics that reported positive findings, i. e. homeopathy is better than placebo, are false-positive results due to error, bias or fraud.
True effects with significant non-central distributions would have a greater density of low p-values than high p-values resulting in a right-skewed p-curve (like the dotted green lines in the above graphs). The fact that such a shape is not observed for studies of homeopathy confirms the many analyses previously demonstrating that ULTRAMOLECULAR HOMEOPATHIC REMEDIES ARE PLACEBOS.