This was essentially the question raised in a correspondence with a sceptic friend. His suspicion was that statistical methods might produce false-positive overall findings, if the research is done by enthusiasts of the so-called alternative medicine (SCAM) in question (or other areas of inquiry which I will omit because they are outside my area of expertise). Consciously or inadvertently, such researchers might introduce a pro-SCAM bias into their work. As the research is done mostly by such enthusiasts; the totality of the evidence would turn out to be heavily skewed in favour of the SCAM under investigation. The end-result would then be a false-positive overall impression about the SCAM which is less based on reality than on the wishful thinking of the investigators.
How can one deal with this problem?
How to minimise the risk of being overwhelmed by false-positive research?
Today, we have several mechanisms and initiatives that are at least partly aimed at achieving just this. For instance, there are guidelines on how to conduct the primary research so that bias is minimised. The CONSORT statements are an example. As many studies pre-date CONSORT, we need a different approach for reviews of clinical trials. The PRISMA guideline or the COCHRANE handbook are attempts to make sure systematic reviews are transparent and rigorous. These methods can work quite well in finding the truth, but one needs to be aware, of course, that some researchers do their very best to obscure it. I have also tried to go one step further and shown that the direction of the conclusion correlates with the rigour of the study (btw: this was the paper that prompted Prof Hahn’s criticism and slander of my work and person).
So, problem sorted?
The trouble is that over-enthusiastic researchers may not always adhere to these guidelines, they may pretend to adhere but cut corners, or they may be dishonest and cheat. And what makes this even more tricky is the possibility that they do all this inadvertently; their enthusiasm could get the better of them, and they are doing research not to TEST WHETHER a treatment works but to PROVE THAT it works.
In the realm of SCAM we have a lot of this – trust me, I have seen it often with my own eyes, regrettably sometimes even within my own team of co-workers. The reason for this is that SCAM is loaded with emotion and quasi-religious beliefs; and these provide a much stronger conflict of interest than money could ever do, in my experience.
And how might we tackle this thorny issue?
After thinking long and hard about it, I came up in 2012 with my TRUSTWORTHYNESS INDEX:
If we calculated the percentage of a researcher’s papers arriving at positive conclusions and divided this by the percentage of his papers drawing negative conclusions, we might have a useful measure. A realistic example might be the case of a clinical researcher who has published a total of 100 original articles. If 50% had positive and 50% negative conclusions about the efficacy of the therapy tested, his TI would be 1.
Depending on what area of clinical medicine this person is working in, 1 might be a figure that is just about acceptable in terms of the trustworthiness of the author. If the TI goes beyond 1, we might get concerned; if it reaches 4 or more, we should get worried.
An example would be a researcher who has published 100 papers of which 80 are positive and 20 arrive at negative conclusions. His TI would consequently amount to 4. Most of us equipped with a healthy scepticism would consider this figure highly suspect.
Of course, this is all a bit simplistic, and, like all other citation metrics, my TI provides us not with any level of proof; it merely is a vague indicator that something might be amiss. And, as stressed already, the cut-off point for any scientist’s TI very much depends on the area of clinical research we are dealing with. The lower the plausibility and the higher the uncertainty associated with the efficacy of the experimental treatments, the lower the point where the TI might suggest something to be fishy.
Based on this concept, I later created the ALTERNATIVE MEDICINE HALL OF FAME. This is a list of researchers who manage to go through life researching their particular SCAM without ever publishing a negative conclusion about it. In terms of TI, these people have astronomically high values. The current list is not yet long, but it is growing:
John Weeks (editor of JCAM)
Deepak Chopra (US entrepreneur)
Cheryl Hawk (US chiropractor)
David Peters (osteopathy, homeopathy, UK)
Nicola Robinson (TCM, UK)
Peter Fisher (homeopathy, UK)
Simon Mills (herbal medicine, UK)
Gustav Dobos (various, Germany)
Claudia Witt (homeopathy, Germany and Switzerland)
George Lewith (acupuncture, UK)
John Licciardone (osteopathy, US)
The logical consequence of a high TI would be that researchers of that nature are banned from obtaining research funds and publishing papers, because their contribution is merely to confuse us and make science less reliable.
I am sure there are other ways of addressing the problem of being mislead by false-positive research. If you can think of one, I’d be pleased to hear about it.