What can consumers do when they are confronted with confusing or contradictory evidence?

Simply put, in the realm of SCAM, we seem to have two types of people:

1. those who don’t care a hoot about evidence;
2. those who try their best to follow the evidence.

The first group is replete with SCAM enthusiasts who make their decisions based purely on habit, emotion, intuition etc. They are beyond my reach, I fear. It is almost exclusively the second group for whom I write this blog.

And that could be relatively easy, if the evidence were always accessible, understandable, straight forward, conclusive and convincing. But sadly, in SCAM (as in most other areas of healthcare), the evidence is full of apparent and real contradictions. In this situation, it is often difficult even for experts to understand what is going on; for lay people this must be immeasurably more confusing. Yet, it is the lay consumers who often will take the decision to use or not use this or that SCAM. They therefore need our help.

What can consumers do when they are confronted with contradictory evidence?

How can they distinguish right from wrong?

• Some articles claim that homeopathy works – others say it is just a placebo therapy.
• Some experts claim that chiropractic is safe – others say it can do serious harm.
• Some articles claim that SCAM-practitioners are competent – others say this is not true.
• Some experts claim that SCAM is the future – others stress that it is obsolete.

What can a lay person with no or very little understanding of science do to see through this fog of contradictions?

Let me try to provide consumers with a step by step approach to get closer to the truth by asking a few incisive questions:

1. WHERE DID YOU READ THE CLAIM? If it was in a newspaper, magazine, website, etc. take it with a pinch of salt (double the dose of salt, if it’s from the Daily Mail).
2. CAN YOU RETRACE THE CLAIM TO A SCIENTIFIC PAPER? This might challenge you skills as a detective, but it is always well-worth finding the original source of a therapeutic claim in order to judge its credibility. If no good source can be found, I advise caution.
3. IN WHICH MEDICAL JOURNAL WAS THE CLAIM PUBLISHED? Be aware of the fact that there are dozens of SCAM-journals that would publish virtually any rubbish.
4. WHO ARE THE AUTHORS OF THE SCIENTIFIC PAPER? It might be difficult for a lay person to evaluate their credibility. But there might be certain pointers; for instance, authors affiliated to a university tend to be more credible than SCAM-practitioners who have no such affiliations or authors working for a lobby-group.
5. WHAT SORT OF ARTICLE IS THE ORIGINAL SOURCE OF THE CLAIM? Is it a proper experimental study or a mere opinion piece? If possible, try to find a good-quality (perhaps even a Cochrane) review on the subject.
6. ARE THERE OTHER RESEARCHERS WHO HAVE ARRIVED AT SIMILAR CONCLUSIONS? If the claim is based on just one solitary piece of research or opinion, it clearly weighs less than a consensus of experts.
7. DO PUBLICATIONS EXIST THAT DISAGREE WITH THE CLAIM? Even if there are several scientific papers from different teams of researchers supporting the claim, it is important to find out whether the claim is shared by all experts in the field.

Eventually, you might get a good impression about the veracity of the claim. But sometimes you also might end up with a bunch of systematic reviews of which several support, while others reject the claim. And all of them could look similarly credible to your untrained eyes. Does that mean your attempt to find the truth of the matter has been frustrated?

Not necessarily!

In this case, you would probably consider the following options:

1. You could do a simple ‘pea count’; this would tell you whether the majority of reviews is pro or contra the claim. However, this might be your worst bet for arriving at a sound conclusion. The quantity of the evidence usually is far less important than its quality.
2. If you have no training to judge the quality of a review, you might just go with the most recent and up-to-date review. This, however, would also be fraught with problems, as you can, of course, not be sure that the most recent one is also the least biased assessment.
3. Perhaps you can somehow get an impression about the respectability of the source. If, for instance, there is a recent Cochrane review, I advise to go with that one.
4. Look up the profession of the authors of the review. The pope is unlikely to condemn Catholicism; likewise, you will find very few homeopaths who are critical of homeopathy, or chiropractors who are critical of chiropractic, etc. I know this is a very crude ‘last resort’ for replacing an authorative evaluation of the claim. But, if that’s all you have, it is better than nothing. Ask yourself who can normally be trusted more, the SCAM-practitioner or lobbyist who makes a living from the claim or an independent academic who has no such conflict of interest?

If all of this does not help you to decide whether a therapeutic claim is trustworthy or not, my advice has always been to reflect on this: IF IT SOUNDS TOO GOOD TO BE TRUE, IT PROBABLY IS.