Medical treatments with no direct effect, such as homeopathy, are surprisingly popular. But how does a good reputation of such treatments spread and persist? Researchers from the Centre for the Study of Cultural Evolution in Stockholm believe that they have identified the mechanism.
They argue that most medical treatments result in a range of outcomes: some people improve while others deteriorate. If the people who improve are more inclined to tell others about their experiences than the people who deteriorate, ineffective or even harmful treatments would maintain a good reputation.
They conducted a fascinating study to test the hypothesis that positive outcomes are overrepresented in online medical product reviews, examined if this reputational distortion is large enough to bias people’s decisions, and explored the implications of this bias for the cultural evolution of medical treatments.
The researchers compared outcomes of weight loss treatments and fertility treatments as evidenced in clinical trials to outcomes reported in 1901 reviews on Amazon. Subsequently, in a series of experiments, they evaluated people’s choice of weight loss diet after reading different reviews. Finally, a mathematical model was used to examine if this bias could result in less effective treatments having a better reputation than more effective treatments.
The results of these investigations confirmed the hypothesis that people with better outcomes are more inclined to write reviews. After 6 months on the diet, 93% of online reviewers reported a weight loss of 10 kg or more, while just 27% of clinical trial participants experienced this level of weight change. A similar positive distortion was found in fertility treatment reviews. In a series of experiments, the researchers demonstrated that people are more inclined to begin a diet that was backed by many positive reviews, than a diet with reviews that are representative of the diet’s true effect. A mathematical model of medical cultural evolution suggested that the size of the positive distortion critically depends on the shape of the outcome distribution.
The authors concluded that online reviews overestimate the benefits of medical treatments, probably because people with negative outcomes are less inclined to tell others about their experiences. This bias can enable ineffective medical treatments to maintain a good reputation.
To me, this seems eminently plausible; but there are, of course, other reasons why bogus treatments survive or even thrive – and they may vary in their importance to the overall effect from treatment to treatment. As so often in health care, things are complex and there are multiple factors that contribute to a phenomenon.
15 Responses to How ineffective treatments retain a good reputation
Leave a Reply to FrankO Cancel reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
This is interesting. The precise opposite seems to apply to reviews on e.g. TripAdvisor and Amazon. There people who feel ripped off and angry seem to be predisposed to post a negative review. Perhaps snake oil users feel more embarassed to have been duped?
As I was reading the article, I was thinking the same thing. Perhaps someone feels ripped off by a shampoo, or whatever (and yells about it in review comments) but, somehow not by homeopathy, fad diet books, etc (assuming they notice it did nothing) — or are embarrassed …
Maybe it is less embarassement but the fact that they blame themselves for the negative outcome. If you go on a certain diet and don’t lose weight (or not as much as expected) you’re probably more likely to think you’ve done something wrong like not being disciplined enough. You wouldn’t necessarily think it’s the particular program’s fault – especially since you’ve very likely experienced several other treatments failing on you anyhow. Whereas with a shampoo you’d automatically blame the product as you’d expect it to be standardized and to have the same outcome for everybody.
But why is there more likelihood of positive anecdotes than negative ones? Could it be the propensity of proponents of snake oil to blame the victim when it doesn’t work?
Unless I’m mistaken, Amazon differentiates between those who actually bought the product and those who are merely commenting. So in theory it should be possible to filter out the noise (from people who have seen through the sham without buying) and concentrate on genuine customers of the products.
How would Amazon perform this differentiation? By IP address wouldn’t work due to roaming IP allocations. By e-mail address wouldn’t work due to the purchaser’s e-mail address frequently being different from the product user’s e-mail address. E.g. one member of a household has an Amazon account for purchases made by all members of their family.
I’ve been asked to write reviews on Amazon by authors who’ve sent me a copy of their book. I refuse to do this, but there is no process to prevent me from doing it — other than ethics 🙂
Amazon uses the logged in account holder.
That’s the *verified purchaser after the reviewer’s name.
Amazon adds the words “Verified Purchase” where a review has been posted by someone using an account that has also been used to buy the product.
Aye. “I’m not worthy” “I didn’t do it right” etc. Oi. Those poor ppl.
MLM companies blame the victim when he-or-she begins to notice it’s a scam…
I must confess that when I first read Paragraph 4, I was wondering briefly why they were only using reviews written on Amazon back in 1901. Then … oh right.
Not enough coffee this morning.
@FrankO and Acleron,
If I booked a hotel then upon arrival I discovered that the hotel doesn’t actually exist, followed by performing a bit of online research revealing that others had found the same, I would be reticent about publicly admitting my ineptitude/laziness while booking the hotel.
If that hotel does exist, but gives very poor service, then I have tangible things to publicly complain about.
It boils down to the tangibility of the service being provided — i.e. tangible as in: clear and definite; real.
sCAM is NOT alternative tangible medicine; sCAM is intangible medicine that “works” occasionally due to the placebo reaction.
The humiliation resulting from my non-existent hotel example would be on a par with the humiliation resulting from paying for intangible health treatments. Most victims of these scams feel far too ashamed to speak out about it.
Most hotel and travel booking agents provide a means of obtaining reimbursement from scams; the sCAM industry provides no such promissory. Buyer beware.
“by placebo …”
Also, if you were going to get better anyway because you waited until the thing was at its worst before you sought treatment.
(I know you know. Sorry. Just completeness.)
I’m glad you highlighted this issue. The effect to which you refer is frequently called “regression to the mean”, which isn’t correct[1], but much more importantly it is not part of the placebo reaction per se; it is an independent confounding variable that often causes false positive results (a type I statistical error).
A properly conducted placebo-controlled trial will minimize the influence of this error and other type I and type II errors. sCAM practitioners, clients, and apologists share one thing in common: illiteracy in statistics, the scientific method, and critical thinking.
[1] The logical fallacy involved is the “regressive fallacy” aka the “regression fallacy”:
http://www.skepdic.com/regressive.html
http://en.wikipedia.org/wiki/Regression_fallacy
The term “regression to the mean” is applicable to cases where the initial and/or final data samples just happen to be away from the mean (or the trend) due to randomness, or any other uncorrelated and unpredictable reasons. In practice, the effects of such end-pointing errors are reduced by applying one of the Generalized Hamming windows to the sampled data. I don’t think such methods are appropriate to the analysis of short-term trials of medical interventions, but they are highly appropriate to the long-term monitoring and analysis of patient progress.
PS: Many thanks to you and Mojo for your Amazon “Verified Purchaser” comments.
One other feature that probably helps boost the positive reputation of sCAM is the unshakeably positive faith of the healers themselves. I’ve met several, and most of them speak messianically about their particular brand of ineffective treatment. Common to all is a tendency to recall only the patients who appear to benefit and to forget the other 50% who don’t. Of course, this effect is amplified by patients who think they’ve benefitted returning, while those who are unimpressed don’t come back. sCAM practitioners have a clientele statistically loaded with satisfied customers.
I now await a response from someone saying the same applies to real medicine, with the dissatisfied customers going to the various forms of sCAM.
“Survivorship bias, or survival bias, is the logical error of concentrating on the people or things that ‘survived’ some process and inadvertently overlooking those that did not because of their lack of visibility.
Survivorship bias can lead to overly optimistic beliefs because failures are ignored…”
http://en.wikipedia.org/wiki/Survivorship_bias
Once this overly optimistic belief has developed then the “backfire effect” takes over to create true believers:
“The Misconception: When your beliefs are challenged with facts, you alter your opinions and incorporate the new information into your thinking.
The Truth: When your deepest convictions are challenged by contradictory evidence, your beliefs get stronger.” — David McRaney
http://youarenotsosmart.com/2011/06/10/the-backfire-effect/
Also relevant is the continued influence effect:
http://www.skepdic.com/continuedinfluenceeffect.html