I have said it so often that I hesitate to state it again: an uncritical researcher is a contradiction in terms. This begs the question as to how critical the researchers of alternative medicine truly are. In my experience, most tend to be uncritical in the extreme. But how would one go about providing evidence for this view? In a previous blog-post, I have suggested a fairly simple method: to calculate an index of negative conclusions drawn in the articles published by a specific researcher. This is what I wrote:
If we calculated the percentage of a researcher’s papers arriving at positive conclusions and divided this by the percentage of his papers drawing negative conclusions, we might have a useful measure. A realistic example might be the case of a clinical researcher who has published a total of 100 original articles. If 50% had positive and 50% negative conclusions about the efficacy of the therapy tested, his trustworthiness index (TI) would be 1.
Depending on what area of clinical medicine this person is working in, 1 might be a figure that is just about acceptable in terms of the trustworthiness of the author. If the TI goes beyond 1, we might get concerned; if it reaches 4 or more, we should get worried.
An example would be a researcher who has published 100 papers of which 80 are positive and 20 arrive at negative conclusions. His TI would consequently amount to 4. Most of us equipped with a healthy scepticism would consider this figure highly suspect.
So how would alternative medicine researchers do, if we applied this method for assessing their trustworthiness? Very poorly, I fear – but that is speculation! Let’s see some data. Let’s look at one prominent alternative medicine researcher and see. As an example, I have chosen Professor George Lewith (because his name is unique which avoids confusion with researchers), did a quick Medline search to identify the abstracts of his articles on alternative medicine, and extracted the crucial sentence from the conclusions of the most recent ones:
- The study design of registered TCM trials has improved in estimating sample size, use of blinding and placebos
- Real treatment was significantly different from sham demonstrating a moderate specific effect of PKP
- These findings highlight the importance of helping patients develop coherent illness representations about their LBP before trying to engage them in treatment-decisions, uptake, or adherence
- Existing theories of how context influences health outcomes could be expanded to better reflect the psychological components identified here, such as hope, desire, optimism and open-mindedness
- …mainstream science has moved on from the intellectual sterility and ad hominem attacks that characterise the sceptics’ movement
- Trustworthy and appropriate information about practitioners (e.g. from professional regulatory bodies) could empower patients to make confident choices when seeking individual complementary practitioners to consult
- Comparative effectiveness research is an emerging field and its development and impact must be reflected in future research strategies within complementary and integrative medicine
- The I-CAM-Q has low face validity and low acceptability, and is likely to produce biased estimates of CAM use if applied in England, Romania, Italy, The Netherlands or Spain
- Our main finding was of beta power decreases in primary somatosensory cortex and SFG, which opens up a line of future investigation regarding whether this contributes toward an underlying mechanism of acupuncture.
- …physiotherapy was appraised more negatively in the National Health Service than the private sector but osteopathy was appraised similarly within both health-care sectors
This is a bit tedious, I agree, so I stop after just 10 articles. But even this short list does clearly indicate the absence of negative conclusions. In fact, I see none at all – arguably a few neutral ones, but nothing negative. All is positive in the realm of alternative medicine research then? In case you don’t agree with that assumption, you might prefer to postulate that this particular alternative medicine researcher somehow avoids negative conclusions. And if you believe that, you are not far from considering that we are being misinformed.
Alternative medicine is not really a field where one might reasonably expect that rigorous research generates nothing but positive results; even to expect 50 or 40% of such findings would be quite optimistic. It follows, I think, that if researchers only find positives, something must be amiss. I have recently demonstrated that the most active research homeopathic group (Professor Witt from the Charite in Berlin) has published nothing but positive findings; even if the results were not quite positive, they managed to formulate a positive conclusion. Does anyone doubt that this amounts to misinformation?
So, I do have produced at least some tentative evidence for my suspicion that some alternative medicine researchers misinform us. But how precisely do they do it? I can think of several methods for avoiding publishing a negative result or conclusion, and I fear that all of them are popular with alternative medicine researchers:
- design the study in such a way that it cannot possibly give a negative result
- manipulate the data
- be inventive when it comes to statistics
- home in on to the one positive aspect your generally negative data might show
- do not write up your study; like this nobody will ever see your negative results
And why do they do it? My impression is that they use science not for testing their interventions but for proving them. Critical thinking is a skill that alternative medicine researchers do not seem to cultivate. Often they manage to hide this fact quite cleverly and for good reasons: no respectable funding body would give money for such an abuse of science! Nevertheless, the end-result is plain to see: no negative conclusions are being published!
There are at least two further implications of the fact that alternative medicine researchers misinform the public. The first concerns the academic centres in which these researchers are organised. If a prestigious university accommodates a research unit of alternative medicine, it gives considerable credence to alternative medicine itself. If the research that comes out of the unit is promotional pseudo-science, the result, in my view, amounts to misleading the public about the value of alternative medicine.
The second implication relates to the journals in which researchers of alternative medicine prefer to publish their articles. Today, there are several hundred journals specialised in alternative medicine. We have shown over and over again that these journals publish next to nothing in terms of negative results. In my view, this too amounts to systematic misinformation.
My conclusion from all this is depressing: the type of research that currently dominates alternative medicine is, in fact, pseudo-research aimed not at rigorously falsifying hypotheses but at promoting bogus treatments. In other words alternative medicine researchers crucially contribute to the ‘sea of misinformation’ in this area.
14 Responses to Drowning in a sea of misinformation. Part 14: Alternative medicine researchers
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
to be fair, I should perhaps provide equivalent ‘sound bites’ from the conclusions of my own most recent 10 Medline-listed abstracts. here they are:
1 In the Republic of Korea, traditional medical practices often appear to have adverse effects
2 We found insufficient evidence to determine whether acupuncture is effective for controlling menopausal vasomotor symptoms
3 The evidence of the effectiveness of OMT for pediatric conditions remains unproven due to the paucity and low methodological quality of the primary studies
4 …the evidence on ginseng as an effective treatment for managing menopause symptoms is limited
5 Many patients and consumers in the UK use CAM; healthcare professionals should therefore responsibly advise their patients about the use of CAM.
6 Oral borage oil and evening primrose oil lack effect on eczema
7 Overall, GPs were less likely to endorse CAMs than previously shown (38% versus 19%).
8 …the hypothesis that dietary supplements are effective treatments for PC patients is not supported by sound clinical evidence
9 There is an indication that ALE has potential in lowering cholesterol levels, but the evidence is, as yet, not convincing
(3 of the 10 papers were Cochrane reviews)
What you say is very true, but your ratios are more than a bit skewed.
Leaving aside the bayesian stuff which I don’t understand, a simple experimental-control study on something that finds an effect at the p<0.05 significance level, is supposed to have a 5% false positive rate. The Type-II error rate then depends on sample size, control group size and the effect size. The power of the study is then (1-Type-II error). You know all this, of course, but let's run some numbers:
Suppose a researcher is very lucky and is running studies on a therapy with a huge effect on the order of 1 SD of effect size. Even with small studies of about 10-15, his Type II error rate is only going to be about 20% at worst, or about 80% of the time (leaving bayesian issues aside), he's going to find an effect.
So, your lucky CAM research in that scenario is going to publish 4 positive papers for every negative paper, just by [overly] simple statistical analysis, so his TI is going to be 4:1 just because he's lucky enough to be studying something with that huge an effect. Such effects are rare in CAM research, but the preliminary results on TM and PTSD are pointing in that direction.
Now, more realistically, Cohen's d is going to be moderate: 0.5 effect size. Studying things in THAT scenario, one is going to have a less impressive success rate, but as long as the statistical power of a study on a real effect is greater than 50%, you're going to have a TI of greater than 1, which, according to YOU, means that the researcher is untrustworthy, even though the research is perfectly sound.
Here is, I believe, a realworld example of what actually goes on with CAM researchers, specifically the researchers out of the TM university in Iowa.
Before they publish anything, they do lots and lots of tiny pilot projects, hoping to find a positive effect of TM vs control on . Once they get that positive hit, they shop around for a journal willing to publish. If all else fails, it gets published in the _Collected Papers of Research on TM_ volumes put out by the TM organization.
Once they get multiple positive results from a given comparison, they start searching for external funding.
If they continue to get positive hits, they search for more prominent sources of funding plus external collaborators. Once that happens, they have a much harder time ignoring negative results: external collaborators are far more likely to want to publish, regardless of whether or not the outcome puts TM in a favorable light.
If they continue to get favorable results, they go after the big grant money. The latest cardiovascular research on TM is funded by the NIH (but not the Office of Complementary Medicine, as far as I know). The ongoing PTSD research on TM is being funded by the military and the VA and is part of a larger project of comparing different treatment modalities including mindfulness and TM.
Once the research on TM gets to the point where strong positive findings are being generated from public grant money by teams that include collaborators outside the TM mainstream, the next phase of research should start: direct head-to-head studies of various competing forms of meditation conducted either by teams comprised of advocates for competing research, or even better, completely independent research conducted by people with no vested interest in any form of research.
Here’s an early example of the first type of study. Each meditation practice that was studied -TM, mindfulness, “low mindfulness relaxation” [Benson’s Relaxation Response by another name]- had at least one advocate on the team of researchers. The meditation teachers used were all trained by the respective advocate to mimic the professionalism of the TM teachers, including wearing business attire, memorizing their presentation, and using professionally done charts referring to real research on that form of meditation. Test subjects were randomly assigned to a meditation teachers, and their expectations were evaluated after the presentation was over (no significant differences in expectations were found between groups). Data collection was done at the rest homes in the Boston area by graduate students at Harvard University who were blind to the group assignment. The researchers themselves were also blind to the research assignment:
http://www.ncbi.nlm.nih.gov/pubmed/2693686
Transcendental meditation, mindfulness, and longevity: an experimental study with the elderly
Any future head-to-head studies comparing meditation need to follow at least that rigorous a design, don’t you agree?
I don’t know of any examples of the fully independent research done on TM except some preliminary stuff done in the 70’s, much of which was too small to find any effect-size less than 1.5 SD, but if the next round of PTSD research should prove as impressive as the pilot studies have, the VA and DoD will be obligated to commission fully independent studies to do head-to-head comparisons of all the candidate treatments for PTSD, I am sure (or at least I *hope* that this is part of the planned process -you never know with the US government).
Anyway, my point is, even with an obviously biased and partisan group of researchers, real science can eventually get done, no matter what the Trust Index you assign to them.
By the way, there are certain fields of study where it is politically incorrect to publish *positive* research. Specifically, anything having to do with Genetically Modified Organisms and safety: any study that finds a difference between a GMO and the control group is automatically going to be attacked by the entire scientific community, includikng not just the researchers in the field, but the heads of many, if not most, of the major professional scientific organizations.
Researchers know this. They are also aware that since it is possible to publish ANY null-finding paper in a reasonably prestigious journal, as long as the study design isn’t outrageously bad (for a very liberal definition of “outrageous”), they are going to swamp said journals with hundreds of tiny, null-finding studies, because those are cheap and easy to perform.
Once that happens, you get the GMO industry meme, “hundreds and hundreds of studies that show that GMOs are no different than normal food.”
So the Trust index of researchers in the GMO industry should be measured by how many *POSITIVE* studies they publish, but by how many negative studies they publish.
And there are virtually zero positive studies (that is, that find a difference between GMOs and non-GMOs) in the entire literature. No study on animal toxicity and GMOs has ever been published that reports a significant difference between GMOs and non-GMOs.
But the media, and the major scientific bodies ignore this fact.
In fact, you can find Letters to the Editor signed by just about every head of every major scientific body in the world that assert that Golden Rice was ready for farmers to start growing in 2000 and it is only due to excessive regulation that it hasn’t been on the market since 2000, which ignores the fact that the Golden Rice produced in 2000 had 1/23 the beta carotene content of the current version, and the estimate is that the current version of GR requires that a child eat at least 1 bowl a day to get 60% of the MDR for Vitamin A. According to the signatories of the afore-mentioned Open Letter, this meant that had farmers started growing Golden Rice in 2000 as the collective scientific community of the world asserts, that children would feasibly be eating 23 bowls of rice a day and that farmers would have been willing to grow a food that was the wrong color that would require 23x as much consumption in order to justify eating it in the first place.
Not in the realm of your original article, but it DOES show that hype and deception are rampant in Science, no matter which field, no matter how prominent and “trustworthy” the scientist.
Singling out CAM is politically correct, but deceptive. Everyone indulges in bad science if it suits them and they can get away with it.
What about if you have a -97 score?
You write as you did not read what he wrote..
Anyhow, I m very curious to read Dr. Ernst comment about this.
The role of biased researchers, journals, and CAM schools makes me somewhat compassionate for some CAM practitioners. Often, CAM practitioners are the biggest victims of CAM fraud. They have put their whole life energy, and often more money than they’ll ever make by practicing, into learning, promoting, and using their chosen CAM therapy. Many of these people had nothing but good intentions when they started.
I suspect that only a small percentage of CAM practitioners actually bothers to read research in even pro-CAM journals. I know a much smaller percentage will read critical books, studies, or blogs. It is entirely possible to be in the CAM world and only read supportive “research” and go to supportive CEU classes, etc. Many CAM artists feel like they are “fighting the good fight” and that if there is not good science supporting their modalities, it is due to a Big Pharma/AMA/FDA conspiracy.
It takes a big commitment to increasing one’s scientific literacy and questioning what one was taught in CAM school to see the problems with CAM. There are still only a handful of good books out there explaining the differences between medical science and medical pseudoscience. And, unfortunately, it is easy for a True Believer to reject or ignore such literature. Ad Hominem attacks are only unacceptable if one is interested in being rational and logical. Thus, CAM critics such as Dr. Ernst or Dr. Barrett are often slurred by the CAM apologists while their actual arguments and data are simply ignored.
I hope another part in this series is CAM schools (and the government-sponsored student loans which enable CAM students to go into significant debt to attend them).
It is easy to point at large groups of CAM practitioners and proclaim “quack!” And it may be technically correct. But many of them still don’t *know* they are quacks, and most of them are also victims to some degree. The similarity with religious cults is significant.