I have said it so often that I hesitate to state it again: an uncritical researcher is a contradiction in terms. This begs the question as to how critical the researchers of alternative medicine truly are. In my experience, most tend to be uncritical in the extreme. But how would one go about providing evidence for this view? In a previous blog-post, I have suggested a fairly simple method: to calculate an index of negative conclusions drawn in the articles published by a specific researcher. This is what I wrote:
If we calculated the percentage of a researcher’s papers arriving at positive conclusions and divided this by the percentage of his papers drawing negative conclusions, we might have a useful measure. A realistic example might be the case of a clinical researcher who has published a total of 100 original articles. If 50% had positive and 50% negative conclusions about the efficacy of the therapy tested, his trustworthiness index (TI) would be 1.
Depending on what area of clinical medicine this person is working in, 1 might be a figure that is just about acceptable in terms of the trustworthiness of the author. If the TI goes beyond 1, we might get concerned; if it reaches 4 or more, we should get worried.
An example would be a researcher who has published 100 papers of which 80 are positive and 20 arrive at negative conclusions. His TI would consequently amount to 4. Most of us equipped with a healthy scepticism would consider this figure highly suspect.
So how would alternative medicine researchers do, if we applied this method for assessing their trustworthiness? Very poorly, I fear – but that is speculation! Let’s see some data. Let’s look at one prominent alternative medicine researcher and see. As an example, I have chosen Professor George Lewith (because his name is unique which avoids confusion with researchers), did a quick Medline search to identify the abstracts of his articles on alternative medicine, and extracted the crucial sentence from the conclusions of the most recent ones:
- The study design of registered TCM trials has improved in estimating sample size, use of blinding and placebos
- Real treatment was significantly different from sham demonstrating a moderate specific effect of PKP
- These findings highlight the importance of helping patients develop coherent illness representations about their LBP before trying to engage them in treatment-decisions, uptake, or adherence
- Existing theories of how context influences health outcomes could be expanded to better reflect the psychological components identified here, such as hope, desire, optimism and open-mindedness
- …mainstream science has moved on from the intellectual sterility and ad hominem attacks that characterise the sceptics’ movement
- Trustworthy and appropriate information about practitioners (e.g. from professional regulatory bodies) could empower patients to make confident choices when seeking individual complementary practitioners to consult
- Comparative effectiveness research is an emerging field and its development and impact must be reflected in future research strategies within complementary and integrative medicine
- The I-CAM-Q has low face validity and low acceptability, and is likely to produce biased estimates of CAM use if applied in England, Romania, Italy, The Netherlands or Spain
- Our main finding was of beta power decreases in primary somatosensory cortex and SFG, which opens up a line of future investigation regarding whether this contributes toward an underlying mechanism of acupuncture.
- …physiotherapy was appraised more negatively in the National Health Service than the private sector but osteopathy was appraised similarly within both health-care sectors
This is a bit tedious, I agree, so I stop after just 10 articles. But even this short list does clearly indicate the absence of negative conclusions. In fact, I see none at all – arguably a few neutral ones, but nothing negative. All is positive in the realm of alternative medicine research then? In case you don’t agree with that assumption, you might prefer to postulate that this particular alternative medicine researcher somehow avoids negative conclusions. And if you believe that, you are not far from considering that we are being misinformed.
Alternative medicine is not really a field where one might reasonably expect that rigorous research generates nothing but positive results; even to expect 50 or 40% of such findings would be quite optimistic. It follows, I think, that if researchers only find positives, something must be amiss. I have recently demonstrated that the most active research homeopathic group (Professor Witt from the Charite in Berlin) has published nothing but positive findings; even if the results were not quite positive, they managed to formulate a positive conclusion. Does anyone doubt that this amounts to misinformation?
So, I do have produced at least some tentative evidence for my suspicion that some alternative medicine researchers misinform us. But how precisely do they do it? I can think of several methods for avoiding publishing a negative result or conclusion, and I fear that all of them are popular with alternative medicine researchers:
- design the study in such a way that it cannot possibly give a negative result
- manipulate the data
- be inventive when it comes to statistics
- home in on to the one positive aspect your generally negative data might show
- do not write up your study; like this nobody will ever see your negative results
And why do they do it? My impression is that they use science not for testing their interventions but for proving them. Critical thinking is a skill that alternative medicine researchers do not seem to cultivate. Often they manage to hide this fact quite cleverly and for good reasons: no respectable funding body would give money for such an abuse of science! Nevertheless, the end-result is plain to see: no negative conclusions are being published!
There are at least two further implications of the fact that alternative medicine researchers misinform the public. The first concerns the academic centres in which these researchers are organised. If a prestigious university accommodates a research unit of alternative medicine, it gives considerable credence to alternative medicine itself. If the research that comes out of the unit is promotional pseudo-science, the result, in my view, amounts to misleading the public about the value of alternative medicine.
The second implication relates to the journals in which researchers of alternative medicine prefer to publish their articles. Today, there are several hundred journals specialised in alternative medicine. We have shown over and over again that these journals publish next to nothing in terms of negative results. In my view, this too amounts to systematic misinformation.
My conclusion from all this is depressing: the type of research that currently dominates alternative medicine is, in fact, pseudo-research aimed not at rigorously falsifying hypotheses but at promoting bogus treatments. In other words alternative medicine researchers crucially contribute to the ‘sea of misinformation’ in this area.
Quite a few people seem to be amazed about the amount of work and the enthusiasm I invest into this blog. To be honest, I am amazed too. Why do I do it? And why do I do it in the way I do it?
I am sure that my critics have a choice of answers at the ready; in one way or another, the following theories have been put forward:
- I was bought by the pharmaceutical industry.
- I have had a bad experience with one or several alternative practitioners.
- I want to get rich and/or famous.
- I feel that conventional medicine is flawless
- I don’t know what I am writing about.
None of these notions is remotely true, of course.
Big Pharma has never approached me, and so far nobody at all has ever tried to influence the contents of my posts. I never had a bad personal experience with alternative medicine; on the contrary, I had several quite positive ones. The blog does not earn me money; it even costs a little to run it, and nobody other than I pay for it. I do not think that fame is something one can achieve through doing a blog, and I certainly do not aim at doing so. Sadly, I know only too well about the many flaws in conventional medicine; it is often far from perfect but at least it makes progress; alternative medicine, by contrast, seems almost entirely static. As I have studied most aspects of alternative medicine for two decades and published more than any other person on this subject, I think the allegation of incompetence might be a bit far-fetched.
So, why then?
When I decided to become a doctor I, like most medical students, did so mainly to help suffering individuals. When I became a researcher, I felt more removed from this original ideal. Yet I told myself that, by conducting research, I might eventually contribute to a better health care of tomorrow. Helping suffering patients was still firmly on the agenda. But then I realised that my articles in peer-reviewed medical journals somehow missed an important target: in alternative medicine, one ought to speak not just to health care professionals but also to consumers and patients; after all, it is they who often make the therapeutic decisions in this area.
Once I had realised this, I started addressing the general public by writing for The Guardian and other newspapers, giving public lectures and publishing books for a lay audience, like TRICK OR TREATMENT…The more I did this sort of thing, the more I noticed how important this activity was. And when a friend offered to help me set up a blog, I did not hesitate for long.
So, the reason for my enthusiasm for this blog turns out to be the same as the one that enticed me to go into medicine in the first place. I do believe that it is helpful for consumers to know the truth about alternative medicine. Considering the thousands of sources of daily misinformation in this area, there is an urgent need for well-informed, critical information. By providing it, I am sure I can assist people to make better therapeutic decisions. In a way, I am back where I started all those years ago: hoping to help suffering patients in the most direct way my expertise allows.
One of the most gratifying aspect of my work in Exeter was being able to offer posts to visiting researchers from across the world. Some of these co-workers, after returning to their home countries, became prominent scientists in their own right, and quite a few remained in contact and continued to collaborate with me or with members of my team. In one of these collaborative projects, we wanted to investigate adverse events attributed to traditional medical treatments in the Republic of Korea.
For this purpose, we reviewed adverse events recorded in the Republic of Korea, between 1999 and 2010, by the Food and Drug Administration, the Consumer Agency or the Association of Traditional Korean Medicine. Records of adverse events attributed to the use of traditional medical practices, including reports of medicinal accidents and consumers’ complaints, were evaluated.
Overall, 9624 records of adverse events were identified. Liver problems after the administration of herbal medicines were the most frequently reported adverse events. Only eight of the adverse events were recorded by the pharmacovigilance system run by the Food and Drug Administration. Of the 9624 events, 1389 – mostly infections, cases of pneumothorax and burns – were linked to physical therapies (n = 285) or acupuncture/moxibustion (n = 1104).
We concluded that traditional medical practices often appear to have adverse effects, yet almost all of the adverse events attributed to such practices between 1999 and 2010 were missed by the national pharmacovigilance system. The Consumer Agency and the Association of Traditional Korean Medicine should be included in the national pharmacovigilance system.
The assumption that alternative treatments are entirely harmless is widespread, not least because it is incessantly promoted via millions of web-site, thousands of books, newspaper articles, VIPs like Prince Charles etc. etc. Consumers are incessantly being told that NATURAL = SAFE. Yet, if we look closely, most alternative treatments are not natural and, as this investigation demonstrates, they are certainly not devoid of risks.
I already see the apologists preparing to comment that, compared to conventional therapies, alternative treatments are very safe. So let me pre-empt this fallacy by pointing out (yet again) that 1) in the absence of adequate surveillance systems, nobody can say how frequent adverse-effects of alternative treatments really are, and that 2) even severe adverse effects can normally be tolerated, if the treatment in question has been shown to be efficacious.
So, instead of commenting on my repeated reports about the risks of alternative medicine, I invite, in fact, I challenge my critics to answer this simple question: For how many alternative therapies is there a well-documented positive risk/benefit balance?
If one spends a lot of time, as I presently do, sorting out old files, books, journals etc., one is bound to come across plenty of weird and unusual things. I for one, am slow at making progress with this task, mainly because I often start reading the material that is in front of me. It was one of those occasions that I had begun studying a book written by one of the more fanatic proponent of alternative medicine and stumbled over the term THE PROOF OF EXPERIENCE. It made me think, and I began to realise that the notion behind these four words is quite characteristic of the field of alternative health care.
When I studied medicine, in the 1970s, we were told by our peers what to do, which treatments worked for which conditions and why. They had all the experience and we, by definition, had none. Experience seemed synonymous with proof. Nobody dared to doubt the word of ‘the boss’. We were educated, I now realise, in the age of EMINENCE-BASED MEDICINE.
All of this gradually changed when the concepts of EVIDENCE-BASED MEDICINE became appreciated and generally adopted by responsible health care professionals. If now the woman or man on top of the medical ‘pecking order’ claims something that is doubtful in view of the published evidence, it is possible (sometimes even desirable) to say so – no matter how junior the doubter happened to be. As a result, medicine has thus changed for ever: progress is no longer made funeral by funeral [of the bosses] but new evidence is much more swiftly translated into clinical practice.
Don’t get me wrong, EVIDENCE-BASED MEDICINE does not does not imply disrespect EXPERIENCE; it merely takes it for what it is. And when EVIDENCE and EXPERIENCE fail to agree with each other, we have to take a deep breath, think hard and try to do something about it. Depending on the specific situation, this might involve further study or at least an acknowledgement of a degree of uncertainty. The tension between EXPERIENCE and EVIDENCE often is the impetus for making progress. The winner in this often complex story is the patient: she will receive a therapy which, according to the best available EVIDENCE and careful consideration of the EXPERIENCE, is best for her.
NOT SO IN ALTERNATIVE MEDICINE!!! Here EXPERIENCE still trumps EVIDENCE any time, and there is no need for acknowledging uncertainty: EXPERIENCE = proof!!!
In case you think I am exaggerating, I recommend thumbing through a few books on the subject. As I already stated, I have done this quite a bit in recent months, and I can assure you that there is very little evidence in these volumes to suggest that data, research, science, etc.. matter a hoot. No critical thinking is required, as long as we have EXPERIENCE on our side!
‘THE PROOF OF EXPERIENCE’ is still a motto that seems to be everywhere in alternative medicine. In many ways, it seems to me, this motto symbolises much of what is wrong with alternative medicine and the mind-set of its proponents. Often, the EXPERIENCE is in sharp contrast to the EVIDENCE. But this little detail does not seem to irritate anyone. Apologists of alternative medicine stubbornly ignore such contradictions. In the rare case where they do comment at all, the gist of their response normally is that EXPERIENCE is much more relevant than EVIDENCE. After all, EXPERIENCE is based on hundreds of years and thousands of ‘real-life’ cases, while EVIDENCE is artificial and based on just a few patients.
As far as I can see, nobody in alternative medicine pays more than a lip service to the fact that EXPERIENCE can be [and often is] grossly misleading. Little or no acknowledgement exists of the fact that, in clinical routine, there are simply far too many factors that interfere with our memories, impressions, observations and conclusions. If a patient gets better after receiving a therapy, she might have improved for a dozen reasons which are unrelated to the treatment per se. And if a patient does not get better, she might not come back at all, and the practitioner’s memory will therefore fail register such events as therapeutic failures. Whatever EXPERIENCE is, in health care, it rarely constitutes proof!
The notion of THE PROOF OF EXPERIENCE, it thus turns out, is little more than self-serving, wishful thinking which characterises the backward attitude that seems to be so remarkably prevalent in alternative medicine. No tension between EXPERIENCE and EVIDENCE is noticeable because the EVIDENCE is being ignored; as a result, there is no progress. The looser is, of course, the patient: she will receive a treatment based on criteria which are less than reliable.
Isn’t it time to burry the fallacy of THE PROOF OF EXPERIENCE once and for all?
Why another blog offering critical analyses of the weird and wonderful stuff that is going on in the world of alternative medicine? The answer is simple: compared to the plethora of uncritical misinformation on this topic, the few blogs that do try to convey more reflected, sceptical views are much needed; and the more we have of them, the better.
But my blog is not going to provide just another critique of alternative medicine; it is going to be different, I hope. The reasons for this are fairly obvious: I have researched alternative medicine for two decades. My team and I have conducted about 40 clinical trials and published more than 100 systematic reviews of alternative medicine. We were by far the most productive research unit in this area. For 14 years, we hosted an annual international conference for researchers in this field. I know many of the leading investigators personally, and I understand their way of thinking. I have rehearsed every possible argument for or against alternative medicine dozens of times.
In a nutshell, I am not someone who judges alternative medicine from the outside; I come from within the field. Arguably, I am the only researcher in this area who is willing [or capable?] to state publicly what is wrong with alternative medicine. This is perhaps one of the advantages of being retired and writing a blog in an entirely private capacity.
People who have criticised this or that alternative therapy without first-hand experience of it have always been dismissed by believers as ill-informed; the argument usually is “this guy does not know what he is talking about”. Thus criticism from the outside was hardly ever taken seriously by those who needed it most. Yet it would be difficult to dismiss my arguments on such grounds: I can demonstrate that I have first-hand experience and know what I am talking about. I am clearly not an outsider.
People who criticise alternative medicine tend to claim that all of it is unscientific rubbish which we should discard. However, I am not convinced that this opinion is correct. I aim to adhere to the principles of evidence-based medicine and know that they can be applied to alternative medicine as much as to any other area of healthcare. This means that I will not dismiss everything that comes under the umbrella of alternative medicine. Our research has shown some treatments to work for some conditions, and where this is the case, I will always say so.
What follows is, I think, quite simple: this blog will differ from other blogs on the subject. It will provide critical evaluation because, in my view (and here I will express my views, not those of my Uni or anyone else), this is what is needed. But it will not engage in wholesale alternative medicine-bashing. Most importantly, it will provide comments and perspectives that are based on many years of conducting and publishing research in this area.
Since first writing these lines, it has occurred to me that it might be nice to welcome a few guest-bloggers to express their opinions. Anyone who feels like contributing should therefore contact me, and we will see what we can work out.
Before we start discussing some of the the issues around alternative medicine, let me establish a few ground rules for the debates on this blog. I do like clearly expressed views and intend to be as outspoken as politeness allows. I hope that commentators will do the same, no matter whether they agree or disagree with me. Yet a few, simple, principles should be observed by everyone commenting on my blog.
Libellous statements are not allowed.
Comments must be on topic.
Nothing published here should be taken as medical advice.
All my statements are made in a private capacity and are comments in a legal sense.
Conflicts of interest should always be disclosed.
I will take the liberty of stopping the discussion on any particular topic, if I feel that enough has been said and things are getting boring or repetitive.