As I am drafting this post, I am in a plane flying back from Finland. The in-flight meal reminded me of the fact that no food is so delicious that it cannot be spoilt by the addition of too many capers. In turn, this made me think about the paper I happened to be reading at the time, and I arrived at the following theory: no trial design is so rigorous that it cannot to be turned into something utterly nonsensical by the addition of a few amateur researchers.
The paper I was reading when this idea occurred to me was a randomised, triple-blind, placebo-controlled cross-over trial of homeopathy. Sounds rigorous and top quality? Yes, but wait!
Essentially, the authors recruited 86 volunteers who all claimed to be suffering from “mental fatigue” and treated them with Kali-Phos 6X or placebo for one week (X-potencies signify dilution steps of 1: 10, and 6X therefore means that the salt had been diluted 1: 1000000 ). Subsequently, the volunteers were crossed-over to receive the other treatment for one week.
The results failed to show that the homeopathic medication had any effect (not even homeopaths can be surprised about this!). The authors concluded that Kali-Phos was not effective but cautioned that, because of the possibility of a type-2-error, they might have missed an effect which, in truth, does exist.
In my view, this article provides an almost classic example of how time, money and other resources can be wasted in a pretence of conducting reasonable research. As we all know, clinical trials usually are for testing hypotheses. But what is the hypothesis tested here?
According to the authors, the aim was to “assess the effectiveness of Kali-Phos 6X for attention problems associated with mental fatigue”. In other words, their hyposesis was that this remedy is effective for treating the symptom of mental fatigue. This notion, I would claim, is not a scientific hypothesis, it is a foolish conjecture!
Arguably any hypothesis about the effectiveness of a highly diluted homeopathic remedy is mere wishful thinking. But, if there were at least some promissing data, some might conclude that a trial was justified. By way of justification for the RCT in question, the authors inform us that one previous trial had suggested an effect; however, this study did not employ just Kali-Phos but a combined homeopathic preparation which contained Kalium-Phos as one of several components. Thus the authors’ “hypothesis” does not even amount to a hunch, not even to a slight incling! To me, it is less than a shot in the dark fired by blind optimists – nobody should be surprised that the bullet failed to hit anything.
It could even be that the investigators themselves dimly realised that something is amiss with the basis of their study; this might be the reason why they called it an “exploratory trial”. But an exploratory study is one whithout a hypothesis, and the trial in question does have a hyposis of sorts – only that it is rubbish. And what exactly did the authos meant to explore anyway?
That self-reported mental fatigue in healthy volunteers is a condition that can be mediatised such that it merits treatment?
That the test they used for quantifying its severity is adequate?
That a homeopathic remedy with virtually no active ingredient generates outcomes which are different from placebo?
That Hahnemann’s teaching of homeopathy was nonsense and can thus be discarded (he would have sharply condemned the approach of treating all volunteers with the same remedy, as it contradicts many of his concepts)?
That funding bodies can be fooled to pay for even the most ridiculous trial?
That ethics-committees might pass applications which are pure nonsense and which are thus unethical?
A scientific hypothesis should be more than a vague hunch; at its simplest, it aims to explain an observation or phenomenon, and it ought to have certain features which many alt med researchers seem to have never heard of. If they test nonsense, the result can only be nonsense.
The issue of conducting research that does not make much sense is far from trivial, particularly as so much (I would say most) of alt med research is of such or even worst calibre (if you do not believe me, please go on Medline and see for yourself how many of the recent articles in the category “complementary alternative medicine” truly contribute to knowledge worth knowing). It would be easy therefore to cite more hypothesis-free trials of homeopathy.
One recent example from Germany will have to suffice: in this trial, the only justification for conducting a full-blown RCT was that the manufacturer of the remedy allegedly knew of a few unpublished case-reports which suggested the treatment to work – and, of course, the results of the RCT eventually showed that it didn’t. Anyone with a background in science might have predicied that outcome – which is why such trials are so deplorably wastefull.
Research-funds are increasingly scarce, and they must not be spent on nonsensical projects! The money and time should be invested more fruitfully elsewhere. Participants of clinical trials give their cooperation willingly; but if they learn that their efforts have been wasted unnecessarily, they might think twice next time they are asked. Thus nonsensical research may have knock-on effects with far-reaching consequences.
Being a researcher is at least as serious a profession as most other occupations; perhaps we should stop allowing total amateurs wasting money while playing at being professioal. If someone driving a car does something seriously wrong, we take away his licence; why is there not a similar mechanism for inadequate researchers, funders, ethics-committees which prevents them doing further damage?
At the very minimum, we should critically evaluate the hypothesis that the applicants for research-funds propose to test. Had someone done this properly in relatiom to the two above-named studies, we would have saved about £150,000 per trial (my estimate). But as it stands, the authors will probably claim that they have produced fascinating findings which urgently need further investigation – and we (normally you and I) will have to spend three times the above-named amount (again, my estimate) to finance a “definitive” trial. Nonsense, I am afraid, tends to beget more nonsense.
52 Responses to Research can be a waste of time, money and volunteers’ cooperation
Leave a Reply to Pete 628 Cancel reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
There is something clearly unethical if not criminal about shoddy government funded “research” such as clinical trials to measure effects of some herbal remedy popular in folklore but never really observed to have significant medicinal effects.
Echinacia and Ginkgo Biloba are two commonly used herbal remedies for various conditions. At the US Government website for the National Center for Complementary and Alternative Medicine (NCCAM) there are 19 and 45 listed clinical trial studies respectively for those herbal products. Most of those clinical trials are completed but few have posted results. Of the sparse results that are posted at that website there appears to be no conclusions.
No conclusions? What kind of research is that? If one digs a bit deeper there appears to be a common refrain that more study is needed since compelling proof of efficacy or lack thereof is not apparent.
To engage in expensive research to prove or disprove an unlikely hypothesis does not seem at all like good science. Even worse, once the money has been spent on such “research” it is unconscionable to not report the experiment’s outcome. Rest assured that the costs of the clinical trials are not reported at the website.
The NCCAM budget for 2005 was $123 million. For fiscal year 2009, it was $122 million. In 2011 it was $127.7 million.
And it gets worse. The National Institute of Health also funds the Office of Cancer Complementary and Alternative Medicine (OCCAM) which is dedicated to the idea of researching novel complementary and alternative treatment programs for cancer. This bureau is funded to the tune of about $120 million per year.
NCCAM and OCCAM together account for about $240 million tax dollars each year and yield very little if any medical advancement. That is a huge loss of research funding that could be better spent by more serious researchers in any discipline. To not spend that money at all would be a superior use of the funds than the NCCAM / OCCAM pseudoscience boondoggles.
See the NCCAM website at: http://nccam.nih.gov/research/clinicaltrials
Cant comment on homeopathy but I can say that chiropractic is doing quality research despite getting a tiny fraction of the funds that is available for medical research. http://www.oscachiro.org/displaycommon.cfm?an=1&subarticlenbr=65
However the situation regarding medical research despite being given billions of dollars is not that great either.
In a recent review of retracted studies in the drug- and biomedical literature, drug studies were found to have the highest number of incidents of scientific misconduct. Nearly 75 percent of retracted drug studies were attributed to scientific misconduct such as data falsification, fabrication, questionable veracity, unethical conduct and/or plagiarism
Scientific fraud is on the rise. According to the most recent data, the number of scientific retractions have climbed more than 15-fold since 2001
According to a former drug company researcher, 47 of 53 landmark cancer studies published in the world’s top journals could not be replicated—a validation process that is essential to establish scientific merit
Despite retractions, fabricated research may still inadvertently live on, as it may have been cited by numerous other studies and once a finding is accepted in the medical community, it’s difficult to flush out. Worse yet, according to a recent study nearly 32 percent of retracted papers were not noted as having been retracted by the journal in question, leaving readers completely in the dark.
interesting points!
evidence for the statements or references would be helpful, i think.
as far as chiro research in the UK is concerned, its quantity declined ever since chiros were regulated by statute. does that imply that regulation is viewed as a level of recognition that renders further research redundant? do chiros see research as a tool to gain recognition? i wonder.
The silence regarding the sad state of affairs regarding medical research is deafening!!
there is no silence on my side: i asked you for evidence and you fell silent!
So are you happy regarding the state of medical research?
i did not say that. research needs constant improvement.
‘as far as chiro research in the UK is concerned, its quantity declined ever since chiros were regulated by statute’……….What nonsense!!! Merely your opinion which is clearly extremely prejudiced.
‘does that imply that regulation is viewed as a level of recognition that renders further research redundant?’…..Of course not! what kind of nonsense is that? Are you feeling OK?
‘do chiros see research as a tool to gain recognition?’……….the reason for research is not necessarily about recognition rather it is about understanding the mechanisms how chiropractic care enhances health and improving function and therefore quality of life.
When you say regognition I guess you mean regulation. I do not believe The General Chiropractic Council has done much for chiropractic in the UK….just my opinion.
what nonsense? i am afraid in you upside down word nonsense is sense and vice versa. here is the article which showed this quite clearly: THE EFFECTS OF STATUTORY REGULATION OF OSTEOPATHY AND CHIROPRACTIC ON RESEARCH ACTIVITY IN THE UK. PHYSICAL THERAPY REVIEWS 2005, 10: 67 [sorry, it does not seem to be available on-line]
Ahem: THE EFFECT OF STATUTORY REGULATION OF OSTEOPATHY AND CHIROPRACTIC ON RESEARCH ACTIVITY IN THE UK
Eugen can read the entire article here:
http://dcscience.net/canter-ernst-2005%20The%20Effect%20of%20Statutory.pdf
Do you really think I take any ‘research’ done by Prof Ernst seriously regarding alt med. I would like to know what has made him so bitter that he now wants to destroy all the alt med therapies especially as he spent more than 30 years in that field. I also wonder why he gave up his German passport (according to Wikipedia)
that’s why you probably will stay ignorant.
you cannot even read wiki properly: i took british citizenship; i did not give up my german passport; no need to look for skeletons in my cellar – there aren’t any.
I’m sure Prof Ernst will gladly explain anything in his research that you don’t understand – that is, if you can find anything specific to question.
Or are you only capable of throwing insults?
is that what you learn in chiro-college? NEVER READ ANYTHING THAT THREATENS YOUR PRECONCEIVED IDEAS OT THAT MIGHT BE CRITICAL OF YOUR TREASURED BELIEFS.
ad hominem?
I obviously got it wrong regarding your your German passport and therefore apologise.
no need to apologise; you were wrong so often before that apologies would clog up this blog.
What do you think is an ad hominem?
Eugen Roth said
So, do you have an alternative ‘opinion’ that you can back up with evidence?
Why is it nonsense? Perhaps that is ‘Merely your opinion which is clearly extremely prejudiced.’?
Cart and horses. First, show that chiropractic can indeed enhance health, then move on to understanding how it does it.
On the contrary. It has given vast amounts of undeserved legitimacy to an industry that has failed to justify itself in any clinically meaningful way.
Why dont you read my first post above and particularly the link to the research which shows chiropractic is in fact extremely effective in having a postive effect on the brain and the autonomic nervous system. I will not respond to the rest of your post as it is nothing more than being argumentative.
I saw your cherry-picked list of trials.
You think that questioning what you said is argumentative? That made me laugh!
So Prof….does improved brain function and Heart Rate Variability improve health and quality of life.
probably not as far as HRV is concerned: correlation is not causation.
You clearly would not acknowledge that they do as that would mean you would have to admit that chiropractic is not quackery and therefore you would not ‘famous’ in the sceptic circles.
so, are you saying that, in the chiro-world of make belief, correlation IS causation?
‘in the chiro-world of make belief’……….another misrepresentation about the chiropractic profession suggesting that there is no scientific basis to chiropractic but just ‘make believe’……The trouble with beliefs or should I say dogma is that one’s opinion becomes unchangeable despite any compelling evidence to the contrary. No evidence is acceptable to change one stance….one becomes immovable. For instance: You claim chiropractic procedures cause arteries in the neck to disintegrate leading to stroke and death. However research shows that chiropractic care places no more strain on the internal carotid artery than normal range of movement in the cervical spine.But you continue relentlessly claiming chiropractic= stroke. Normally a scientist dispassionately weighs up the evidence and comes to a reasonable conclusion. However as that is not your agenda but merely to be a sceptic you will continue to discount any evidence.
Oh by the way: Is correlation causation?…No…and I never said it was……quote: ‘you use every fallacy in the book’.
let me comment on ERs post below (there is no place for it)
1) the chiro world of make belief: one has no shortage of example for this; for instance the claim that chiro if effective for otitis or bed-wetting.
2)the vascular accidents relate to the VERTEBRAL not carotid artery in most cases
3) asking a question, as i did, cannot be a fallacy [i thought you know all about fallacies]
So Prof in a previous post I repeatedly asked you to comment on why 5400 medical doctors in Germany practise ‘chirotherapie’ if indeed it is quackery as you and your mates would like the public to believe. I wondered why you refused to comment and I see why: Because you taught them at the Hanover School of Physical Medicine (according to Wikipedia)That is funny I almost fell off my chair laughing. So you were a homeopath for 30 years and you taught German medical doctors ‘Chirotherapie’. Brilliant! You are truely an expert in so many different fields. Lol….brilliant!
you are like a broken record!
let me repeat my answer: it is your ‘ad populum’ fallacy again; seems to be your favourite.
oh, and i did teach at hannover, but only med students and not chiro. sorry to disappoint you yet again.
So you did not ever teach spinal manipulative therapy? Is that correct?
not in hannover and not to non-medics.
Eugen
This might help you. It’s a beginner’s guide and hardly exhaustive, but it will inform you about logical fallacies and why they ensure you lose an argument: Thou shalt not commit logical fallacies.
If you were capable of understanding that, we might be able to have a better discussion.
By this comment you are trying to create the idea that I am lacking in understanding and not capable of a reasonable discussion. I see this as condescending and an attempt to discredit my comments and make me look stupid……in other words a rather underhand way to win the argument. Maybe you need to brush up on your ethics.
in view of your previous comments and tone, this one is truly funny!!!
Eugen: By your frequent use of fallacious arguments instead of sound, reasoned ones, you have aptly demonstrated to all that you are not very familiar with those fallacies.
And I have not won the argument: you have not presented any reasoned one yet, but have mostly resorted to ad homs. Do you need me to list them for you?
Hi Eugen,
Try reading
http://youarenotsosmart.com/the-book/
Very readable and explains why the brain has evolved to mislead us.
Ye Ian my brain misleads me but not in your case. You must be on the verge of enlightenment.
Prof Ernst said: the vascular accidents relate to the VERTEBRAL not carotid artery in most cases……. however strokes relating from vertebral artery are rare and no evidence exists proving chiropractic is a cause of stroke. 2 in 10 million is no real risk. No procedures have no risk at all and are 100% safe.
.
‘ the chiro world of make belief: one has no shortage of example for this; for instance the claim that chiro if effective for otitis or bed-wetting.’……….yet again I state that chiropractic care ( please note I do not use the word ‘treatment’) does not treat any condition, illness or symptom. Chiropractic care is about improving brain function, nerve transmission, homoeostasis of the autonomic nervous system, etc , general health and physiological function. The result of that maybe that various conditions like otitis media improve. Why cant you get the concept that chiropractic care does not fit into the allopathic model of diagnosing and treating illnesses and symptoms?? Its is a HEALTH CARE SYSTEM and not a DISEASE TREATMENT SYSTEM. I guess the reason why you refuse ‘to get it’ is that if you accept that chiropractic care does not treat illness but improves physiological function your basis to criticise chiropractic is made null and void,
you are still off topic – more importantly, you are yet again incorrect.
1) there is plenty of evidence for upper spinal manipulation to cause strokes; even court cases have been won (i do not cite evidence because you posted elsewhere that you will not read what i produce)
2) your “allopathic model” is clearly a figment of your imagination: if chiro care imroves brain function (or whatever) which, in turn, improves otitis – then this surely can be shown in a rigorous RCT. this would be EBM, and not an “allopathic model”.
THIS IS THE LAST TIME I COMMENT ON YOUR POSTS THAT ARE NOT FOLLOWINF THE GROUND RULES
Regarding the allopathic model point. More than 100 years ago chiropractic made it clear that it did not follow the medical model. Chiropractic said it was ‘seperate and distinct’ as it was not diagnosing and treating disease. Many chiropractors were arrested in the early days and went to jail for ‘practising medicine without a license’ however eventually the judiciary got the idea that I have tried to explain to you. Chiropractic is about improving function, physiology, state of wellbeing and health. If one looks at homeopathy and naturopathy for example they treat disease with ‘non-medical’ procedures. But they are following the same kind of approach to the problem as medical doctors do just using remedies or herbs to TREAT a disease instead of drugs and surgery. If we look at exercise for example I am quite certain you would agree it is not a TREATMENT for any disease but that it has physiological benefits that could lead to various medical conditions to be improved. The same could be said about diet. Gerson is a TREATMENT for cancer and other disease using dieting, juicing, herbs etc. However anybody(without any diseases) could improve their diet and benefit. Their cholestrol HDL/LDL ratio could improve, their cortisol levels could improve, their HRV could improve. For instance patients with cancer who exercise and eat healthily benefit but exercise and diet is not a treatment like chemotherapy and radiation is. Anyway you can get irritable about the model of HEALTH CARE that chiropractors follow, it makes no difference…that is the way it is. Another example that may help to illustrate what I mean…Chinese Medicine is not just about treating diseases and symptoms. For instances ‘tonic herbs’ are given to maintain health. Just like adaptogens (Rhodiola) are given to help the body deal with stress but not as a treatment for any particular disease. Maybe this misunderstanding about what chiropractic really is leads you to your poor opinion of chiropractors. You mentioned RCTs for otitis media. This is trying to measure the effectiveness of something that chiropractic is not…a treatment for otitis media. A better investigation would be to ascertain whether long term chiropractic patients enjoy better health, wellbeing and quality of life. I cannot understand why you find this unacceptable. But I guess I must be deluded living in ‘the chiropractic world of make believe’….(your quote)…..another personal attack to add to ‘paranoia’
Oh another personal attack: ‘i am afraid you are talking out of your proverbial!’………….but it is I that resorts to ad hominems…..MMmmmm
What a silly thing to say, of course an RCT would still be able to show (or more likely disprove) your claims.
ER: if the claim is that chiro generates better health generally, of course, an RCT would be able to test this. where is this type of evidence. but an RCT for otitis is just as justified – according to the concept you mentioned above. if it were not, why would chiros do such trials? they do exist, you know – only that their results are negative.
Hahaha, Edzard Ernst consultation hypothesis defining a reliable source?? No! the “Wikipedia”. And the web socialscience!. This supports my idea that Ernst is not good in methodology (eg in one article supports evidence-based medicine eliminating cohort studies!)
Interestingly one of the authors of the negative items is the same Martin Bland, the statistician who participated in the program Horizon (who seems to have deceived his alleged James Randi replication protocol Madeleine Ennis).
Indeed, this study Bland finance it. Do you have any problem Ernst?
I guess this is like the standard work of physicist David Grimmes junk, who seems to have made a copy/paste of an anti homeopathy blog (in fact Ernst Grimmes thanks for your help). By the way, how is that FACT approved an article of this type? Surely this failing peer review. The article?
I DON’T THINK THAT I EVER ADVOCATED ELIMINATING COHORT STUDIES.
OUT JOURNAL FACT IS INDEED PEER-REVIEWED.
Clearly. wasting money and effort on badly planned studies is a problem.
But also the choice of research field can easily be wrong.
There are of course more or less promising topics available.
A number of studies gets performed trying to “test” concepts that are already disproven with any doubt like homoeopathy and also others.
I wonder, If professionels like you and others do not have enough power, to focus the research in alternative medicine on relevant topics due to influence in committes and as editors.
In my opinion, one possibly relevant topic would be e.g. regular consumption of (locally produced) honey prior to hay fever season.
If one searches in Medline, the first match is of 1957 and there is still not enough data with clear results published.
I find this very surprising regarding the high prevalence of hay fever.
The honey for hay fever myth has been busted. Furthermore, a very common irritant is grass pollen, which bees do not collect.
So you expect the resuluts of Saarinen et al. (DOI:10.1159/000319821) to be due to placebo and/or chance?
Participants in that trial self-reported… Oh well, the conclusions were: “Patients who preseasonally used BPH had significantly better control of their symptoms than did those on conventional medication only, and they had marginally better control compared to those on RH. The results should be regarded as preliminary, but they indicate that BPH could serve as a complementary therapy for birch pollen allergy.”
What the conclusion is really saying: It’s rather pointless adding the allergen (birch pollen) to regular honey for any other purpose than making money from it as a complementary therapy.
The finding that regular honey was significantly better than using only conventional medication is interesting (incorrect, of course, due to the small number of participants and poor trial design) because it raises questions such as: Would sucking boiled sweets or using a soothing linctus be equally effective? Were members of the control group using suitable medication (there were only 17 members of this group)?
Smoking tobacco has been shown to significantly reduce coughing and sneezing in some hay fever sufferers, but I wouldn’t personally recommend it as a complementary therapy.
Oh, I did a mistake:
10.1016/S1081-1206(10)61996-5
Here the effectiveness of the method got disproven.
I have thought, there would only exist the abstract and no further data would be published about the topic sorry.