The announcement was made during the German sceptics conference ‘Skepkon‘ in Koeln. As I could not be present, I obtained the photo via Twitter.
So, if you are a homeopath or a fan of homeopathy, all you have to do – as the above slide says – is to reproducibly identify homeopathic remedies in high potency. The procedure for obtaining the money has to follow three pre-defined steps:
- Identification of three homeopathic preparations in high potency according to a proscribed protocol.
- Documentation of a method enabling a third party to identify the remedies.
- Verification of the experiment by repeating it.
Anyone interested must adhere to the full instructions published by the German sceptics GWUP:
1. Review of test protocol
Together with a representative of GWUP, the applicants review and agree on this protocol prior to the start of the test. Minor changes may be applied if justified, provided they are mutually agreed to in advance and do not affect the validity of the test, especially the blinding and randomization of the samples. In any case we want to avoid that the results get compromised or their credibility impeached by modifications of the protocol while the test is already under way. After mutual confirmation, the test protocol is binding for the whole duration of the test and its evaluation.
2. Selection of drugs
The applicant proposes which three drugs should be used in the trial. This gives them the opportunity to select substances that they think they could distinguish particularly well as homeopathic remedies. The potency may be selected freely as well, whereby the following conditions must be observed:
– all drugs must be available as sugar globules of the same grade (“Globuli” in German);
– the same potency, namely D- or C-potency above D24 / C12, is used for all three drugs;
– all drugs can be procured from the same producer.
3. Procurement of samples
The samples will be purchased by GWUP and shipped from the vendor directly to the notary who will perform the randomization. GWUP will purchase sufficient numbers of packages to complete the series of 12 samples according to the randomization list. The procurement will ensure that the samples derive from different batches of production as follows.
3.1. Common remedies
Common remedies, i.e. remedies sold in high numbers, will be procured from randomly selected pharmacies from the biggest cities in Germany (Berlin, Hamburg, Munich, Cologne, Frankfurt, Stuttgart…). Each pharmacy supplies a bottle for each of the three selected remedies and ships it directly to the notary in charge of randomization. If the applicants need a sample of known content for calibration, then this will be procured from yet another pharmacy in another German city.
3.2. Special remedies
If due to low sales it is possible that the above procedure is not sufficient to have all samples from different batches, a randomly selected pharmacy will be appointed to produce all the samples from raw materials purchased from the producer. GWUP will procure the mother tinctures, the raw sugar pills, and bottles and packages, to be shipped directly to the respective pharmacy who then will do the potentization, label the bottles and send them to the notary. If there are extra samples of known content required for calibration, then an extra set of samples will be produced. One set of samples will be kept in a sealed package for future reference.
The applicant and GWUP mutually agree on which procedure is used before the start of procurement. If more than 10 grams of globules per sample are required for the procedure used for inentification, the applicant has to indicate this in advance, and GWUP will take this into account.
4. Randomization / blinding
The randomization and blinding is done by a sworn-in public notary in Würzburg, Germany, who is selected by a random procedure. Würzburg is chosen because the first part of the task is to be evaluated at the University of Würzburg, for all participants based in Europe. For overseas applicants, the location will be mutually agreed on.
The notary receives a coding list showing how the three drugs A, B and C are to be distributed among the twelve samples. This list is compiled by the GWUP representative by throwing dice. The notary also determines which drug is assigned to which letter by throwing dice. Note that the drugs may not be present in the set in equal numbers.
The notary completely removes the original label from the bottle and replaces it with the number without opening the bottle. The randomization protocol is deposited in a sealed envelope with the notary public without a copy being made beforehand. The notary disposes of surplus packs. If special remedies are processed, one set of marked samples is sealed and forwarded to GWUP for later reference in a sealed package.
The coded bottles are sent from the notary to the applicant without individual packaging and documentation. The applicant confirms receipt of the samples.
The applicant identifies which of the 12 bottles contains which drug, using any method and procedure of his choice. There is no limit as to the method used for identification, and this well may be a procedure not currently recognized by modern science. However, GWUP at the start requires a short and rough outline of how the applicant wants to proceed, and GWUP reserves the right to reject applications whose sincerity for scientific work seems questionable.
The applicant is also required to specify a period of time within which they will be able to produce their results. This period may not exceed six months. If it expires without the applicant being able to show their results, the outcome will be considered negative. However, the candidate may apply for an extension in good time before the deadline, if they can provide a reasonable explanation and is not caused by the respective identification process as such.
The applicant is explicitly advised to observe ethics standards, and to procure the consent of an appropriate ethics committee if their method involves testing on humans or animals.
6. Result Pt. 1
If reasonable, the applicant may present their findings as part of the PSI-Tests held annually by GWUP at the University of Würzburg. The applicant’s result will be compared to the coding protocol from the notary. The number of bottles in which the notary’s record corresponds to the applicant’s details is determined. The result includes a description of the method used, if possible with meaningful intermediate data such as measurement protocols or symptom lists of drug provings.
The first part of the test is considered a success if the content of no more than one bottle is identified incorrectly and a description of the procedure is produced.
7. Result Pt. 2 and 3: Replication and Verification
Replication of the test is to ensure that a successful first result was not caused by chance alone. In addition, the procedure explained by the applicant is to be verified in a way depending on its nature. The objective is to verify that the identification was indeed performed by using this very method, and that the description is complete and suitable for a third party to achieve the same outcome.
For replication, steps 2 to 5 will be repeated. Applicants may select to use the same drugs as before. In this case they will then procured from another manufacturer or prepared by a different pharmacy with raw material from a different supplier. Alternatively, the candidate may indicate three new drugs which then can be obtained from the original vendor.
For a successful replication the same precision as before is required, that is, that only one out of 12 bottles may be identified incorrectly.
The evaluation and presentation of these results may take place at any location, press or other media may be invited to the event as agreed to by the applicant and GWUP.
Is anyone going to take up this challenge?
Personally, I don’t hold my breath.
Many years ago (at a time when homeopaths still saw me as one of their own), I had plans to do a similar but slightly less rigorous test as part of a doctoral thesis for one of my students.
Our investigation was straight forward: we approached several of the world’s leading/most famous homeopaths and asked them to participate. Their task was to tell us which homeopathic remedy they thought was easiest to differentiate from a placebo. Subsequently we would post them several vials – I think the number was 10 – and ask them to tell us which contained the remedy of their choice (in a C30 potency), and which the placebo (the distribution was 50:50, and the authenticity of each vial was to be confirmed by a notary). The experimental method for identifying which was which was entirely left to each participating homeopath; they were even allowed to use multiple, different tests. Based on the results, we would then calculate whether their identification skills were better than pure chance.
Sadly, the trial never happened. Initially, we had a positive response from some homeopaths who were interested in participating. However, when they then saw the exact protocol, they all pulled out.
But times may have changed; perhaps today there are some homeopaths out there who actually believe in homeopathy?
Perhaps our strategy to work only with ‘the best’ homeopaths was wrong?
Perhaps there are some homeopaths who are less risk-adverse?
I sure hope that lots of enthusiastic homeopaths will take up this challenge.
GOOD LUCK! And watch this space.
We recently discussed the deplorable case of Larry Nassar and the fact that the ‘American Osteopathic Association’ stated that intravaginal manipulations are indeed an approved osteopathic treatment. At the time, I thought this was a shocking claim. So, imagine my surprise when I was alerted to a German trial of osteopathic intravaginal manipulations.
Here is the full and unaltered abstract of the study:
Introduction: 50 to 80% of pregnant women suffer from low back pain (LBP) or pelvic pain (Sabino und Grauer, 2008). There is evidence for the effectiveness of manual therapy like osteopathy, chiropractic and physiotherapy in pregnant women with LBP or pelvic pain (Liccardione et al., 2010). Anatomical, functional and neural connections support the relationship between intrapelvic dysfunctions and lumbar and pelvic pain (Kanakaris et al., 2011). Strain, pressure and stretch of visceral and parietal peritoneum, bladder, urethra, rectum and fascial tissue can result in pain and secondary in muscle spasm. Visceral mobility, especially of the uterus and rectum, can induce tension on the inferior hypogastric plexus, which may influence its function. Thus, stretching the broad ligament of the uterus and the intrapelvic fascia tissue during pregnancy can reinforce the influence of the inferior hypogastric plexus. Based on above facts an additional intravaginal treatment seems to be a considerable approach in the treatment of low back pain in pregnant women.
Objective: The purpose of this study was to compare the effect of osteopathic treatment including intravaginal techniques versus osteopathic treatment only in females with pregnancy-related low back pain.
Methods: Design: The study was performed as a randomized controlled trial. The participants were randomized by drawing lots, either into the intervention group including osteopathic and additional intravaginal treatment (IV) or a control group with osteopathic treatment only (OI). Setting: Medical practice in south of Germany.
Participants 46 patients were recruited between the 30th and 36th week of pregnancy suffering from low back pain.
Intervention Both groups received three treatments within a period of three weeks. Both groups were treated with visceral, mobilization, and myofascial techniques in the cervical, thoracic and lumbar spine, the pelvic and the abdominal region (American Osteopathic Association Guidelines, 2010). The IV group received an additional treatment with intravaginal techniques in supine position. This included myofascial techniques of the M. levator ani and the internal obturator muscles, the vaginal tissue, the pubovesical and uterosacral ligaments as well as the inferior hypogastric plexus.
Main outcome measures As primary outcome the back pain intensity was measured by Visual Analogue Scale (VAS). Secondary outcome was the disability index assessed by Oswestry-Low-Back-Pain-Disability-Index (ODI), and Pregnancy-Mobility-Index (PMI).
Results: 46 participants were randomly assigned into the intervention group (IV; n = 23; age: 29.0 ±4.8 years; height: 170.1 ±5.8 cm; weight: 64.2 ±10.3 kg; BMI: 21.9 ±2.6 kg/m2) and the control group (OI; n = 23; age: 32.0 ±3.9 years; height: 168.1 ±3.5 cm; weight: 62.3 ±7.9 kg; BMI: 22.1 ±3.2 kg/m2). Data from 42 patients were included in the final analyses (IV: n=20; OI: n=22), whereas four patients dropped out due to general pregnancy complications. Back pain intensity (VAS) changed significantly in both groups: in the intervention group (IV) from 59.8 ±14.8 to 19.6 ±8.4 (p<0.05) and in the control group (OI) from 57.4 ±11.3 to 24.7 ±12.8. The difference between groups of 7.5 (95%CI: -16.3 to 1.3) failed to demonstrate statistical significance (p=0.93). Pregnancy-Mobility-Index (PMI) changed significantly in both groups, too. IV group: from 33.4 ±8.9 to 29.6 ±6.6 (p<0.05), control group (OI): from 36.3 ±5.2 to 29.7 ±6.8. The difference between groups of 2.6 (95%CI: -5.9 to 0.6) was not statistically significant (p=0.109). Oswestry-Low-Back-Pain-Disability-Index (ODI) changed significantly in the intervention group (IV) from 15.1 ±7.8 to 9.2 ±3.6 (p<0.05) and also significantly in the control group (OI) from 13.8 ±4.9 to 9.2 ±3.0. Between-groups difference of 1.3 (95%CI: -1.5 to 4.1) was not statistically significant (p=0.357).
Conclusions: In this sample a series of osteopathic treatments showed significant effects in reducing pain and increasing the lumbar range of motion in pregnant women with low back pain. Both groups attained clinically significant improvement in functional disability, activity and quality of life. Furthermore, no benefit of additional intravaginal treatment was observed.
END OF QUOTE
My first thoughts after reading this were: how on earth did the investigators get this past an ethics committee? It cannot be ethical, in my view, to allow osteopaths (in Germany, they have no relevant training to speak of) to manipulate women intravaginally. How deluded must an osteopath be to plan and conduct such a trial? What were the patients told before giving informed consent? Surely not the truth!
My second thoughts were about the scientific validity of this study: the hypothesis which this trial claims to be testing is a far-fetched extrapolation, to put it mildly; in fact, it is not a hypothesis, it’s a very daft idea. The control-intervention is inadequate in that it cannot control for the (probably large) placebo effects of intravaginal manipulations. The observed outcomes are based on within-group comparisons and are therefore most likely unrelated to the treatments applied. The conclusion is as barmy as it gets; a proper conclusion should clearly and openly state that the results did not show any effects of the intravaginal manipulations.
In summary, this is a breathtakingly idiotic trial, and everyone involved in it (ethics committee, funding body, investigators, statistician, reviewers, journal editor) should be deeply ashamed and apologise to the poor women who were abused in a most deplorable fashion.
Reflexology is an alternative therapy that is subjectively pleasant and objectively popular; it has been the subject on this blog before (see also here and here). Reflexologists assume that certain zones on the sole of our feet correspond to certain organs, and that their manual treatment can influence the function of these organs. Thus reflexology is advocated for all sorts of conditions, including infant colic.
The aim of this new study was to explore the effect of reflexology on infantile colic.
A total of 64 babies with colic were included in this study. Following a paediatrician’s diagnosis, two groups (study and control) were created. Socio-demographic data (including mother’s age, educational status, and smoking habits of parents) and medical history of the baby (including gender, birth weight, mode of delivery, time of the onset breastfeeding after birth, and nutrition style) were collected. The Infant Colic Scale (ICS) was used to estimate the colic severity in the infants. Reflexology was applied to the study group by the researcher and their mother 2 days a week for 3 weeks. The babies in the control group did not receive reflexology. Assessments were performed before and after the intervention in both groups.
The results show that the two groups were similar regarding socio-demographic background and medical history. While there was no difference between the groups in ICS scores before application of reflexology, the mean ICS score of the study group was significantly lower than that of control group at the end of the intervention.
The authors concluded that reflexology application for babies suffering from infantile colic may be a promising method to alleviate colic severity.
The authors seem to attribute the outcome to specific effects of reflexology.
However, they are mistaken!
Because their study does not control for the non-specific effects of the intervention.
Reflexology has not been shown to work for anything (“the best clinical evidence does not demonstrate convincingly reflexology to be an effective treatment for any medical condition“), and there is plenty of evidence to show that holding the baby, massaging it, cuddling it, rocking it or doing just about anything with it will have an effect, e. g.:
I think, in a way, this is rather good news; we do not need to believe in the hocus-pocus of reflexology in order to help our crying infants.
Have you ever wondered whether doctors who practice homeopathy are different from those who don’t.
Silly question, of course they are! But how do they differ?
Having practised homeopathy myself during my very early days as a physician, I have often thought about this issue. My personal (and not very flattering) impressions were noted in my memoir where I describe my experience working in a German homeopathic hospital:
… some of my colleagues used homeopathy and other alternative approaches because they could not quite cope with the often exceedingly high demands of conventional medicine. It is almost understandable that, if a physician was having trouble comprehending the multifactorial causes and mechanisms of disease and illness, or for one reason or another could not master the equally complex process of reaching a diagnosis or finding an effective therapy, it might be tempting instead to employ notions such as dowsing, homeopathy or acupuncture, whose theoretical basis, unsullied by the inconvenient absolutes of science, was immeasurably more easy to grasp.
Some of my colleagues in the homeopathic hospital were clearly not cut out to be “real” doctors. Even a very junior doctor like me could not help noticing this somewhat embarrassing fact…
But this is anecdote and not evidence!
So, where is the evidence?
It was published last week and made headlines in many UK daily papers.
Our study was aimed at finding out whether English GP practices that prescribe any homeopathic preparations might differ in their prescribing of other drugs. We identified practices that made any homeopathy prescriptions over six months of data. We measured associations with four prescribing and two practice quality indicators using multivariable logistic regression.
Only 8.5% of practices (644) prescribed homeopathy between December 2016 and May 2017. Practices in the worst-scoring quartile for a composite measure of prescribing quality were 2.1 times more likely to prescribe homeopathy than those in the best category. Aggregate savings from the subset of these measures where a cost saving could be calculated were also strongly associated. Of practices spending the most on medicines identified as ‘low value’ by NHS England, 12.8% prescribed homeopathy, compared to 3.9% for lowest spenders. Of practices in the worst category for aggregated price-per-unit cost savings, 12.7% prescribed homeopathy, compared to 3.5% in the best category. Practice quality outcomes framework scores and patient recommendation rates were not associated with prescribing homeopathy.
We concluded that even infrequent homeopathy prescribing is strongly associated with poor performance on a range of prescribing quality measures, but not with overall patient recommendation or quality outcomes framework score. The association is unlikely to be a direct causal relationship, but may reflect underlying practice features, such as the extent of respect for evidence-based practice, or poorer stewardship of the prescribing budget.
Since our study was reported in almost all of the UK newspapers, it comes as no surprise that, in the interest of ‘journalistic balance’, homeopaths were invited to give their ‘expert’ opinions on our work.
Margaret Wyllie, head of the British Homeopathic Association, was quoted commenting: “This is another example of how real patient experience and health outcomes are so often discounted, when in actuality they should be the primary driver for research to improve our NHS services. This study provides no useful evidence about homeopathy, or about prescribing, and gives absolutely no data that can improve the health of people in the UK.”
The Faculty of Homeopathy was equally unhappy about our study and stated: “The study did not include any measures of patient outcomes, so it doesn’t tell us how the use of homeopathy in English general practice correlates with patients doing well or badly, nor with how many drugs they use.”
Cristal Summer from the Society of Homeopathy said that our research was just a rubbish bit of a study.
Peter Fisher, the Queen’s homeopath and the president of the Faculty of Homeopathy, stated: “We don’t know if these measures correlate with what matters to patients – whether they get better and have side-effects.”
A study aimed at determining whether GP practices that prescribe homeopathic preparations differ in their prescribing habits from those that do not prescribe homeopathics can hardly address these questions, Peter. A test of washing machines can hardly tell us much about the punctuality of trains. And an investigation into the risks of bungee jumping will not inform us about the benefits of regular exercise. Call me biased, but to me these comments indicate mainly one thing: HOMEOPATHS SEEM TO HAVE GREAT DIFFICULTIES UNDERSTANDING SCIENTIFIC PAPERS.
I much prefer the witty remarks of Catherine Bennett in yesterday’s Observer: Homeopath-GPs, naturally, have mustered in response and challenge Goldacre’s findings, with a concern for methodology that could easily give the impression that there is some evidential basis for their parallel system, beyond the fact that the Prince of Wales likes it. In fairness to Charles, his upbringing is to blame. But what is the doctors’ excuse?
Amongst all the implausible treatments to be found under the umbrella of ‘alternative medicine’, Reiki might be one of the worst, i. e. least plausible and outright bizarre (see for instance here, here and here). But this has never stopped enthusiasts from playing scientists and conducting some more pseudo-science.
This new study examined the immediate symptom relief from a single reiki or massage session in a hospitalized population at a rural academic medical centre. It was designed as a retrospective analysis of prospectively collected data on demographic, clinical, process, and quality of life for hospitalized patients receiving massage therapy or reiki. Hospitalized patients requesting or referred to the healing arts team received either a massage or reiki session and completed pre- and post-therapy symptom questionnaires. Differences between pre- and post-sessions in pain, nausea, fatigue, anxiety, depression, and overall well-being were recorded using an 11-point Likert scale.
Patients reported symptom relief with both reiki and massage therapy. Reiki improved fatigue and anxiety more than massage. Pain, nausea, depression, and well being changes were not different between reiki and massage encounters. Immediate symptom relief was similar for cancer and non-cancer patients for both reiki and massage therapy and did not vary based on age, gender, length of session, and baseline symptoms.
The authors concluded that reiki and massage clinically provide similar improvements in pain, nausea, fatigue, anxiety, depression, and overall well-being while reiki improved fatigue and anxiety more than massage therapy in a heterogeneous hospitalized patient population. Controlled trials should be considered to validate the data.
Don’t I just adore this little addendum to the conclusions, “controlled trials should be considered to validate the data” ?
The thing is, there is nothing to validate here!
The outcomes are not due to the specific effects of Reiki or massage; they are almost certainly caused by:
- the extra attention,
- the expectation of patients,
- the verbal or non-verbal suggestions of the therapists,
- the regression towards the mean,
- the natural history of the condition,
- the concomitant therapies administered in parallel,
- the placebo effect,
- social desirability.
Such pseudo-research only can only serve one purpose: to mislead (some of) us into thinking that treatments such as Reiki might work.
What journal would be so utterly devoid of critical analysis to publish such unethical nonsense?
Ahh … it’s our old friend the Journal of Alternative and Complementary Medicine
Say no more!
Generally speaking, Cochrane reviews provide the best (most rigorous, transparent and independent) evidence on the effectiveness of medical or surgical interventions. It is therefore important to ask what they tell us about homeopathy. In 2010, I did exactly that and published it as an overview of the current best evidence. At the time, there were 6 relevant Cochrane reviews. They covered the following conditions: cancer, attention-deficit hyperactivity disorder, asthma, dementia, influenza and induction of labour. And their results were clear: they did not show that homeopathic medicines have effects beyond placebo.
Now a further Cochrane review has been published.
Does it change this situation?
This systematic review assessed the effectiveness and safety of oral homeopathic medicinal products compared with placebo or conventional therapy to prevent and treat acute respiratory tract infections (ARTIs) in children. The researchers conducted extensive literature searches, checked references, and contacted study authors to identify additional studies. They included all double-blind, randomised controlled trials (RCTs) or double-blind cluster-RCTs comparing oral homeopathy medicinal products with identical placebo or self selected conventional treatments to prevent or treat ARTIs in children aged 0 to 16 years.
Eight RCTs of 1562 children receiving oral homeopathic medicinal products or a control treatment (placebo or conventional treatment) for upper respiratory tract infections (URTIs). Four treatment studies examined the effect on recovery from URTIs, and four studies investigated the effect on preventing URTIs after one to three months of treatment and followed up for the remainder of the year. Two treatment and two prevention studies involved homeopaths individualising treatment for children. The other studies used predetermined, non-individualised treatments. All studies involved highly diluted homeopathic medicinal products.
Several key limitations to the included studies were identified, in particular methodological inconsistencies and high attrition rates, failure to conduct intention-to-treat analysis, selective reporting, and apparent protocol deviations. The authors deemed three studies to be at high risk of bias in at least one domain, and many had additional domains with unclear risk of bias. Three studies received funding from homeopathy manufacturers; one reported support from a non-government organisation; two received government support; one was co-sponsored by a university; and one did not report funding support.
Methodological inconsistencies and significant clinical and statistical heterogeneity precluded robust quantitative meta-analysis. Only four outcomes were common to more than one study and could be combined for analysis. Odds ratios (OR) were generally small with wide confidence intervals (CI), and the contributing studies found conflicting effects, so there was little certainty that the efficacy of the intervention could be ascertained.
All studies assessed as at low risk of bias showed no benefit from oral homeopathic medicinal products; trials at uncertain and high risk of bias reported beneficial effects. The authors found low-quality evidence that non-individualised homeopathic medicinal products confer little preventive effect on ARTIs (OR 1.14, 95% CI 0.83 to 1.57). They also found low-quality evidence from two individualised prevention studies that homeopathy has little impact on the need for antibiotic usage (N = 369) (OR 0.79, 95% CI 0.35 to 1.76).
The authors also assessed adverse events, hospitalisation rates and length of stay, days off school (or work for parents), and quality of life, but were not able to pool data from any of these secondary outcomes. There is insufficient evidence from two pooled individualised treatment studies (N = 155) to determine the effect of homeopathy on short-term cure (OR 1.31, 95% CI 0.09 to 19.54; very low-quality evidence) and long-term cure rates (OR 1.01, 95% CI 0.10 to 9.96; very low-quality evidence). Adverse events were reported inconsistently; however, serious events were not reported. One study found an increase in the occurrence of non-severe adverse events in the treatment group.
The authors concluded that pooling of two prevention and two treatment studies did not show any benefit of homeopathic medicinal products compared to placebo on recurrence of ARTI or cure rates in children. We found no evidence to support the efficacy of homeopathic medicinal products for ARTIs in children. Adverse events were poorly reported, so conclusions about safety could not be drawn.
In their paper, the authors state that “there are no established explanatory models for how highly diluted homeopathic medicinal products might work. For this reason, homeopathy remains highly controversial because the key concepts governing this form of medicine are not consistent with the established laws of conventional therapeutics.” In other words, there is no reason why highly diluted homeopathic remedies should work. Yet, remarkably, when asked what conditions responds best to homeopathy, most homeopaths would probably include ARTI of children.
The authors also point out that “The results of this review are consistent with all previous systematic reviews on homeopathy. Funders and study investigators contemplating any further research in this area need to consider whether further research will advance our knowledge, given the uncertain mechanism of action and debate about how the lack of a measurable dose can make them effective. The studies we identified did not use a uniform approach to choosing and measuring outcomes or assigning appropriate time points for outcome measurement. The use of validated symptom scales would facilitate future meta-analyses. It is unclear if there is any benefit from individualised (classical) homeopathy over the use of commercially available products.”
Even though I agree with the authors on most of their views and comment their excellent work, I would be more outspoken regarding the need of further research. In my view, it would be a foolish, wasteful and therefore unethical activity to fund, plan or conduct further research in this area.
Since many months, I have noticed a proliferation of so-called pilot studies of alternative therapies. A pilot study (also called feasibility study) is defined as a small scale preliminary study conducted in order to evaluate feasibility, time, cost, adverse events, and improve upon the study design prior to performance of a full-scale research project. Here I submit that most of the pilot studies of alternative therapies are, in fact, bogus.
To qualify as a pilot study, an investigation needs to have an aim that is in line with the above-mentioned definition. Another obvious hallmark must be that its conclusions are in line with this aim. We do not need to conduct much research to find that even these two elementary preconditions are not fulfilled by the plethora of pilot studies that are currently being published, and that proper pilot studies of alternative medicine are very rare.
Three recent examples of dodgy pilot studies will have to suffice (but rest assured, there are many, many more).
The aim of this study was to evaluate the effects of foot reflexotherapy on pain and postural balance in elderly individuals with low back pain. And the conclusions drawn by its authors were that this study demonstrated that foot reflexotherapy induced analgesia but did not affect postural balance in elderly individuals with low back pain.
The aim of this study was to investigate the effect of Tai Chi training on dual-tasking performance that involved stepping down and compared it with that of conventional exercise among stroke survivors. And the conclusions read: These results suggest a beneficial effect of Tai Chi training on cognition among stroke survivors without compromising physical task performance in dual-tasking.
The aim of this study was to evaluate the efficacy [of acupuncture] over 12 weeks of treatment and 12 weeks of follow-up. And the conclusion: Acupuncture decreases WC, HC, HbA1c, TG, and TC values and blood pressure in MetS.
It is almost painfully obvious that these studies are not ‘pilot’ studies as defined above.
So, what are they, and why are they so popular in alternative medicine?
The way I see it, they are the result of amateur researchers conducting pseudo-research for publication in lamentable journals in an attempt to promote their pet therapies (I have yet to find such a study that reports a negative finding). The sequence of events that lead to the publication of such pilot studies is usually as follows:
- An enthusiast or a team of enthusiasts of alternative medicine decide that they will do some research.
- They have no or very little know-how in conducting a clinical trial.
- They nevertheless feel that such a study would be nice as it promotes both their careers and their pet therapy.
- They design some sort of a plan and start recruiting patients for their trial.
- At this point they notice that things are not as easy as they had imagined.
- They have too few funds and too little time to do anything properly.
- This does, however, not stop them to continue.
- The trial progresses slowly, and patient numbers remain low.
- After a while the would-be researchers get fed up and decide that their study has enough patients to stop the trial.
- They improvise some statistical analyses with their results.
- They write up the results the best they can.
- They submit it for publication in a 3rd class journal and, in order to get it accepted, they call it a ‘pilot study’.
- They feel that this title is an excuse for even the most obvious flaws in their work.
- The journal’s reviewers and editors are all proponents of alternative medicine who welcome any study that seems to confirm their belief.
- Thus the study does get published despite the fact that it is worthless.
Some might say ‘so what? no harm done!’
But I beg to differ: these studies pollute the medical literature and misguide people who are unable or unwilling to look behind the smoke-screen. Enthusiasts of alternative medicine popularise these bogus trials, while hiding the fact that their results are unreliable. Journalists report about them, and many consumers assume they are being told the truth – after all it was published in a ‘peer-reviewed’ medical journal!
My conclusions are as simple as they are severe:
- Such pilot studies are the result of gross incompetence on many levels (researchers, funders, ethics committees, reviewers, journal editors).
- They can cause considerable harm, because they mislead many people.
- In more than one way, they represent a violation of medical ethics.
- The could be considered scientific misconduct.
- We should think of stopping this increasingly common form of scientific misconduct.
In recent days, journalists across the world had a field day (mis)reporting that doctors practising integrative medicine were doing something positive after all. I think that the paper shows nothing of the kind – but please judge for yourself.
The authors of this article wanted to determine differences in antibiotic prescription rates between conventional General Practice (GP) surgeries and GP surgeries employing general practitioners (GPs) additionally trained in integrative medicine (IM) or complementary and alternative medicine (CAM) (referred to as IM GPs) working within National Health Service (NHS) England.
They conducted a retrospective study on antibiotic prescription rates per STAR-PU (Specific Therapeutic group Age–sex weighting Related Prescribing Unit) using NHS Digital data over 2016. Publicly available data were used on prevalence of relevant comorbidities, demographics of patient populations and deprivation scores. setting Primary Care. Participants were 7283 NHS GP surgeries in England. The association between IM GPs and antibiotic prescribing rates per STAR-PU with the number of antibiotic prescriptions (total, and for respiratory tract infection (RTI) and urinary tract infection (UTI) separately) as outcome. results IM GP surgeries (n=9) were comparable to conventional GP surgeries in terms of list sizes, demographics, deprivation scores and comorbidity prevalence.
Statistically significant fewer total antibiotics were prescribed at NHS IM GP surgeries compared with conventional NHS GP surgeries. In contrast, the number of antibiotics prescribed for UTI were similar between both practices.
The authors concluded that NHS England GP surgeries employing GPs additionally trained in IM/CAM have lower antibiotic prescribing rates. Accessibility of IM/CAM within NHS England primary care is limited. Main study limitation is the lack of consultation data. Future research should include the differences in consultation behaviour of patients self-selecting to consult an IM GP or conventional surgery, and its effect on antibiotic prescription. Additional treatment strategies for common primary care infections used by IM GPs should be explored to see if they could be used to assist in the fight against antimicrobial resistance.
The study was flimsy to say the least:
- It was retrospective and is therefore open to no end of confounders.
- There were only 9 surgeries in the IM group.
Moreover, the results were far from impressive. The differences in antibiotic prescribing between the two groups of GP surgeries were minimal or non-existent. Finally, the study was financed via an unrestricted grant of WALA Heilmittel GmbH, Germany (“approx. 900 different remedies conforming to the anthroposophic understanding of man and nature”) and its senior author has a long track record of publishing papers promotional for anthroposophic medicine.
Such pseudo-research seems to be popular in the realm of CAM, and I have commented before on similarly futile projects. The comparison, I sometimes use is that of a Hamburger restaurant:
Employees by a large Hamburger chain set out to study the association between utilization of Hamburger restaurant services and vegetarianism. The authors used a retrospective cohort design. The study population comprised New Hampshire residents aged 18-99 years, who had entered the premises of a Hamburger restaurant within 90 days for a primary purpose of eating. The authors excluded subjects with a diagnosis of cancer. They measured the likelihood of vegetarianism among recipients of services delivered by Hamburger restaurants compared with a control group of individuals not using meat-dispensing facilities. They also compared the cohorts with regard to the money spent in Hamburger restaurants. The adjusted likelihood of being a vegetarian was 55% lower among the experimental group compared to controls. The average money spent per person in Hamburger restaurants were also significantly lower among the Hamburger group.
To me, it is obvious that such analyses must produce a seemingly favourable result for CAM. In the present case, there are several reasons for this:
- GPs who volunteer to be trained in CAM tend to be in favour of ‘natural’ treatments and oppose synthetic drugs such as antibiotics.
- Education in CAM would only re-inforce this notion.
- Similarly, patients electing to consult IM GPs tend to be in favour of ‘natural’ treatments and oppose synthetic drugs such as antibiotics.
- Such patients might be less severely ill that the rest of the patient population (the data from the present study do in fact imply this to be true).
- These phenomena work in concert to generate less antibiotic prescribing in the IM group.
In the final analysis, all this finding amounts to is a self-fulfilling prophecy: grocery shops sell less meat than butchers! You don’t believe me? Perhaps you need to read a previous post then; it concluded that physicians practicing integrative medicine (the 80% who did not respond to the survey were most likely even worse) not only use and promote much quackery, they also tend to endanger public health by their bizarre, irrational and irresponsible attitudes towards vaccination.
What is upsetting with the present paper, in my view, are the facts that:
- a reputable journal published this junk,
- the international press has a field-day reporting this study implying that CAM is a good thing.
The fact is that it shows nothing of the kind. Imagine we send GPs on a course where they are taught to treat all their patients with blood-letting. This too would result in less prescription of antibiotics, wouldn’t it? But would it be a good thing? Of course not!
True, we prescribe too much antibiotics. Nobody doubts that. And nobody doubts that it is a big problem. The solution to this problem is not more CAM, but less antibiotics. To realise the solution we do not need to teach GPs CAM but we need to remind them of the principles of evidence-based practice. And the two are clearly not the same; in fact, they are opposites.
An announcement (it’s in German, I’m afraid) proudly declaring that ‘homeopathy fulfils the criteria of evidence-based medicine‘ caught my attention.
Here is the story:
In 2016, Dr. Melanie Wölk, did a ‘Master of Science’* at the ‘Donau University’ in Krems, Austria investigating the question whether homeopathy follows the rules of evidence-based medicine (EBM). She arrived at the conclusion that YES, IT DOES! This pleased the leading Austrian manufacturer of homeopathics (Dr Peithner) so much and so durably that, on 23 March 2018, he gave her a ‘scientific’ award (the annual Peithner award) for her ‘research’.
So far so good.
Her paper is unpublished, or at least not available on Medline; therefore, I am unable to evaluate it directly. All I know about it from the announcement is that she did her ‘research at the ‘Zentrum für Traditionelle Chinesische Medizin und Komplementärmedizin‘ of the said university. A quick Medline search revealed that this unit has never published anything, not a single paper, it seems! Disappointed I search for Dr. Christine Schauhuber, the leader of the unit; and again I find no Medline-listed publications in her name. My interim conclusion is thus that this institution might not be at the cutting edge of science.
But what do we know about Dr. Melanie Wölk’s award-winning master thesis *?
The announcement tells us that she investigated all RCTs published between 2010 and 2016. In addition, she evaluated:
- the ‘Swiss report’,
- the NHMRC report,
- Shang 2005,
- Ernst 2002,
- the Frass sepsis trial of 2005,
- Linde 1997 (why not Linde 1999? I ask myself; perhaps because this re-analysis of the same material came to a largly negative conclusion?)
On that basis, she arrived at her positive verdict – not just tentatively, but without doubt (“Das Ergebnis steht fest”).
Dr Peithner, the owner of the company and awarder of the prize, was quoted stating that this is a very important piece of work for homeopathy; it shows yet again what we see in our daily routine, namely that homeopathics are effective. Wölk’s investigation demonstrates furthermore that high-quality trials of homeopathy do exist, and that it is time to end the witch-hunt aimed at discrediting an effective therapy. Conventional medicine and homeopathy ought to finally work hand in hand – for the benefit of our patients. (“Für die Homöopathie ist das eine sehr wichtige Arbeit, die wieder zeigt, was wir in der ärztlichen Praxis täglich erleben, nämlich dass homöopathische Arzneimittel wirken. Wölks Untersuchung zeigt weiters deutlich, dass es sehr wohl hochqualitative Homöopathie-Studien gibt und es an der Zeit ist, die Hexenjagd zu beenden, mit der eine wirksame medizinische Therapie diskreditiert werden soll. Konventionelle Medizin und Homöopathie sollten endlich Hand in Hand arbeiten – zum Wohle der Patientinnen und Patienten.”)
I do hope that Dr Wölk uses the prize money (by no means a fortune; see photo) to buy some time for publishing her work (one of my teachers, all those years ago, used to say ‘unpublished research is no research’) so that we can all benefit from it. Until it becomes available, I should perhaps mention that the description of her methodology (publications between 2010 and 2016 [plus a few other papers that nicely fitted the arguments?]; including one Linde review and not his more recent re-analysis [see above]) does not inspire me to think that Dr Wölk’s research was anywhere near rigorous, systematic or complete. In the same vein, I am tempted to point out that the Swiss report is probably the very last document I would select, if I wanted to generate an objective picture about the value of homeopathy.
Taking all this into account, I conclude that we seem to be dealing here with a
- pseudo-prize (given by a commercial firm to further its business) for a piece of
- pseudo-research (the project seems to have been aimed to white-wash homeopathy) into
- pseudo-medicine (a treatment that has been tested extensively but has not been shown to work beyond placebo).
*Wölk, Melanie: Eminenz oder Evidenz: Die Homöopathie auf dem Prüfstand der Evidence based Medicine. Masterarbeit zur Erlangung des akademischen Abschlusses Master of Science im Universitätslehrgang Natural Medicine. Donau-Universität Krems, Department für Gesundheitswissenschaften und Biomedizin. Krems, Mai 2016.
The media have (rightly) paid much attention to the three Lancet-articles on low back pain (LBP) which were published this week. LBP is such a common condition that its prevalence alone renders it an important subject for us all. One of the three papers covers the treatment and prevention of LBP. Specifically, it lists various therapies according to their effectiveness for both acute and persistent LBP. The authors of the article base their judgements mainly on published guidelines from Denmark, UK and the US; as these guidelines differ, they attempt a synthesis of the three.
Several alternative therapist organisations and individuals have consequently jumped on the LBP bandwagon and seem to feel encouraged by the attention given to the Lancet-papers to promote their treatments. Others have claimed that my often critical verdicts of alternative therapies for LBP are out of line with this evidence and asked ‘who should we believe the international team of experts writing in one of the best medical journals, or Edzard Ernst writing on his blog?’ They are trying to create a division where none exists,
The thing is that I am broadly in agreement with the evidence presented in Lancet-paper! But I also know that things are a bit more complex.
Below, I have copied the non-pharmacological, non-operative treatments listed in the Lancet-paper together with the authors’ verdicts regarding their effectiveness for both acute and persistent LBP. I find no glaring contradictions with what I regard as the best current evidence and with my posts on the subject. But I feel compelled to point out that the Lancet-paper merely lists the effectiveness of several therapeutic options, and that the value of a treatment is not only determined by its effectiveness. Crucial further elements are a therapy’s cost and its risks, the latter of which also determines the most important criterion: the risk/benefit balance. In my version of the Lancet table, I have therefore added these three variables for non-pharmacological and non-surgical options:
|EFFECTIVENESS ACUTE LBP||EFFECTIVENESS PERSISTENT LBP||RISKS||COSTS||RISK/BENEFIT BALANCE|
|Advice to stay active||+, routine||+, routine||None||Low||Positive|
|Education||+, routine||+, routine||None||Low||Positive|
|Superficial heat||+/-||Ie||Very minor||Low to medium||Positive (aLBP)|
|Exercise||Limited||+/-, routine||Very minor||Low||Positive (pLBP)|
|CBT||Limited||+/-, routine||None||Low to medium||Positive (pLBP)|
|Rehab||Ie||+/-||Minor||Medium to high||Questionable|
Routine = consider for routine use
+/- = second line or adjunctive treatment
Ie = insufficient evidence
Limited = limited use in selected patients
vfbmae = very frequent, minor adverse effects
sae = serious adverse effects, including deaths, are on record
aLBP = acute low back pain
The reason why my stance, as expressed on this blog and elsewhere, is often critical about certain alternative therapies is thus obvious and transparent. For none of them (except for massage) is the risk/benefit balance positive. And for spinal manipulation, it even turns out to be negative. It goes almost without saying that responsible advice must be to avoid treatments for which the benefits do not demonstrably outweigh the risks.
I imagine that chiropractors, osteopaths and acupuncturists will strongly disagree with my interpretation of the evidence (they might even feel that their cash-flow is endangered) – and I am looking forward to the discussions around their objections.