MD, PhD, FMedSci, FSB, FRCP, FRCPEd

study design

1 2 3 6

Meniscus-injuries are common and there is no consensus as to how best treat them. Physiotherapists tend to advocate exercise, while surgeons tend to advise surgery.

Of course, exercise is not a typical alternative therapy but, as many alternative practitioners might disagree with this statement because they regularly recommend it to their patients, it makes sense to cover it on this blog. So, is exercise better than surgery for meniscus-problems?

The aim of this recent Norwegian study aimed to shed some light on this question. Specifically wanted to determine whether  exercise therapy is superior to arthroscopic partial meniscectomy for knee function in  patients with degenerative meniscal tears.

A total of 140 adults with degenerative medial meniscal tear verified by magnetic resonance imaging were randomised to either receiving 12 week supervised exercise therapy alone, or arthroscopic partial meniscectomy alone. Intention to treat analysis of between group difference in change in knee injury and osteoarthritis outcome score (KOOS4), defined a priori as the mean score for four of five KOOS subscale scores (pain, other symptoms, function in sport and recreation, and knee related quality of life) from baseline to two-year follow-up and change in thigh muscle strength from baseline to three months.

The results showed no clinically relevant difference between the two groups in change in KOOS4 at two years (0.9 points, 95% confidence interval −4.3 to 6.1; P=0.72). At three months, muscle strength had improved in the exercise group (P≤0.004). No serious adverse events occurred in either group during the two-year follow-up. 19% of the participants allocated to exercise therapy crossed over to surgery during the two-year follow-up, with no additional benefit.

The authors concluded that the observed difference in treatment effect was minute after two years of follow-up, and the trial’s inferential uncertainty was sufficiently small to exclude clinically relevant differences. Exercise therapy showed positive effects over surgery in improving thigh muscle strength, at least in the short-term. Our results should encourage clinicians and middle-aged patients with degenerative meniscal tear and no definitive radiographic evidence of osteoarthritis to consider supervised exercise therapy as a treatment option.

As I stated above, I mention this trial because exercise might be considered by some as an alternative therapy. The main reason for including it is, however, that it is in many ways an exemplary good study from which researchers in alternative medicine could learn.

Like so many alternative therapies, exercise is a treatment for which placebo-controlled studies are difficult, if not impossible. But that does not mean that rigorous tests of its value are impossible. The present study shows the way how it can be done.

Meaningful clinical research is no rocket science; it merely needs well-trained scientists who are willing to test the (rather than promote) their hypotheses. Sadly such individuals are as rare as gold dust in the realm of alternative medicine.

On 25 and 26 May of this year I wrote two posts about an acupuncture trial that, in my view, was dodgy. To refresh your memory, here is the relevant part of the 2nd post:

This new study was designed as a randomized, sham-controlled trial of acupuncture for persistent allergic rhinitis in adults investigated possible modulation of mucosal immune responses. A total of 151 individuals were randomized into real and sham acupuncture groups (who received twice-weekly treatments for 8 weeks) and a no acupuncture group. Various cytokines, neurotrophins, proinflammatory neuropeptides, and immunoglobulins were measured in saliva or plasma from baseline to 4-week follow-up.

Statistically significant reduction in allergen specific IgE for house dust mite was seen only in the real acupuncture group. A mean (SE) statistically significant down-regulation was also seen in pro-inflammatory neuropeptide substance P (SP) 18 to 24 hours after the first treatment. No significant changes were seen in the other neuropeptides, neurotrophins, or cytokines tested. Nasal obstruction, nasal itch, sneezing, runny nose, eye itch, and unrefreshed sleep improved significantly in the real acupuncture group (post-nasal drip and sinus pain did not) and continued to improve up to 4-week follow-up.

The authors concluded that acupuncture modulated mucosal immune response in the upper airway in adults with persistent allergic rhinitis. This modulation appears to be associated with down-regulation of allergen specific IgE for house dust mite, which this study is the first to report. Improvements in nasal itch, eye itch, and sneezing after acupuncture are suggestive of down-regulation of transient receptor potential vanilloid 1.

…the trial itself raises a number of questions:

  1. Which was the primary outcome measure of this trial?
  2. What was the power of the study, and how was it calculated?
  3. For which outcome measures was the power calculated?
  4. How were the subjective endpoints quantified?
  5. Were validated instruments used for the subjective endpoints?
  6. What type of sham was used?
  7. Are the reported results the findings of comparisons between verum and sham, or verum and no acupuncture, or intra-group changes in the verum group?
  8. What other treatments did each group of patients receive?
  9. Does anyone really think that this trial shows that “acupuncture is a safe, effective and cost-effective treatment for allergic rhinitis”?

In the comments section, the author wrote: “after you have read the full text and answered most of your questions for yourself, it might then be a more appropriate time to engage in any meaningful discussion, if that is in fact your intent”, and I asked him to send me his paper. As he does not seem to have the intention to do so, I will answer the questions myself and encourage everyone to have a close look at the full paper [which I can supply on request].

  1. The myriad of lab tests were defined as primary outcome measures.
  2. Two sentences are offered, but they do not allow me to reconstruct how this was done.
  3. No details are provided.
  4. Most were quantified with a 3 point scale.
  5. Mostly not.
  6. Needle insertion at non-acupoints.
  7. The results are a mixture of inter- and intra-group differences.
  8. Patients were allowed to use conventional treatments and the frequency of this use was reported in patient diaries.
  9. I don’t think so.

So, here is my interpretation of this study:

  • It lacked power for many outcome measures, certainly the clinical ones.
  • There were hardly any differences between the real and the sham acupuncture group.
  • Most of the relevant results were based on intra-group changes, rather than comparing sham with real acupuncture, a fact, which is obfuscated in the abstract.
  • In a controlled trial fluctuations within one group must never be interpreted as caused by the treatment.
  • There were dozens of tests for statistical significance, and there seems to be no correction for multiple testing.
  • Thus the few significant results that emerged when comparing sham with real acupuncture might easily be false positives.
  • Patient-blinding seems questionable.
  • McDonald as the only therapist of the study might be suspected to have influenced his patients through verbal and non-verbal communications.

I am sure there are many more flaws, particularly in the stats, and I leave it to others to identify them. The ones I found are, however, already serious enough, in my view, to call for a withdrawal of this paper. Essentially, the authors seem to have presented a study with largely negative findings as a trial with positive results showing that acupuncture is an effective therapy for allergic rhinitis. Subsequently, McDonald went on social media to inflate his findings even more. One might easily ask: is this scientific misconduct or just poor science?

END OF QUOTE

This and the previous post created lots of discussion and comments. However, the question whether the study in question amounted to scientific misconduct was never satisfactorily resolved. Therefore, I decided to write to the editor of ‘Ann Allergy Asthma Immunol‘ where the trial had been published. He answered by saying I would need to file an official complaint for him to address the issue. On 13 June, I therefore sent him the following email:

Thank you for your letter of 3/6/2016 suggesting I make a formal complaint about the paper entitled ‘EFFECT OF ACUPUNCTURE ON HOUSE DUST MITE…’ [ Ann Allergy Asthma Immunol 2016] by McDonald et al. I herewith wish to file such a complaint.

The article in question reports an RCT of acupuncture for persistent allergic rhinitis. It followed a parallel group design with 3 groups receiving the following interventions:

1.       Acupuncture

2.       Sham-acupuncture

3.       No treatment

There was a plethora of outcome measures and time points on which they were measured. A broad range of parameters was defined as primary endpoints.

The conclusion reached by the authors essentially was that acupuncture affected several outcome measures in a positive sense, thus supporting the notion that acupuncture is efficacious [“Symptoms and quality of life improved significantly and were still continuing to improve 4 weeks after treatment ceased.”] This conclusion, however, is misleading and needs correcting.

The main reasons for this are as follows:

·         Despite the fact that the authors did many dozens of statistical tests for significance, they did not correct for this multiplicity of tests. Consequently, some or most of the significant results are likely to be false positive.

·         Many of the positive results of this paper were not obtained by comparing one group to another but by doing before/after comparisons within one group. This approach defies the principle of a controlled clinical trial. For doing intra-group comparisons, we obviously do not need any control group at all. The findings from intra-group comparisons are prominently reported in the paper, for instance in the abstract, giving the impression that they originate from inter-group comparisons. One has to read the paper very carefully to find that, when inter-group comparisons were conducted, their results did NOT confirm the findings from the reported intra-group comparisons. As this is the case for most of the symptomatic endpoints, the impression given is seriously misleading and needs urgent correction.

On the whole, the article is a masterpiece of obfuscation and misrepresentation of the actual data. I urge you to consider the harm than can be done by such a misleading publication. In my view, the best way to address this problem is to withdraw the article.

I look forward to your decision.

Regards

E Ernst

END OF QUOTE

I had to send several reminders but my most recent one prompted the following response dated 7/11/ 2016:

Dear Professor Ernst,
Thank you for your patience while we worked through the process of considering your complaints levied on the article entitled ‘EFFECT OF ACUPUNCTURE ON HOUSE DUST MITE…’ [ Ann Allergy Asthma Immunol 2016] by McDonald et al. I considered the points that you made in your previous letter, sought input from our editorial team (including our biostatistics editor) , our publisher and the authors themselves. I sent the the  charges ( point by point) anonymously to the authors and allowed them time to respond which they did. I had their responses reviewed by selected editors and , as a result of this process, have decided not to pursue any corrective or punitive action based upon the following :
 
  1. Our editorial team recognizes that this is not the best clinical trial we have published in the Annals of Allergy, Asthma and Immunology. However, neither is is the worst. As in most published research studies, there are always things that could have been done better to make it a stronger paper. Never-the-less, the criticism falls fall short of any sort of remedy that would include withdrawal of the manuscript.
  2. Regarding your accusation that the multiple positive endpoint resulted in the authors making specific therapeutic claims, our assessment is that no specific therapeutic claim was made but rather the authors maintained that the data support the value of acupuncture in improving symptoms and quality of life in patients with AR. We do not believe there was overreach in those statements.
  3. The authors’ stated intent was to show immune changes associated with clinical markers of improvement in the active acupuncture group compared to controls. The authors maintain (and our editors agree) that their data assessments were primarily based upon three statistical tests not “dozens” (as stated in your original letter of complaint).  The power analysis and sample size calculations were presented to us and deemed adequate , making the probability of a type I error quite low.
  4. The authors acknowledge in their paper that there could be limitations to their data interpretation based upon potential disparities between intra- and intergroup comparisons. The editors felt their transparency was adequately disclosed.
In summary, as editor-in-chief of the Annals of Allergy, Asthma and Immunology, I did not find sufficient merit in your charges to initiate any corrective or punitive action for this manuscript. I understand you will strongly disagree with this decision and I regret that. However, in the final analysis, my primary intent is to preserve the objectivity, fairness, and integrity of our journal and its review process. I believe I have accomplished that in this instance.
 
Sincerely
END OF QUOTE
This seems to settle the issue: the study in question does not involve scientific misconduct!
Or does it?
I would be grateful for the view of the readers of this post.

Stable angina is a symptom of coronary heart disease which, in turn, is amongst the most frequent causes of death in developed countries. It is an alarm bell to any responsible clinician and requires causal, often life-saving treatments of which we today have several options. The last thing a patient needs in this condition is ACUPUNCTURE, I would say.

Yet acupuncture is precisely the therapy such patients might be tempted to employ.

Why?

Because irresponsible or criminally naïve acupuncturists advertise it!

Take this website, for instance; it informs us that a meta-analysis of eight clinical trials conducted between 2000 and 2014 demonstrates the efficacy of acupuncture for the treatment of stable angina. In all eight clinical trials, patients treated with acupuncture experienced a greater rate of angina relief than those in the control group treated with conventional drug therapies (90.1% vs 75.7%)….

I imagine that this sounds very convincing to patients and I fear that many might opt for acupuncture instead of potentially invasive/unpleasant but life-saving intervention. The original meta-analysis to which the above promotion referred to is equally optimistic. Here is its abstract:

Angina pectoris is a common symptom imperiling patients’ life quality. The aim of this study is to evaluate the efficacy and safety of acupuncture for stable angina pectoris. Clinical randomized-controlled trials (RCTs) comparing the efficacy of acupuncture to conventional drugs in patients with stable angina pectoris were searched using the following database of PubMed, Medline, Wanfang and CNKI. Overall odds ratio (ORs) and weighted mean difference (MD) with their 95% confidence intervals (CI) were calculated by using fixed- or random-effect models depending on the heterogeneity of the included trials. Total 8 RCTs, including 640 angina pectoris cases with 372 patients received acupuncture therapy and 268 patients received conventional drugs, were included. Overall, our result showed that acupuncture significantly increased the clinical curative effects in the relief of angina symptoms (OR=2.89, 95% CI=1.87-4.47, P<0.00001) and improved the electrocardiography (OR=1.83, 95% CI=1.23-2.71, P=0.003), indicating that acupuncture therapy was superior to conventional drugs. Although there was no significant difference in overall effective rate relating reduction of nitroglycerin between two groups (OR=2.13, 95% CI=0.90-5.07, P=0.09), a significant reduction on nitroglycerin consumption in acupuncture group was found (MD=-0.44, 95% CI=-0.64, -0.24, P<0.0001). Furthermore, the time to onset of angina relief was longer for acupuncture therapy than for traditional medicines (MD=2.44, 95% CI=1.64-3.24, P<0.00001, min). No adverse effects associated with acupuncture therapy were found. Acupuncture may be an effective therapy for stable angina pectoris. More clinical trials are needed to systematically assess the role of acupuncture in angina pectoris.

In the discussion section of the full paper, the authors explain that their analysis has several weaknesses:

Several limitations were presented in this meta-analysis. Firstly, conventional drugs in control group were different, this may bring some deviation. Secondly, for outcome of the time to onset of angina relief with acupuncture, only one trial included. Thirdly, the result of some outcomes presented in different expression method such as nitroglycerin consumption. Fourthly, acupuncture combined with traditional medicines or other factors may play a role in angina pectoris.

However, this does not deter them to conclude on a positive note:

In conclusion, we found that acupuncture therapy was superior to the conventional drugs in increasing the clinical curative effects of angina relief, improving the electrocardiography, and reducing the nitroglycerin consumption, indicating that acupuncture therapy may be effective and safe for treating stable angina pectoris. However, further clinical trials are needed to systematically and comprehensively evaluate acupuncture therapy in angina pectoris.

So, why do I find this irresponsibly and dangerously misleading?

Here a just a few reasons why this meta-analysis should not be trusted:

  • There was no systematic attempt to evaluate the methodological rigor of the primary studies; any meta-analysis MUST include such an assessment, or else it is not worth the paper it was printed on.
  • The primary studies all look extremely weak; this means they are likely to be false-positive.
  • They often assessed not acupuncture alone but in combination with other treatments; consequently the findings cannot be attributed to acupuncture.
  • All the primary studies originate from China; we have seen previously (see here and here) that Chinese acupuncture trials deliver nothing but positive results which means that their results cannot be trusted: they are false-positive.

My conclusion: the authors, editors and reviewers responsible for this article should be ashamed; they committed or allowed scientific misconduct, mislead the public and endangered patients’ lives.

Antrodia cinnamomea (AC) is a fungus which is used in Taiwan as a remedy for cancer, hypertension, hangover and other conditions. There are several commercial AC products and the annual market is worth over $100 million in Taiwan alone.

Several studies have suggested anti-cancer properties in vitro but few clinical trials have been reported. Now Taiwanese researchers published a double-blind, randomized clinical study to investigate whether AC had acceptable safety and efficacy in advanced cancer patients receiving chemotherapy.

Patients with advanced and/or metastatic adenocarcinoma, performance status (PS) 0-2, and adequate organ function who had previously been treated with standard chemotherapy were randomly assigned to receive routine chemotherapy regimens with AC (20 ml twice daily) orally for 30 days or placebo. The primary endpoint was 6-month overall survival (OS); the secondary endpoints were disease control rate (DCR), quality of life (QoL), adverse event (AE), and biochemical features within 30 days of treatment.

A total of 37 subjects with gastric, lung, liver, breast, and colorectal cancer (17 in the AC group, 20 in the placebo group) were enrolled in the study. Disease progression was the primary cause of death in 4 (33.3 %) AC and 8 (66.7 %) placebo recipients. Mean OSs were 5.4 months for the AC group and 5.0 months for the placebo group (p = 0.340), and the DCRs were 41.2 and 55 %, respectively (p = 0.33). Most hematologic, liver, or kidney functions did not differ significantly between the two groups, but platelet counts were lower in the AC group than in the placebo group (p = 0.02). QoL assessments were similar in the two groups, except that the AC group showed significant improvements in quality of sleep (p = 0.04).

12906_2016_1312_fig2_html

The above figure shows the survival curves for both groups.

The authors concluded as follows: Although we found a lower mortality rate and longer mean OS in the AC group than in the control group, AC combined with chemotherapy was not shown to improve the outcome of advanced cancer patients, possibly due to the small sample size. In fact, the combination may present a potential risk of lowered platelet counts. Adequately powered clinical trials will be necessary to address this question.

I agree, the survival curve looks promising. But we must not get carried away: this was a tiny sample size and a relatively short treatment period. Thus the difference could be a coincidence or an artefact.

The investigators are sufficiently cautious in the interpretation of their findings, and most of us would probably agree that it is necessary to submit such traditional remedies to proper scientific tests. Yet, I feel a sense of unease when I read such articles.

On the one hand, it is possible that such investigations meaningfully contribute to progress. On the other hand, I wonder whether they merely end up providing a significant boost to the trade of bogus remedies sold at high prices to desperate patients. Do the benefits really out-weigh the risks? We will probably never know.

But to minimize the risk, the authors should now swiftly conduct a more definitive trial and create some clarity about the value or otherwise of this traditional cancer remedy.

A new study tested the efficacy of chiropractic spinal manipulative therapy (CSMT) for migraine. It was designed as a three-armed, single-blinded, placebo -controlled RCT of 17 months duration including 104 migraineurs with at least one migraine attack per month. Active treatment consisted of CSMT (group 1) and the placebo was a sham push manoeuvre of the lateral edge of the scapula and/or the gluteal region (group 2). The control group continued their usual pharmacological management (group 3).

The RCT began with a one-month run-in followed by three months intervention. The outcome measures were quantified at the end of the intervention and at 3, 6 and 12 months of follow-up. The primary end-point was the number of migraine days per month. Secondary end-points were migraine duration, migraine intensity and headache index, and medicine consumption.

The results show that migraine days were significantly reduced within all three groups from baseline to post-treatment (P < 0.001). The effect continued in the CSMT and placebo groups at all follow-up time points (groups 1 and 2), whereas the control group (group 3) returned to baseline. The reduction in migraine days was not significantly different between the groups. Migraine duration and headache index were reduced significantly more in the CSMT than in group 3 towards the end of follow-up. Adverse events were few, mild and transient. Blinding was strongly sustained throughout the RCT.

The authors concluded that it is possible to conduct a manual-therapy RCT with concealed placebo. The effect of CSMT observed in our study is probably due to a placebo response.

Chiropractors often cite clinical trials which suggest that CSMT might be effective. The effects sizes are rarely impressive, and it is tempting to suspect that the outcomes are mostly due to bias. Chiropractors, of course, deny such an explanation. Yet, to me, it seems fairly obvious: trials of CSMT are not blind, and therefore the expectation of the patient is likely to have major influence on the outcome.

Because of this phenomenon (and several others, of course), sceptics are usually unconvinced of the value of chiropractic. Chiropractors often respond by claiming that blind studies of physical intervention such as CSMT are not possible. This, however, is clearly not true; there have been several trials that employed sham treatments which adequately mimic CSMT. As these frequently fail to show what chiropractors had hoped, the methodology is intensely disliked by chiropractors.

The above study is yet another trial that adequately controls for patients’ expectation, and it shows that the apparent efficacy of CSMT disappears when this source of bias is properly accounted for. To me, such findings make a lot of sense, and I suspect that most, if not all the ‘positive’ studies of CSMT would turn out to be false positive, once such residual bias is eliminated.

This new study is amazing in several respects. It was conducted in Spain by otolaryngologists, and one of its authors is an employee of Boiron, the world’s biggest manufacturer of homeopathic products. It was designed as a double blind, placebo-controlled RCT. Patients aged 2 months to 12 years suffering from otitis media with effusion (OME), as diagnosed by pneumatic otoscopy (PNO) and tympanometry, were randomized into two groups. Both groups received aerosol therapy (mucolytics and corticosteroids). In addition, the experimental group received a homeopathic remedy of Agraphis nutans 5CH, Thuya Occidentalis 5CH, Kalium muriaticum 9CH and Arsenicum iodatum. The placebo group received placebos instead. Both of the treatments were continued for 3 months. Patients were evaluated by PNO examination and tympanometry at baseline, at 45 and 90 days.

A total of 97 patients were enrolled in this study. In the homeopathy group, 61.9% of individuals were cured according to PNO results by the 3rd visit compared with 56.8% of patients treated with placebo. 4.8% of patients in the homeopathy group suffered a recurrence (positive PNO in the 2nd visit changed to negative in the 3rd visit), while 11.4% did in the placebo group. These inter-group differences were not statistically significant. Adverse events were distributed similarly, except in the case of upper respiratory tract infections, which were less frequent in homeopathic group.

The authors of this new RCT concluded that the homeopathic scheme used as adjuvant treatment cannot be claimed to be an effective treatment in children with OME.

No surprises then – we already know that homeopathic remedies are placebos!

Sure, but at least two amazing features need to be pointed out:

  • I am delighted that the authors did not try to spin the results such that they appear to be positive. Some investigators might have emphasised the fact that there was a (non-significant) trend in favour of homeopathy, and that, for a secondary outcome measure (upper respiratory infections), it even reached the level of statistical significance.
  • Considering that this study was obviously Boiron-sponsored and its list of authors included an employee of this firm, such honesty can’t have been easy to maintain.
  • The design of this RCT is also worth a mention: most alt med proponents seem to think that ‘adjunctive’ use of alt med needs to be tested via the infamous ‘A+B vs B’ design which fails to control for placebo effects and therefore invariably produces false positive findings. The authors of this trial did the right thing by randomising their patients into usual care + homeopathy vs usual care + placebo. This is very simple and has the advantage to actually provide a meaningful result.

In view of all this, I raise my hat to the Spanish researchers: very well done!!!

If all trials of homeopathy were conducted and reported in this honourable fashion, the collective evidence would be in a much better state and far less confusing.

After > 200 years of existence, homeopathy still remains unproven – in fact, most rational thinkers would call it disproven. Today only homeopaths doubt this statement; they work hard to find a water-tight proof that might show the doubters to be wrong.

What is better suited for this purpose than a few rigorous animal experiments?

Engystol® is a popular homeopathic product promoted as an anti-viral agent manufactured by Heel GmbH, Baden-Baden, Germany. In several in vivo and in vitro studies, it apparently affected an immune response. This new study was to “evaluate the innate and adaptive immuno-modulatory effects of oral Engystol® (1 or 10 tablets/L water consumed), prior to and post antigenic challenge in a mouse model with a well-characterized and clinically measureable immune system.”

The investigators first evaluated the murine immune response when oral Engystol® was given alone for 28 days. to mice. The animals were then challenged with an antigen-specific H5N1 HA vaccine while on Engystol® for an additional 33 days. Serum and supernatants from cultured splenic lymphocytes were collected and screened with a 32-cytokine panel. Serum vaccine epitope-specific IgG titers plus T cell and B cell phenotypes from splenic tissue were also evaluated.

The results showed that Engystol® alone did not alter immunity. However, upon vaccine challenge, Engystol® decreased CD4+/CD8+ ratios, altered select cytokines/chemokines, and anti-H5N1 HA IgG titers were increased in the group of mice receiving 10 tablet/L.

The authors concluded that “these data suggest that Engystol® can modulate immunity upon antigenic challenge.”

Engystol is being advertised as “a homeopathic preparation which has been scientifically proven to significantly reduce the duration and severity of symptoms during an acute viral infection and help protect from subsequent infections.” I was unable find good evidence for this claim and therefore have to assume that it is bogus. The only human trial I was able to locate was this one:

OBJECTIVE:

To compare the effects of a complex homeopathic preparation (Engystol; Heel GmbH, Baden-Baden, Germany) with those of conventional therapies with antihistamines, antitussives, and nonsteroidal antiinflammatory drugs on upper respiratory symptoms of the common cold in a setting closely related to everyday clinical practice.

DESIGN:

Nonrandomized, observational study over a treatment period of maximally two weeks.

SETTING:

Eighty-five general and homeopathic practices in Germany.

PARTICIPANTS:

Three hundred ninety-seven patients with upper respiratory symptoms of the common cold.

INTERVENTIONS:

Engystol-based therapy or common over-the-counter treatments for the common cold. Patients receiving this homeopathic treatment were allowed other short-term medications, but long-term use of analgesics, antibiotics, and antiinflammatory agents was not permitted. Patients were allowed nonpharmacological therapies such as vitamins, thermotherapies, and others.

MAIN OUTCOME MEASURES:

The effects of treatment were evaluated on the variables fatigue, sensation of illness, chill/tremor, aching joints, overall severity of illness, sum of all clinical variables, temperature, and time to symptomatic improvement.

RESULTS:

Both treatment regimens provided significant symptomatic relief, and this homeopathic treatment was noninferior in a noninferiority analysis. Significantly more patients (P < .05) using Engystol-based therapy reported improvement within 3 days (77.1% vs 61.7% for the control group). No adverse events were reported in any of the treatment groups.

CONCLUSION:

This homeopathic treatment may be a useful component of an integrated symptomatic therapy for the common cold in patients and practitioners choosing an integrative approach to medical care.

Let me comment on the human study first. It is an excellent example of the bias that can be introduced by non-randomization. The patients in the homeopathic group obviously were those who chose to be treated homeopathically. Consequently they had high expectations in this therapy. Consequently they reported better results than the control group. In other words the reported outcomes have nothing to do with the homeopathic remedy.

But what about the animal study? Animals, we hear so often, do not exhibit a placebo response. Does that render this investigation any more reliable?

The answer, I am afraid is no.

The animal study in question had no control group at all. Therefore a myriad of factors could have caused the observed result. This study is very far from a poof of homeopathy!

But even if the findings of the two studies had not been the result of bias and confounding, I would be more than cautious about viewing them as anything near conclusive. The reason lies in the nature of this particular homeopathic remedy.

Engystol® contains Vincetoxicum hirundinaria (D6), Vincetoxicum hirundinaria (D10), Vincetoxicum hirundinaria (D30), sulphur (D4) and sulphur (D10). In other words, it is one of those combination remedies which are not sufficiently dilute to be devoid of active molecules. Sulphur D4, for instance, means that the remedy contains one part of sulphur in 10 000 parts of diluent. It is conceivable, even likely that such a concentration might affect certain immune parameters, I think.

And my conclusion from all this?

The proof of homeopathy – if it ever came – would need to be based on investigations that are more rigorous than these two rather pathetic studies.

In alternative medicine, good evidence is like gold dust and good evidence showing that alternative therapies are efficacious is even rarer. Therefore, I was delighted to come across a brand-new article from an institution that should stand for reliable information: the NIH, no less.

According to its authors, this new article “examines the clinical trial evidence for the efficacy and safety of several specific approaches—acupuncture, manipulation, massage therapy, relaxation techniques including meditation, selected natural product supplements (chondroitin, glucosamine, methylsulfonylmethane, S-adenosylmethionine), tai chi, and yoga—as used to manage chronic pain and related disability associated with back pain, fibromyalgia, osteoarthritis, neck pain, and severe headaches or migraines.”

The results of this huge undertaking are complex, of course, but in a nutshell they are at least partly positive for alternative medicine. Specifically, the authors state that “based on a preponderance of positive trials vs negative trials, current evidence suggests that the following complementary approaches may help some patients manage their painful health conditions: acupuncture and yoga for back pain; acupuncture and tai chi for OA of the knee; massage therapy for neck pain with adequate doses and for short-term benefit; and relaxation techniques for severe headaches and migraine. Weaker evidence suggests that massage therapy, SM, and osteopathic manipulation might also be of some benefit to those with back pain, and relaxation approaches and tai chi might help those with fibromyalgia.”

This is excellent news! Finally, we have data from an authoritative source showing that some alternative treatments can be recommended for common pain conditions.

Hold on, not so fast! Yes, the NIH is a most respectable organisation, but we must not blindly accept anything of importance just because it appears to come form a reputable source. Let’s look a bit closer at the actual evidence provided by the authors of this paper.

Reading the article carefully, it is impossible not to get troubled. Here are a few points that concern me most:

  • the safety of a therapy cannot be evaluated on the basis of data from RCTs (particularly as it has been shown repeatedly that trials of alternative therapies often fail to report adverse effects); much larger samples are needed for that; any statements about safety in the aims of the paper are therefore misplaced;
  • the authors talk about efficacy but seem to mean effectiveness;
  • the authors only included RCTs from the US which must result in a skewed and incomplete picture;
  • the article is from the National Center for Complementary and Integrative Health which is part of the NIH but which has been criticised repeatedly for being biased in favour of alternative medicine;
  • not all of the authors seem to be NIH staff, and I cannot find a declaration of conflicts of interest;
  • the discussion of the paper totally lacks any critical thinking;
  • there is no assessment of the quality of the trials included in this review.

My last point is by far the most important. A summary of this nature that fails to take into account the numerous limitations of the primary data is, I think, as good as worthless. As I know most of the RCTs included in the analyses, I predict that the overall picture generated by this review would have changed substantially, if the risks of bias in the primary studies had been accounted for.

Personally, I find it lamentable that such a potentially worthy exercise ended up employing such lousy methodology. Perhaps even more lamentable is the fact that the NIH (or one of its Centers) can descend that low; to mislead the public in this way borders on scientific misconduct and is, in my view, unethical and unacceptable.

Cranio-sacral therapy has been a subject on this blog before, for instance here, here and here. The authors of this single-blind, randomized trial explain in the introduction of their paper that “cranio-sacral therapy is an alternative and complementary therapy based on the theory that restricted movement at the cranial sutures of the skull negatively affect rhythmic impulses conveyed through the cerebral spinal fluid from the cranium to the sacrum. Restriction within the cranio-sacral system can affect its components: the brain, spinal cord, and protective membranes. The brain is said to produce involuntary, rhythmic movements within the skull. This movement involves dilation and contraction of the ventricles of the brain,  which produce the circulation of the cerebral spinal fluid. The theory states that this fluctuation mechanism causes reciprocal tension within the membranes, transmitting motion to the cranial bones and the sacrum. Cranio-sacral therapy and cranial osteopathic manual therapy originate from the observations made by William G. Sutherland, who said that the bones of the human skeleton have mobility. These techniques are based mainly on the study of anatomic and physiologic mechanisms in the skull and their relation to the body as a whole, which includes a system of diagnostic and therapeutic techniques aimed at treatment and prevention of diseases. These techniques are  based on the so-called primary respiratory movement, which is manifested in the mobility of the cranial bones, sacrum,  dura, central nervous system, and cerebrospinal fluid. The main difference between the two therapies is that cranial osteopathy, in addition to a phase that works in the direction of the lesion (called the functional phase), also uses a phase that worsens the injury, which is called structural phase.”

With this study, the researchers wanted to evaluate the effects of cranio-sacral therapy on disability, pain intensity, quality of life, and mobility in patients with low back pain. Sixty-four patients with chronic non-specific low back pain were assigned to an experimental group receiving 10 sessions of craniosacral therapy, or to the control group receiving 10 sessions of classic massage. Craniosacral therapy took 50 minutes and was conducted as follows: With pelvic diaphragm release, palms are placed in transverse position on the superior aspect of the pubic bone, under  the L5–S1 sacrum, and finger pads are placed on spinal processes.  With respiratory diaphragm release, palms are placed transverse under T12/L1 so that the spine lies along the start of fingers and the border of palm, and the anterior hand is placed on the breastbone. For thoracic inlet release, the thumb and index finger are placed on the opposite sides of the clavicle, with the posterior hand/palm of the hand cupping C7/T1. For the hyoid release, the thumb and index finger are placed on the hyoid, with the index finger on the occiput and the cupping finger pads on the cervical vertebrae. With the sacral technique for stabilizing L5/sacrum, the fingers contact the sulcus and the palm of the hand is in contact with the distal part of the sacral bone. The non-dominant hand of the therapist rested over the pelvis, with one hand on one iliac crest and the elbow/forearm of the other side over the other iliac crest. For CV-4 still point induction, thenar pads are placed under the occipital protuberance, avoiding mastoid sutures.  Classic massage protocol was compounded by the following sequence techniques of soft tissue massage on the  low back: effleurage, petrissage, friction, and kneading. The maneuvers are performed with surface pressure, followed by deep pressure and ending with surface pressure again. The techniques took 30 minutes.

Disability (Roland Morris Disability Questionnaire RMQ, and Oswestry Disability Index) was the primary endpoint. Other outcome measures included the pain intensity (10-point numeric pain rating scale), kinesiophobia (Tampa Scale of Kinesiophobia), isometric endurance of trunk flexor muscles (McQuade test), lumbar mobility in flexion, hemoglobin oxygen saturation, systolic blood pressure, diastolic blood pressure, hemodynamic measures (cardiac index), and biochemical analyses of interstitial fluid. All outcomes were measured at baseline, after treatment, and one-month follow-up.

No statistically significant differences were seen between groups for the main outcome of the study, the RMQ. However, patients receiving craniosacral therapy experienced greater improvement in pain intensity (p ≤ 0.008), hemoglobin oxygen saturation (p ≤ 0.028), and systolic blood pressure (p ≤ 0.029) at immediate- and medium-term and serum potassium (p = 0.023) level and magnesium (p = 0.012) at short-term than those receiving classic massage.

The authors concluded that 10 sessions of cranio-sacral therapy resulted in a statistically greater improvement in pain intensity, hemoglobin oxygen saturation, systolic blood pressure, serum potassium, and magnesium level than did 10 sessions of classic massage in patients with low back pain.

Given the results of this study, the conclusion is surprising. The primary outcome measure failed to show an inter-group difference; in other words, the results of this RCT were essentially negative. To use secondary endpoints – most of which are irrelevant for the study’s aim – in order to draw a positive conclusion seems odd, if not misleading. These positive findings are most likely due to the lack of patient-blinding or to the 200 min longer attention received by the verum patients. They are thus next to meaningless.

In my view, this publication is yet another example of an attempt to turn a negative into a positive result. This phenomenon seems embarrassingly frequent in alternative medicine. It goes without saying that it is not just misleading but also dishonest and unethical.

This post is dedicated to Mel Koppelman.

Those who followed the recent discussions about acupuncture on this blog will probably know her; she is an acupuncturist who (thinks she) knows a lot about research because she has several higher qualifications (but was unable to show us any research published by herself). Mel seems very quick in lecturing others about research methodology. Yesterday, she posted this comment in relation to my previous post on a study of aromatherapy and reflexology:

Professor Ernst, This post affirms yet again a rather poor understanding of clinical trial methodology. A pragmatic trial such as this one with a wait-list control makes no attempt to look for specific effects. You say “it is quite simply wrong to assume that this outcome is specifically related to the two treatments.” Where have specific effects been tested or assumed in this study? Your statement in no way, shape or form negates the author’s conclusions that “aromatherapy massage and reflexology are simple and effective non-pharmacologic nursing interventions.” Effectiveness is not a measure of specific effects.

I am most grateful for this comment because it highlights an issue that I had wanted to address for some time: The meanings of the two terms ‘efficacy and effectiveness’ and their differences as seen by scientists and by alternative practitioners/researchers.

Let’s start with the definitions.

I often use the excellent book of Alan Earl-Slater entitled THE HANDBOOK OF CLINICAL TRIALS AND OTHER RESEARCH. In it, EFFICACY is defined as ‘the degree to which an intervention does what it is intended to do under ideal conditions. EFFECTIVENESS is the degree to which a treatment works under real life conditions. An EFFECTIVENESS TRIAL is a trial that ‘is said to approximate reality (i. e. clinical practice). It is sometimes called a pragmatic trial’. An EFFICACY TRIAL ‘is a clinical trial that is said to take place under ideal conditions.’

In other words, an efficacy trial investigates the question, ‘can the therapy work?’, and an effectiveness trial asks, ‘does this therapy work?’ In both cases, the question relate to the therapy per se and not to the plethora of phenomena which are not directly related to it. It seems logical that, where possible, the first question would need to be addressed before the second – it does make little sense to test for effectiveness, if efficacy has not been ascertained, and effectiveness without efficacy does not seem to be possible.

In my 2007 book entitled UNDERSTANDING RESEARCH IN COMPLEMENTARY AND ALTERNATIVE MEDICINE (written especially for alternative therapists like Mel), I adopted these definitions and added: “It is conceivable that a given therapy works only under optimal conditions but not in everyday practice. For instance, in clinical practice patients may not comply with a therapy because it causes adverse effects.” I should have added perhaps that adverse effects are by no means the only reason for non-compliance, and that non-compliance is not the only reason why an efficacious treatment might not be effective.

Most scientists would agree with the above definitions. In fact, I am not aware of a debate about them in scientific circles. But they are not something alternative practitioners tend to like. Why? Because, using these strict definitions, many alternative therapies are neither of proven efficacy nor effectiveness.

What can be done about this unfortunate situation?

Simple! Let’s re-formulate the definitions of efficacy and effectiveness!

Efficacy, according to some alternative medicine proponents, refers to the therapeutic effects of the therapy per se, in other words, its specific effects. (That coincides almost with the scientific definition of this term – except, of course, it fails to tell us anything about the boundary conditions [optimal or real-life conditions].)

Effectiveness, according to the advocates of alternative therapies, refers to its specific effects plus its non-specific effects. Some researchers have even introduced the term ‘real-life effectiveness’ for this.

This is why, the authors of the study discussed in my previous post, could conclude that “aromatherapy massage and reflexology are simple… effective… interventions… to help manage pain and fatigue in patients with rheumatoid arthritis.” Based on their data, neither aromatherapy nor reflexology has been shown to be effective. They might appear to be effective because patients expected to get better, or patients in the no-treatment control group felt worse for not getting the extra care. Based on studies of this nature, giving patients £10 or a box of chocolate might also turn out to be “simple… effective… interventions… to help manage pain and fatigue in patients with rheumatoid arthritis.” Based on these definitions of efficacy and effectiveness, there are hardly any limits to bogus claims for any old quackery.

Such obfuscation suits proponents of alternative therapies fine because, using such definitions, virtually every treatment anyone might ever think of can be shown to be effective! Wishful thinking, it seems, can fulfil almost any dream, it can even turn the truth upside down.

Or can anyone name an alternative treatment that cannot even generate a placebo response when administered with empathy, sympathy and care? Compared to doing nothing, virtually every ineffective therapy might generate outcomes that make the treatment look effective. Even the anticipation of an effect alone might do the trick. How often have you had a tooth-ache, went to the dentist, and discovered sitting in the waiting room that the pain had gone? Does that mean that sitting in a waiting room is an effective treatment for dental pain?

In fact, some enthusiasts of alternative medicine could soon begin to argue that, with their new definition of ‘effectiveness’, we no longer need controlled clinical trials at all, if we want to demonstrate how effective alternative therapies truly are. We can just do observational studies without a control group, note that lots of patients get better, and ‘Bob is your uncle’!!! This is much faster, saves money, time and effort, and has the undeniable advantage of never generating a negative result.

To most outsiders, all this might seem a bit like splitting hair. However, I fear that it is far from that. In fact, it turns out to be a fairly fundamental issue in almost any discussion about the value or otherwise of alternative medicine. And, I think, it is also a matter of principle that reaches far beyond alternative medicine: if we allow various interest groups, lobbyists, sects, cults etc. to use their own definitions of fundamentally important terms, any dialogue, understanding or progress becomes almost impossible.

1 2 3 6
Recent Comments

Note that comments can now be edited for up to five minutes after they are first submitted.


Click here for a comprehensive list of recent comments.

Categories