study design
Excessive eccentric exercise of inadequately conditioned skeletal muscle results in focal sites of injury within the muscle fibres. 
These injuries cause pain which usually is greatest about 72 hours after the exercise. This type of pain is called delayed-onset muscle soreness (DOMS) and provides an accessible model for studying the effects of various treatments that are said to have anaesthetic activities; it can easily be reproducibly generated without lasting harm or ethical concerns.
In so-called alternative medicine (SCAM) DOMS is employed regularly to test treatments which are promoted for pain management. Thus several acupuncture trials using this method have become available. Yet, the evidence for the effects of acupuncture on DOMS is inconsistent which begs the question whether across all trials an effects emerges.
The aim of this systematic review therefore was to explore the effects of acupuncture on DOMS. Studies investigating the effect of acupuncture on DOMS in humans that were published before March 2020 were obtained from 8 electronic databases. The affected muscles, groups, acupuncture points, treatment sessions, assessments, assessment times, and outcomes of the included articles were reviewed. The data were extracted and analysed via a meta-analysis.
A total of 15 articles were included, and relief of DOMS-related pain was the primary outcome. The meta-analysis showed that there were no significant differences between acupuncture and sham/control groups, except for acupuncture for DOMS on day 1 (total SMD = -0.62; 95% CI = -1.12∼0.11, P < 0.05) by comparing with control groups.
The authors concluded that acupuncture for DOMS exhibited very-small-to-small and small-to-moderate effects on pain relief for the sham and no acupuncture conditions, respectively. Evidence indicating the effects of acupuncture on DOMS was little because the outcome data during the follow-up were insufficient to perform an effective meta-analysis.
A mere glance at the Forrest plot reveals that acupuncture is unlikely to have any effect on DOMS at all. The very small average effect that does emerge originates mainly from one outlier, the 2008 study by Itoh et al. This trial was published by three acupuncturists from the Department of Clinical Acupuncture and Moxibustion, Meiji University of Integrative Medicine, Kyoto, Japan. It has numerous weaknesses, for instance there are just 10 volunteers in each group, and can therefore be safely discarded.
In essence, this means that there is no good evidence that acupuncture is effective at reducing pain caused by DOMS.
This Cochrane review assessed the efficacy and safety of aromatherapy for people with dementia. The researchers included randomised controlled trials which compared fragrance from plants in an intervention defined as aromatherapy for people with dementia with placebo aromatherapy or with treatment as usual. All doses, frequencies and fragrances of aromatherapy were considered. Participants in the included studies had a diagnosis of dementia of any subtype and severity.
The investigators included 13 studies with 708 participants. All participants had dementia and in the 12 trials which described the setting, all were resident in institutional care facilities. Nine trials recruited participants because they had significant agitation or other behavioural and psychological symptoms in dementia (BPSD) at baseline. The fragrances used were:
- lavender (eight studies);
- lemon balm (four studies);
- lavender and lemon balm,
- lavender and orange,
- cedar extracts (one study each).
For six trials, assessment of risk of bias and extraction of results was hampered by poor reporting. Four of the other seven trials were at low risk of bias in all domains, but all were small (range 18 to 186 participants; median 66). The primary outcomes were:
- agitation,
- overall behavioural,
- psychological symptoms,
- adverse effects.
Ten trials assessed agitation using various scales. Among the 5 trials for which the confidence in the results was moderate or low, 4 trials reported no significant effect on agitation and one trial reported a significant benefit of aromatherapy. The other 5 trials either reported no useable data or the confidence in the results was very low. Eight trials assessed overall BPSD using the Neuropsychiatric Inventory and there was moderate or low confidence in the results of 5 of them. Of these, 4 reported significant benefit from aromatherapy and one reported no significant effect.
Adverse events were poorly reported or not reported at all in most trials. No more than two trials assessed each of our secondary outcomes of quality of life, mood, sleep, activities of daily living, caregiver burden. There was no evidence of benefit on these outcomes. Three trials assessed cognition: one did not report any data and the other two trials reported no significant effect of aromatherapy on cognition. The confidence in the results of these studies was low.
The authors reached the following conclusions: We have not found any convincing evidence that aromatherapy (or exposure to fragrant plant oils) is beneficial for people with dementia although there are many limitations to the data. Conduct or reporting problems in half of the included studies meant that they could not contribute to the conclusions. Results from the other studies were inconsistent. Harms were very poorly reported in the included studies. In order for clear conclusions to be drawn, better design and reporting and consistency of outcome measurement in future trials would be needed.
This is a thorough review. It makes many of the points that I so often make regarding SCAM research:
- too many of the primary studies are badly designed;
- too many of the primary studies are too small;
- too many of the primary studies are poorly reported;
- too many of the primary studies fail to mention adverse effects thus violating research ethics;
- too many of the primary studies are done by pseudo-scientists who use research for promotion rather than testing hypotheses.
It is time that SCAM researchers, ethic review boards, funders, editors and journal reviewers take these points into serious consideration – if only to avoid clinical research getting a bad reputation and losing the support of patients without which it cannot exist.
About one in three individuals have elevated blood pressure. This is bad news because hypertension is one of the most important risk factors for cardiovascular events like strokes and heart attacks. Luckily, there are many highly effective approaches for treating elevated blood pressure (diet, life-style, medication, etc.), and the drug management of hypertension has improved over the last few decades.
But unfortunately all anti-hypertensive drugs have side-effects and some patients look towards so-called alternative medicine (SCAM) to normalise their blood pressure. Therefore, we have to ask: are SCAMs effective treatments for hypertension? Because of the prevalence of hypertension, this is a question of great importance for public health.
In 2005, I addressed the issue by publishing a review entitled ‘Complementary/alternative medicine for hypertension: a mini-review‘. Here is its abstract:
Many hypertensive patients try complementary/alternative medicine for blood pressure control. Based on extensive electronic literature searches, the evidence from clinical trials is summarised. Numerous herbal remedies, non-herbal remedies and other approaches have been tested and some seem to have antihypertensive effects. The effect size is usually modest, and independent replications are frequently missing. The most encouraging data pertain to garlic, autogenic training, biofeedback and yoga. More research is required before firm recommendations can be offered.
Since the publication of this paper, more systematic reviews have become available. In order to get an overview of this evidence, I conducted a few simple Medline searches for systematic reviews (SRs) of SCAM published between 2005 and today. I included only SRs that were focussed on just one specific therapy as a treatment of just one specific condition, namely hypertension (omitting SRs with titles such as ‘Alternative treatments for cardiovascular conditions’). Reviews on prevention were also excluded. Here is what I found (the conclusions of each SR is quoted verbatim):
- A 2020 SR of auricular acupressure including 18 RCTs: The results demonstrated a favorable effect of auricular acupressure to reduce blood pressure and improve sleep in patients with hypertension and insomnia. Further studies to better understand the acupoints and intervention times of auricular acupressure are warranted.
- A 2020 SR of Chinese herbal medicines (CHM) including 30 studies: CHM combined with conventional Western medicine may be effective in lowering blood pressure and improving vascular endothelial function in patients with hypertension.
- A 2020 SR of Tai chi including 28 RCTs: Tai Chi could be recommended as an adjuvant treatment for hypertension, especially for patients less than 50 years old.
- A 2020 SR of Tai chi including 13 trials: Tai chi is an effective physical exercise in treating essential hypertension compared with control interventions.
- A 2020 SR of Tai chi including 31 controlled clinical trials: Tai Ji Quan is a viable antihypertensive lifestyle therapy that produces clinically meaningful BP reductions (i.e., 10.4 mmHg and 4.0 mmHg of SBP and DBP reductions, respectively) among individuals with hypertension.
- A 2020 SR of pycnogenol including 7 trials: the present meta-analysis does not suggest any significant effect of pycnogenol on BP.
- A 2019 SR of Policosanol including 19 studies: Policosanol could lower SBP and DBP significantly; future long term studies are required to confirm these findings in the general population.
- A 2019 SR of dietary phosphorus including 14 studies: We found no consistent association between total dietary phosphorus intake and BP in adults in the published literature nor any randomized trials designed to examine this association.
- A 2019 SR of ginger including 6 RCTs: ginger supplementation has favorable effects on BP.
- A 2019 SR of corn silk tea (CST) including 5 RCTs: limited evidence showed that CST plus antihypertensive drugs might be more effective in lowering blood pressure compared with antihypertensive drugs alone.
- A 2019 SR of blood letting including 7 RCTs: no definite conclusions regarding the efficacy and safety of BLT as complementary and alternative approach for treatment of hypertension could be drew due to the generally poor methodological design, significant heterogeneity, and insufficient clinical data.
- A 2019 SR of Xiao Yao San (XYS) including 17 trials: XYS adjuvant to antihypertensive drugs maybe beneficial for hypertensive patients in lowering BP, improving depression, regulating blood lipids, and inhibiting inflammation.
- A 2019 SR of Chinese herbal medicines including 9 RCTs: Chinese herbal medicine as complementary therapy maybe beneficial for postmenopausal hypertension.
- A 2019 Cochrane review of guided imagery including 2 trials: There is insufficient evidence to inform practice about the use of guided imagery for hypertension in pregnancy.
- A 2019 Cochrane review of acupuncture including 22 RCTs: At present, there is no evidence for the sustained BP lowering effect of acupuncture that is required for the management of chronically elevated BP.
- A 2019 SR of wet cupping including 7 RCTs: no firm conclusions can be drawn and no clinical recommendations made.
- A 2019 SR of transcendental meditation (TM) including 9 studies: TM was associated with within-group (but not between-groups) improvements in BP.
- A 2019 SR of yoga including 49 trials: yoga is a viable antihypertensive lifestyle therapy that produces the greatest BP benefits when breathing techniques and meditation/mental relaxation are included.
- A 2018 SR of mindfulness-based stress reduction (MBSR) including 5 studies: The MBSR program is a promising behavioral complementary therapy to help people with hypertension lower their blood pressure
- A 2018 SR of beetroot juice (BRJ) including 11 studies: BRJ supplementation should be promoted as a key component of a healthy lifestyle to control blood pressure in healthy and hypertensive individuals.
- A 2018 SR of taurine including 7 studies: ingestion of taurine at the stated doses and supplementation periods can reduce blood pressure to a clinically relevant magnitude, without any adverse side effects.
- A 2018 SR of acupuncture including 30 RCTs: there is inadequate high quality evidence that acupuncture therapy is useful in treating hypertension.
- A 2018 SR of co-enzyme Q10 including 17 RCTs: CoQ10 supplementation may result in reduction in SBP levels, but did not affect DBP levels among patients with metabolic diseases.
- A 2018 SR of a traditional Chinese formula Longdanxiegan decoction (LDXGD) including 9 trials: Due to poor methodological quality of the included trials, as well as potential reporting bias, our review found no conclusive evidence for the effectiveness of LDXGD in treating hypertension.
- A 2018 SR of viscous fibre including 22 RCTs: Viscous soluble fiber has an overall lowering effect on SBP and DBP.
- A 2017 SR of yoga breathing exercise (pranayama) including 13 studies: The pranayama’s effect on BP were not robust against selection bias due to the low quality of studies. But, the lowering BP effect of pranayama is encouraging.
- A 2017 SR of dietary nitrate supplementation including 13 trials: Positive effects of medium-term dietary nitrate supplementation on BP were only observed in clinical settings, which were not corroborated by more accurate methods such as 24-h ambulatory and daily home monitorings.
- A 2017 SR of Vitamin D supplementation including 8 RCTs: vitamin D is not an antihypertensive agent although it has a moderate SBP lowering effect.
- A 2017 SR of pomegranate including 8 RCTs: The limited evidence from clinical trials to date fails to convincingly show a beneficial effect of pomegranate on blood pressure
- A 2017 SR of ‘forest bathing’ including 20 trials: This systematic review shows a significant effect of Shinrin-yoku on reduction of blood pressure.
- A 2017 SR of Niuhuang Jiangya Preparation (NHJYP) including 12 RCTs: Our review indicated that NHJYP has some beneficial effects in EH patients with liver-yang hyperactivity and abundant phlegm-heat syndrome.
- A 2017 SR of Chinese medicines (CM) including 24 studies: CM might be a promising approach for the elderly with isolated systolic hypertension, while the evidence for CM employed alone was insufficient.
- A 2017 SR of beetroot juice including 22 RCTs: Our results demonstrate the blood pressure-lowering effects of beetroot juice and highlight its potential NO3-independent effects.
- A 2017 SR of blueberry including 6 RCTs: the results from this meta-analysis do not favor any clinical efficacy of blueberry supplementation in improving BP
- A 2016 Cochrane review of co-enzyme Q10 including 3 RCTs: This review provides moderate-quality evidence that coenzyme Q10 does not have a clinically significant effect on blood pressure.
- A 2016 SR of Nigella sativa including 11 RCTs: short-term treatment with N. sativa powder can significantly reduce SBP and DBP levels.
- A 2016 SR of vitamin D3 supplementation including 30 RCTs: Supplementation may be beneficial at daily doses >800 IU/day for <6 months in subjects ≥50 years old.
- A 2016 SR of anthocyanin supplementation including 6 studies: results from this meta-analysis do not favor any clinical efficacy of supplementation with anthocyanins in improving blood pressure.
- A 2016 SR of flaxseed including 15 trials: This meta-analysis of RCTs showed significant reductions in both SBP and DBP following supplementation with various flaxseed products.
- A 2016 SR of massage therapy including 9 RCTs: This systematic review found a medium effect of massage on SBP and a small effect on DBP in patients with hypertension or prehypertension.
- A 2015 SR of massage therapy including 24 studies: There is some encouraging evidence of massage for essential hypertension.
- A 2015 SR of transcendental meditation (TM) including 12 studies: an approximate reduction of systolic and diastolic BP of -4.26 mm Hg (95% CI=-6.06, -2.23) and -2.33 mm Hg (95% CI=-3.70, -0.97), respectively, in TM groups compared with control groups.
- A 2015 SR of Zhen Wu Decoction (ZWD) including 7 trials: This systematic review revealed no definite conclusion about the application of ZWD for hypertension due to the poor methodological quality, high risk of bias, and inadequate reporting on clinical data.
- A 2015 SR of acupuncture including 23 RCTs: Our review provided evidence of acupuncture as an adjunctive therapy to medication for treating hypertension, while the evidence for acupuncture alone lowing BP is insufficient.
- A 2015 SR of xuefu zhuyu decoction (XZD) including 15 studies: This meta-analysis provides evidence that XZD is beneficial for hypertension.
- A 2015 SR of Shenqi pill including 4 RCTs: This systematic review firstly provided no definite evidence for the efficacy and safety of Shenqi pill for hypertension based on the insufficient data.
- A 2015 SR of Jian Ling Decoction (JLD) including 10 trials: Owing to insufficient clinical data, it is difficult to draw a definite conclusion regarding the effectiveness and safety of JLD for essential hypertension.
- A 2015 SR of Chinese herbal medicines (CHM) including 5 trials: No definite conclusions about the effectiveness and safety of CHM for resistant hypertension could be drawn.
- A 2015 SR of Chinese medicines (CM) including 27 RCTs: When combined with Western medines, CM as a complementary treatment approach has certain effects for the control of hypertension and protection of target organs.
- A 2015 SR of berberine including 17 RCTs: This study indicates that berberine has comparable therapeutic effect on type 2 DM, hyperlipidemia and hypertension with no serious side effect.
- A 2015 SR of garlic including 9 double-blind trials: Although evidence from this review suggests that garlic preparations may lower BP in hypertensive individuals, the evidence is not strong.
- A 2015 SR of chlorogenic acids (CGAs) including 5 studies: CGA intake causes statistically significant reductions in systolic and diastolic blood pressures.
- A 2014 SR of omega-3 fatty acid supplementation including 70 RCTs: provision of EPA+DHA reduces systolic blood pressure, while provision of ≥2 grams reduces diastolic blood pressure.
- A 2014 SR of green tea including 20 RCTs: Green tea intake results in significant reductions in systolic blood pressure
- A 2014 SR of probiotics including 9 studies: consuming probiotics may improve BP by a modest degree, with a potentially greater effect when baseline BP is elevated, multiple species of probiotics are consumed, the duration of intervention is ≥8 weeks, or daily consumption dose is ≥10(11) colony-forming units.
- A 2014 SR of yoga including 17 trials: The evidence for the effectiveness of yoga as a treatment of hypertension is encouraging but inconclusive.
- A 2014 SR of yoga including 7 RCTs: very low-quality evidence was found for effects of yoga on systolic and diastolic blood pressure.
- A 2014 SR of yoga including 120 studies: yoga is an effective adjunct therapy for HPT and worthy of inclusion in clinical guidelines.
- A 2014 SR of moxibustion: a beneficial effect of using moxibustion interventions on KI 1 to lower blood pressure compared to antihypertensive drugs.
- A 2014 SR of acupuncture including 4 sham-controlled RCTs: acupuncture significantly lowers blood pressure in patients taking antihypertensive medications.
- A 2014 SR of Tuina including 7 RCTs: The findings from our review suggest that Tuina might be a beneficial adjuvant for patients with EH
- A 2014 SR of ‘kidney tonifying’ (KT) Chinese herbal mixture including 6 studies: Compared with antihypertensive drugs alone, KT formula combined with antihypertensive drugs may provide more benefits for patients with SH.
- A 2014 SR of Tongxinluo capsule including 25 studies : There is some but weak evidence about the effectiveness of TXL in treating patients with hypertension.
- A 2014 SR of moxibustion including 5 RCTs: no confirm conclusion about the effectiveness and safety of moxibustion as adjunctive treatment for essential hypertension could be made
- A 2013 SR of Qi Ju Di Huang Wan (QJDHW) including 10 RCTs: QJDHW combined with antihypertensive drugs might be an effective treatment for lowering blood pressure and improving symptoms in patients with essential hypertension.
- A 2013 SR of yoga including 17 studies: Yoga can be preliminarily recommended as an effective intervention for reducing blood pressure.
- A 2013 SR of Tianma Gouteng Yin (TGY) including 22 RCTs: No confirmed conclusion about the effectiveness and safety of TGY as adjunctive treatment for essential hypertension … could be made.
- A 2013 SR of Zhen Gan Xi Feng Decoction (ZGXFD) including 6 RCTs: ZGXFD appears to be effective in improving blood pressure and hypertension-related symptoms for EH
- A 2013 SR of Tianmagouteng decoction including 9 RCTs: Tianmagouteng decoction can decrease both systolic and diastolic blood pressure.
- A 2013 SR of fish oil including 17 RCTs: The small but statistically significant effects of fish-oil supplements in hypertensive participants in this review have important implications for population health and lowering the risk of stroke and ischaemic heart disease.
- A 2013 SR of acupuncture including 35 RCTs: While there are some evidences that suggest potential effectiveness of acupuncture for hypertension, the results were limited by the methodological flaws of the studies.
- A 2013 SR of yoga including 6 studies: There is some encouraging evidence of yoga for lowering SBP and DBP.
- A 2012 SR of spinal manipulation therapy (SMT) including 10 studies: There is currently a lack of low bias evidence to support the use of SMT as a therapy for the treatment of
- A 2012 SR of vitamin C including 29 trials: In short-term trials, vitamin C supplementation reduced SBP and DBP.
- A 2012 SR of magnesium supplementation including 22 trials: magnesium supplementation appears to achieve a small but clinically significant reduction in BP, an effect worthy of future prospective large randomised trials using solid methodology.
- A 2012 SR of Banxia Baizhu Tianma Decoction (BBTD) including 16 RCTs: There is encouraging evidence of BBTD for lowering SBP, but evidence remains weak.
- A 2012 SR of Liu Wei Di Huang Wan (LWDHW) including 6 RCTs: LWDHW combined with antihypertensive drugs appears to be effective in improving blood pressure and symptoms in patients with essential hypertension.
- A 2012 SR of aromatherapy including 5 studies: The existing trial evidence does not show convincingly that aromatherapy is effective for hypertension.
- A 2012 empty Cochrane review: As no trials could be identified, no conclusions can be made about the role of TGYF in the treatment of primary hypertension.
- A 2012 SR of yoga including 10 studies: Not only does yoga reduce high BP but it has also been demonstrated to effectively reduce blood glucose level, cholesterol level, and body weight, major problems affecting the American society.
- A 2011 SR of L-arginine including 11 RCTs: This meta-analysis provides further evidence that oral L-arginine supplementation significantly lowers both systolic and diastolic BP.
- A 2011 SR of soy isoflavones including 14 RCTs: Soy isoflavone extracts significantly decreased SBP but not DBP in adult humans, and no dose-response relationship was observed.
- A 2010 SR of moxibustion including 4 RCTs: There is insufficient evidence to suggest that moxibustion is an effective treatment for hypertension.
- A 2010 SR of acupunctures including 20 studies: Because of the paucity of rigorous trials and the mixed results, these findings result in limited conclusions. More rigorously designed and powered studies are needed.
- A 2010 SR of cupping including 3 trials: the evidence is not significantly convincing to suggest cupping is effective for treating hypertension.
- A 2010 empty Cochrane review: There is insufficient evidence to support the benefit of Roselle for either controlling or lowering blood pressure in patients with hypertension.
- A 2009 SR of acupuncture including 11 RCTs: the notion that acupuncture may lower high BP is inconclusive.
- A 2008 SR of transcendental meditation including 9 studies: The regular practice of Transcendental Meditation may have the potential to reduce systolic and diastolic blood pressure by approximately 4.7 and 3.2 mm Hg, respectively.
- A 2008 SR of relaxation therapies including 25 trials: the evidence in favour of a causal association between relaxation and blood pressure reduction is weak.
- A 2007 SR of qigong including 12 RCTs: There is some encouraging evidence of qigong for lowering SBP, but the conclusiveness of these findings is limited.
- A 2007 SR of co-enzyme Q10 including 12 trials: coenzyme Q10 has the potential in hypertensive patients to lower systolic blood pressure by up to 17 mm Hg and diastolic blood pressure by up to 10 mm Hg without significant side effects.
- A 2007 SR of stress reduction programs including 106 studies: Available evidence indicates that among stress reduction approaches, the Transcendental Meditation program is associated with significant reductions in BP.
- A 2006 Cochrance review of magnesium supplementation including 12 RCTs: the evidence in favour of a causal association between magnesium supplementation and blood pressure reduction is weak and is probably due to bias.
- A 2006 Cochrane review of calcium supplementation including 13 RCTs: evidence in favour of causal association between calcium supplementation and blood pressure reduction is weak and is probably due to bias.
ALMOST 100 NEW SRs!
To be honest, if I had known the volume of the material, I would probably not have tackled this task. Since the publication of my mini-review in 2005, there has been an explosion of similar papers:
- 1 in 2005
- 2 in 2006
- 3 in 2007
- 2 in 2008
- 1 in 2009
- 4 in 2010
- 2 in 2011
- 8 in 2012
- 8 in 2013
- 12 in 2014
- 12 in 2015
- 6 in 2016
- 9 in 2017
- 7 in 2018
- 12 in 2019
As this is based on very simple Medline searches, the list is certainly not complete. Despite this fact, several conclusions seem to emerge:
- There is no shortage of SCAMs that have been tested for hypertension.
- Most seem to have positive effects; in many cases, they seem too good to be true.
- Many of the SRs are of poor methodological quality, based on poor quality primary studies, published in less than reputable journals. Some SRs, for instance, include studies without a control group which is likely to lead to false-positive overall conclusions about the effectiveness of the SCAM in question.
- In recent years, there are more and more SRs by Chinese authors focussed on Chinese herbal mixtures that are unknown and unobtainable outside China. These SRs are invariably based on studies published in Chinese language in journals that are inaccessible. This means it is almost impossible for the reader, reviewer or editor to check their accuracy. The reliability of the conclusions of these SRs must therefore be doubted.
- Most of the primary studies included in the SRs lack long-term data. Thus the usefulness of the SCAM in question is questionable.
- With several of the SCAMs, the dose of the treatment and treatment schedule is less than clear. For instance, one might ask how frequently a patient should have acupuncture to control her hypertension.
- Some of the SCAMs assessed in these SRs seem of doubtful practicality. For instance, it might not be feasible nor economical for patients to receive regular acupuncture to manage their blood pressure.
- Several contradictions emerge from some of the SRs of the same modality. This is particularly confusing because SRs are supposed to be the most reliable type of evidence. In most instances, however, the explanation can easily be found by looking at the quality of the SRs. If SRs are based on uncontrolled studies, or if they fail to critically evaluate the reliability of the included primary trials, they are likely to arrive at conclusions that are too positive. Examples for such confusion are the multiple SRs of co-enzyme Q10 or the three yoga SRs of 2014.
- Because of this confusion, SCAM advocates are able to select false-positive SRs to support their opinion that SCAM is effective.
- Despite a substantial amount of positive evidence, none of the SCAMs have become part of the routine in the management of hypertension. A 2013 statement by the American Heart Association entitled Beyond medications and diet: alternative approaches to lowering blood pressure: a scientific statement from the american heart association concluded that it is reasonable for all individuals with blood pressure levels >120/80 mm Hg to consider trials of alternative approaches as adjuvant methods to help lower blood pressure when clinically appropriate. A suggested management algorithm is provided, along with recommendations for prioritizing the use of the individual approaches in clinical practice based on their level of evidence for blood pressure lowering, risk-to-benefit ratio, potential ancillary health benefits, and practicality in a real-world setting.
What lessons might this brief overview of SRs teach us? I think the following points are worth considering:
- Systematic reviews are the best type of evidence we have for estimating the effectiveness of treatments. But it is essential that they include a strong element of CRITICAL evaluation of the primary studies. Without it, a SR is incomplete and potentially counter-productive.
- The primary studies of SCAM are far too often of poor quality. This means that researchers should thrive to improve the rigour of their investigations.
- Both poor-quality primary studies and uncritically conducted SRs are prone to yielding findings that are too good to be true.
- Editors and reviewers have a responsibility to prevent the publication of trials and SRs that are of poor quality and thus likely to mislead us.
- Those SCAMs that have shown promising effects on hypertension (for instance Tai chi) should now be submitted to further independent scrutiny to find out whether their efficacy and usefulness can be confirmed, for instance, by 24-h ambulatory and daily home blood pressure monitoring and studies testing their acceptability in real life settings. Subsequently, we ought to determine whether the SCAM in question can be reasonably integrated in routine blood pressure management.
- The adjunctive use of a SCAM that has been proven to be effective and practical seems a reasonable approach. Yet, it requires proper scientific scrutiny.
- There is a paucity of cost-effectiveness studies and investigations of the risks of SCAM which needs to be addressed before any SCAM is considered for routine care.
Tasuki is a sort of sash for holding up the sleeves on a kimono. It also retracts the shoulders and keeps the head straight up. By correcting the wearer’s posture, it might even prevent or treat neck pain. The greater the forward head posture, for example, the more frequent are neck problems. However, there is little clinical evidence to support or refute this hypothesis.
This study was conducted to determine whether Tasuki-style posture supporter improves neck pain compared to waiting-list. It was designed as an individually-randomized, open-label, waiting-list-controlled study. Adults with non-specific chronic neck pain who reported 10 points or more on modified Neck Disability Index (mNDI: range, 0-50; higher points indicate worse condition) were enrolled. Participants were randomly assigned 1:1 to the intervention group or to a waiting-list control group. The primary outcome was the change in mNDI at 1 week.
In total, 50 participants were enrolled. Of these participants, 26 (52%) were randomly assigned to the intervention group and 24 to the waiting-list. Attrition rate was low in both groups (1/50). The mean mNDI change score at 1 week was more favourable for Tasuki than waiting-list (between-group difference, -3.5 points (95% confidence interval (CI), -5.3 to -1.8); P = .0002). More participants (58%) had moderate benefit (at least 30% improvement) with Tasuki than with waiting-list (13%) (relative risk 4.6 (95% CI 1.5 to 14); risk difference 0.45 (0.22 to 0.68)).
The author concluded that this trial suggests that wearing Tasuki might moderately improve neck pain. With its low-cost, low-risk, and easy-to-use nature, Tasuki could be an option for those who suffer from neck pain.
In the previous two posts, we discussed how lamentably weak the evidence for acupuncture and spinal manipulation is regarding the management of pain such as ‘mechanical’ neck pain. Here we have a well-reported study with a poor design (no control for non-specific effects) which seems to suggest that simply wearing a Tasuki is just as effective as acupuncture or spinal manipulation.
What is the lesson from this collective evidence?
Is it that we should forget about acupuncture and spinal manipulation for chronic neck pain?
Perhaps.
Or is it that poor trial designs generate unreliable evidence?
More likely.
Or is it that any treatment, however daft, will generate positive outcomes, if the researchers are sufficiently convinced of its benefit?
Yes, I think so.
___________________
PS
If you had chronic neck pain, would you rather have your neck manipulated, needles stuck into your body, or get a Tasuki? (Spoiler: Tasuki is risk-free, the other two treatments are not!)
Together with a co-worker, Prof Walach conducted a systematic review of mistletoe extracts (Rudolf Steiner’s anti-cancer drug) as a treatment for improving the quality of life (QoL) of cancer patients. They included all prospective controlled trials that compared mistletoe extracts with a control in cancer patients and reported QoL or related dimensions.
Walach included 26 publications with 30 data sets. The studies were heterogeneous. The pooled standardized mean difference (random effects model) for global QoL after treatment with mistletoe extracts vs. control was d = 0.61 (95% CI 0.41-0.81, p < 0,00001). The effect was stronger for younger patients, with longer treatment, in studies with lower risk of bias, in randomized and blinded studies. Sensitivity analyses supported the validity of the finding. 50% of the QoL subdomains (e.g. pain, nausea) showed a significant improvement after mistletoe treatment. Most studies had a high risk of bias or at least raise some concern.
The authors concluded that mistletoe extracts produce a significant, medium-sized effect on QoL in cancer. Risk of bias in the analyzed studies is likely due to the specific type of treatment, which is difficult to blind; yet this risk is unlikely to affect the outcome.
This is a surprising conclusion, not least because – as reported on this blog – only a year ago another German team of researchers conducted a similar review and came to a very different conclusion. Here is their abstract again:
Purpose: One important goal of any cancer therapy is to improve or maintain quality of life. In this context, mistletoe treatment is discussed to be highly controversial. The aim of this systematic review is to give an extensive overview about the current state of evidence concerning mistletoe therapy of oncologic patients regarding quality of life and side effects of cancer treatments.
Methods: In September and October 2017, Medline, Embase, Cochrane Central Register of Controlled Trials (CENTRAL), PsycINFO, CINAHL and “Science Citation Index Expanded” (Web of Science) were systematically searched.
Results: The search strategy identified 3647 articles and 28 publications with 2639 patients were finally included in this review. Mistletoe was used in bladder cancer, breast cancer, other gynecological cancers (cervical cancer, corpus uteri cancer, and ovarian cancer), colorectal cancer, other gastrointestinal cancer (gastric cancer and pancreatic cancer), glioma, head and neck cancer, lung cancer, melanoma and osteosarcoma. In nearly all studies, mistletoe was added to a conventional therapy. Regarding quality of life, 17 publications reported results. Studies with better methodological quality show less or no effects on quality of life.
Conclusions: With respect to quality of life or reduction of treatment-associated side effects, a thorough review of the literature does not provide any indication to prescribe mistletoe to patients with cancer.
How can this discrepancy be explained? Which of the reviews is drawing the correct conclusion? Here are some relevant details that could help finding an answer to these questions:
- Walach is a psychologist by training, while the senior author of the 2019 review, Jutta Huebner, is an oncologist.
- Huebner included only randomised clinical trials (RCTs), whereas Walach included any interventional and non-interventional prospective controlled study.
- Huebner included 17 RCTs that reported QoL data, while Walach included 26 publications with 30 data sets including 5 non-randomised studies.
- Several of the primary studies had been published multiple times at different stages of completion. Walach included these as independent data sets, while Huebner included each study only once.
- Huebner looked at QoL, whereas Walach also considered measurements of self-regulation as outcome measures.
- Both reviews point out that the methodological quality of the primary studies was often poor; Walach drew a positive conclusion regardless, while Huebner did not and pointed out that studies with better methodology show less or no effects on quality of life or side effects of cancer therapy.
- Walach’s review was funded by funded by the Förderverein komplementärmedizinische Forschung, Arlesheim, Switzerland, a lobby group for mistletoe therapy, while Huebner’s work was funded by the German Guideline “S3 Leitlinie Komplementärmedizin in der Behandlung von onkologischen PatientInnen (Registernummer 032-055OL)” funded by the German Cancer Aid (Fördernummer 11583) within the German Guideline Program in Oncology and by the working group Prevention and Integrative Oncology of the German Cancer Society.
I am sure there are other important differences, but the ones listed above suffice, I think, to decide which of the two papers is trustworthy and which is not.
This analysis was aimed at assessing the associations of acupuncture use with mortality, readmission and reoperation rates in hip fracture patients using a longitudinal population-based database. A retrospective matched cohort study was conducted using data for the years 1996-2012 from Taiwan’s National Health Insurance Research Database. Hip fracture patients were divided into:
- an acupuncture group consisting of 292 subjects who received at least 6 acupuncture treatments within 183 days of hip fracture,
- and a propensity score matched “no acupuncture” group of 876 subjects who did not receive any acupuncture treatment and who functioned as controls.
The two groups were compared using survival analysis and competing risk analysis.
Compared to non-treated subjects, subjects treated with acupuncture had
- a lower risk of overall death (hazard ratio (HR): 0.41, 95% confidence interval (CI): 0.24-0.73, p = 0.002),
- a lower risk of readmission due to medical complications (subdistribution HR (sHR): 0.64, 95% CI: 0.44-0.93, p = 0.019)
- and a lower risk of reoperation due to surgical complications (sHR: 0.62, 95% CI: 0.40-0.96, p = 0.034).
The authors concluded that postoperative acupuncture in hip fracture patients is associated with significantly lower mortality, readmission and reoperation rates compared with those of matched controls.
That’s a clear and neat finding; the question is, what does it mean?
Here are a few possibilities for consideration:
- As a result of having at least 6 acupuncture sessions, patients had lower rates of mortality, readmission and reoperation.
- As a result of having lower rates of mortality, readmission and reoperation, patients used acupuncture.
- As a result of some other factor, patients had both lower rates of mortality, readmission and reoperation and at least 6 sessions of acupuncture.
Which of the three possibilities is the most likely?
- Some enthusiasts might think that acupuncture makes you live longer. But does anyone truly believe it reduces the likelihood of needing a reoperation? Seriously? Well, I don’t see even a hint of a mechanism by which acupuncture might achieve this. Therefore, I would categorise this possibility as highly unlikely.
- It stands to reason that patients who are alive and well use more acupuncture than those who are dead or in need of surgery. So, this possibility is not entirely inconceivable.
- It seems very likely that people who are more health conscious might use acupuncture and live longer, need less readmissions or surgery. No doubt, this possibility is by far the best explanation of the findings of this retrospective matched cohort study.
If that is so, does this paper tell us anything useful at all?
Not really (that’s why it was published in an acupuncture journal which few people would read)
On second thought, perhaps it does tell us something valuable: retrospective matched cohort studies are hopeless when it comes to establishing cause and effect!
Acupressure is the stimulation of specific points, called acupoints, on the body surface by pressure for therapeutic purposes. The required pressure can be applied manually of by a range of devices. Acupressure is based on the same tradition and assumptions as acupuncture. Like acupuncture, it is often promoted as a panacea, a ‘cure-all’.
Several systematic reviews of the clinical trials of acupressure have been published. An overview published in 2010 included 9 such papers and concluded that the effectiveness of this treatment has not been conclusively demonstrated for any condition.
But since 2010, more trials have become available.
Do they change the overall picture?
The objective of this study was to test the efficacy of acupressure on patient-reported postoperative recovery. The researchers conducted a single centre, three-group, blind, randomised controlled, pragmatic trial assessing acupressure therapy on the PC6, LI4 and HT7 acupoints. Postoperative patients expected to stay in hospital at least 2 days after surgery were included and randomised to three groups:
- In the acupressure group, pressure was applied for 6 min (2 min per acupoint), three times a day after surgery for a maximum of 2 postoperative days during the hospital stay.
- In the sham group, extremely light touch was applied to the acupoints.
- The third group did not receive any such intervention.
All patients also received the normal postoperative treatments.
The primary outcome was the change in the quality of recovery (QoR), using the QoR-15 questionnaire, between postoperative days 1 and 3. Key secondary outcomes included patients’ satisfaction, postoperative nausea and vomiting, pain score and opioid (morphine equivalent) consumption. Assessors for the primary and secondary endpoints were blind to the group allocation.
A total of 163 patients were randomised (acupressure n=55, sham n=53, no intervention n=55). The mean (SD) postoperative change in QoR-15 did not differ statistically (P = 0.27) between the acupressure, sham and no intervention groups: 15.2 (17.8), 14.2 (21.9), 9.2 (21.7), respectively. Patient satisfaction (on a 0 to 10 scale) was statistically different (P = 0.01) among these three groups: 9.1 (1.5), 8.4 (1.6) and 8.2 (2.2), respectively. Changes in pain score and morphine equivalent consumption were not significantly different between the groups.
The authors concluded that two days of postoperative acupressure therapy (up to six treatments) did not significantly improve patient QoR, postoperative nausea and vomiting, pain score or opioid consumption. Acupressure, however, was associated with improved patient satisfaction.
This study is a good example to show why it is so difficult (or even impossible) to use a clinical trial for demonstrating the ineffectiveness of a therapy for any given condition. The above trial fails to show that acupressure had a positive effect on the primary outcome measure. Acupressure fans will, however, claim that:
- there was a positive effect on patient satisfaction,
- the treatment was too intense/long,
- the treatment was not intense/long enough,
- the wrong points were used,
- the sample size was too small,
- the patients were too ill,
- the patients were not ill enough,
- etc., etc.
In the end, such discussions often turn out to be little more than a game of pigeon chess. Perhaps it is best to ask before planning such a trial:
IS THE ASSUMPTION THAT THE TREATMENT WORKS FOR THIS CONDITION PLAUSIBLE?
If the answer is no, why do the study in the first place?
When tested rigorously, the evidence for so-called alternatives medicine (SCAM) is usually weak or even negative. This fact has prompted many SCAM enthusiasts to become utterly disenchanted with rigorous tests such as the randomised clinical trial (RCT). They seem to think that, if the RCT fails to generate the findings we want, let’s use different methodologies instead. In other words, they are in favour of observational studies which often yield positive results.
This line of thinking is prevalent in all forms of SCAM, but probably nowhere more so that in the realm of homeopathy. Homeopaths see that rigorous RCTs tend not to confirm their belief and, to avoid cognitive dissonance, they focus on observational studies which are much more likely to confirm their belief.
In this context, it is worth mentioning a recent article where well-known homeopathy enthusiasts have addressed the issue of observational studies. Here is their abstract:
Background: Randomized placebo-controlled trials are considered to be the gold standard in clinical research and have the highest importance in the hierarchical system of evidence-based medicine. However, from the viewpoint of decision makers, due to lower external validity, practical results of efficacy research are often not in line with the huge investments made over decades.
Method: We conducted a narrative review. With a special focus on homeopathy, we give an overview on cohort, comparative cohort, case-control and cross-sectional study designs and explain guidelines and tools that help to improve the quality of observational studies, such as the STROBE Statement, RECORD, GRACE and ENCePP Guide.
Results: Within the conventional medical research field, two types of arguments have been employed in favor of observational studies. First, observational studies allow for a more generalizable and robust estimation of effects in clinical practice, and if cohorts are large enough, there is no over-estimation of effect sizes, as is often feared. We argue that observational research is needed to balance the current over-emphasis on internal validity at the expense of external validity. Thus, observational research can be considered an important research tool to describe “real-world” care settings and can assist with the design and inform the results of randomised controlled trails.
Conclusions: We present recommendations for designing, conducting and reporting observational studies in homeopathy and provide recommendations to complement the STROBE Statement for homeopathic observational studies.
In their paper, the authors state this:
It is important to realize three areas where observational research can be valuable. For one, as already mentioned, it can be valuable as a preparatory type of research for designing good randomized studies. Second, it can be valuable as a stand-alone type of research, where pragmatic or ethical reasons stand against conducting a randomized study. Additionally, it can be valuable as the only adequate method where choices are involved: for instance, in any type of lifestyle research or where patients have very strong preferences, such as in homeopathy and other CAM. This might also lead to a diversification of research efforts and a broader, more realistic, picture of the effects of therapeutic interventions.
My comments to this are as follows:
- Observational research can be valuable as a preparatory type of research for designing good randomized studies. This purpose is better fulfilled by pilot studies (which are often abused in SCAM).
- Observational research can be valuable as a stand-alone type of research, where pragmatic or ethical reasons stand against conducting a randomized study. Such situations rarely arise in the realm of SCAM.
- Observational research can be valuable as the only adequate method where choices are involved: for instance, in any type of lifestyle research or where patients have very strong preferences, such as in homeopathy and other CAM. I fail to see that this is true.
- Observational research leads to a diversification of research efforts and a broader, more realistic, picture of the effects of therapeutic interventions. The main aim of research into the effectiveness of SCAM should be, in my view, to determine whether the treatment per se works or not. Observational studies are likely to obscure the truth on this issue.
Don’t get me wrong, I am not saying that observational studies are useless; quite to the contrary, they can provide very important information. But what I am trying to express is this:
- We should not allow double standards in medical research. The standards and issues of observational research as they exist in conventional medicine must also apply to SCAM.
- Observational studies cannot easily determine cause and effect between the therapy and the outcome.
- Observational studies cannot be a substitute for RCTs.
- Depending on their exact design, observational studies measure the outcome caused by a whole range of factors, including the therapy per se, the placebo-effect, the natural history of the disease, the regression towards the mean.
- Observational studies are particularly useful in effectiveness research, AFTER the efficacy of a therapy has been established by RCTs.
- If RCT fail to show that a therapy is effective and observational studies seem to indicate that they work, the therapy in question is probably a placebo.
- SCAM-enthusiasts’ preference for observational studies is transparently due to motivated reasoning.
Am I the only one who is tired of hearing that, in India, homeopathy is doing wonders for the current pandemic? All of the reports that I have seen are based on little more than hearsay, anecdotes or pseudo-science. If anyone really wanted to find out whether homeopathy works, they would need more than that; in fact, they would need to conduct a clinical trial.
But wait!
As it happens, there are already ~500 clinical trials of homeopathy. Many show positive effects, but the reliable ones usually don’t. Crucially, the totality of the evidence fails to be positive. So, running further studies is hardly a promising exercise. In fact, considering how utterly implausible homeopathy is, it even seems like an unethical waste of resources.
But many homeopaths disagree, particularly those in India. And it has been reported that several trials have been given the go-ahead in India and are now up and running. This regrettable fact is being heavily exploited for swaying public opinion in favour of homeopathy. The way I see it, the situation is roughly this:
- a few trials of homeopathy are being set up;
- they are designed by enthusiasts of homeopathy who lack research expertise;
- therefore their methodology is weak and biased towards generating a false-positive result;
- while this is going on, the homeopathic propaganda machine is running overtime;
- when the results will finally emerge, they will get published in a 3rd rate journal;
- homeopaths worldwide will celebrate them as a triumph for homeopathy;
- critical thinkers will be dismayed at their quality and will declare that the conclusions drawn by over-enthusiastic homeopaths are not valid;
- in the end, we will be exactly where we were before: quasi-religious believers in homeopathy will feel vexed because their findings are not accepted in science, and everyone else will be baffled by the waste of time, opportunity and resources as well as by the tenacity of homeopaths to make fools of themselves.
But criticising is easy; doing it properly is often more difficult.
So, how should it be done?
The way I see it, one should do the following:
- carefully consider the implausibility of homeopathy;
- thoroughly study the existing evidence on homeopathy;
- abandon all plans to study homeopathy in the light of the above.
But this hardly is inconceivable considering the current situation in India. If further studies of homeopathy are unavoidable, the following procedure might therefore be reasonable:
- assemble a team of experts including trial methodologists, statisticians, epidemiologists and homeopaths;
- ask them to design a rigorous protocol of one or two studies that would provide a definitive answer to the research question posed;
- make sure that, once everyone is happy with the protocol, all parties commit to abiding by the findings that will emerge from these trials;
- conduct the studies under adequately strict supervision;
- evaluate the results according to the protocol;
- publish them in a top journal;
- do the usual press-releases, interviews etc.
In India, it seems that the last point in this agenda came far too early. This is because, in this and several other countries, homeopathy has become more a belief system than a medicine. And because it is about belief, the believers will avert any truly meaningful and rigorous test of homeopathy’s efficacy.
This was essentially the question raised in a correspondence with a sceptic friend. His suspicion was that statistical methods might produce false-positive overall findings, if the research is done by enthusiasts of the so-called alternative medicine (SCAM) in question (or other areas of inquiry which I will omit because they are outside my area of expertise). Consciously or inadvertently, such researchers might introduce a pro-SCAM bias into their work. As the research is done mostly by such enthusiasts; the totality of the evidence would turn out to be heavily skewed in favour of the SCAM under investigation. The end-result would then be a false-positive overall impression about the SCAM which is less based on reality than on the wishful thinking of the investigators.
How can one deal with this problem?
How to minimise the risk of being overwhelmed by false-positive research?
Today, we have several mechanisms and initiatives that are at least partly aimed at achieving just this. For instance, there are guidelines on how to conduct the primary research so that bias is minimised. The CONSORT statements are an example. As many studies pre-date CONSORT, we need a different approach for reviews of clinical trials. The PRISMA guideline or the COCHRANE handbook are attempts to make sure systematic reviews are transparent and rigorous. These methods can work quite well in finding the truth, but one needs to be aware, of course, that some researchers do their very best to obscure it. I have also tried to go one step further and shown that the direction of the conclusion correlates with the rigour of the study (btw: this was the paper that prompted Prof Hahn’s criticism and slander of my work and person).
So, problem sorted?
Not quite!
The trouble is that over-enthusiastic researchers may not always adhere to these guidelines, they may pretend to adhere but cut corners, or they may be dishonest and cheat. And what makes this even more tricky is the possibility that they do all this inadvertently; their enthusiasm could get the better of them, and they are doing research not to TEST WHETHER a treatment works but to PROVE THAT it works.
In the realm of SCAM we have a lot of this – trust me, I have seen it often with my own eyes, regrettably sometimes even within my own team of co-workers. The reason for this is that SCAM is loaded with emotion and quasi-religious beliefs; and these provide a much stronger conflict of interest than money could ever do, in my experience.
And how might we tackle this thorny issue?
After thinking long and hard about it, I came up in 2012 with my TRUSTWORTHYNESS INDEX:
If we calculated the percentage of a researcher’s papers arriving at positive conclusions and divided this by the percentage of his papers drawing negative conclusions, we might have a useful measure. A realistic example might be the case of a clinical researcher who has published a total of 100 original articles. If 50% had positive and 50% negative conclusions about the efficacy of the therapy tested, his TI would be 1.
Depending on what area of clinical medicine this person is working in, 1 might be a figure that is just about acceptable in terms of the trustworthiness of the author. If the TI goes beyond 1, we might get concerned; if it reaches 4 or more, we should get worried.
An example would be a researcher who has published 100 papers of which 80 are positive and 20 arrive at negative conclusions. His TI would consequently amount to 4. Most of us equipped with a healthy scepticism would consider this figure highly suspect.
Of course, this is all a bit simplistic, and, like all other citation metrics, my TI provides us not with any level of proof; it merely is a vague indicator that something might be amiss. And, as stressed already, the cut-off point for any scientist’s TI very much depends on the area of clinical research we are dealing with. The lower the plausibility and the higher the uncertainty associated with the efficacy of the experimental treatments, the lower the point where the TI might suggest something to be fishy.
Based on this concept, I later created the ALTERNATIVE MEDICINE HALL OF FAME. This is a list of researchers who manage to go through life researching their particular SCAM without ever publishing a negative conclusion about it. In terms of TI, these people have astronomically high values. The current list is not yet long, but it is growing:
John Weeks (editor of JCAM)
Deepak Chopra (US entrepreneur)
Cheryl Hawk (US chiropractor)
David Peters (osteopathy, homeopathy, UK)
Nicola Robinson (TCM, UK)
Peter Fisher (homeopathy, UK)
Simon Mills (herbal medicine, UK)
Gustav Dobos (various, Germany)
Claudia Witt (homeopathy, Germany and Switzerland)
George Lewith (acupuncture, UK)
John Licciardone (osteopathy, US)
The logical consequence of a high TI would be that researchers of that nature are banned from obtaining research funds and publishing papers, because their contribution is merely to confuse us and make science less reliable.
I am sure there are other ways of addressing the problem of being mislead by false-positive research. If you can think of one, I’d be pleased to hear about it.
