This post is dedicated to Mel Koppelman.
Those who followed the recent discussions about acupuncture on this blog will probably know her; she is an acupuncturist who (thinks she) knows a lot about research because she has several higher qualifications (but was unable to show us any research published by herself). Mel seems very quick in lecturing others about research methodology. Yesterday, she posted this comment in relation to my previous post on a study of aromatherapy and reflexology:
Professor Ernst, This post affirms yet again a rather poor understanding of clinical trial methodology. A pragmatic trial such as this one with a wait-list control makes no attempt to look for specific effects. You say “it is quite simply wrong to assume that this outcome is specifically related to the two treatments.” Where have specific effects been tested or assumed in this study? Your statement in no way, shape or form negates the author’s conclusions that “aromatherapy massage and reflexology are simple and effective non-pharmacologic nursing interventions.” Effectiveness is not a measure of specific effects.
I am most grateful for this comment because it highlights an issue that I had wanted to address for some time: The meanings of the two terms ‘efficacy and effectiveness’ and their differences as seen by scientists and by alternative practitioners/researchers.
Let’s start with the definitions.
I often use the excellent book of Alan Earl-Slater entitled THE HANDBOOK OF CLINICAL TRIALS AND OTHER RESEARCH. In it, EFFICACY is defined as ‘the degree to which an intervention does what it is intended to do under ideal conditions. EFFECTIVENESS is the degree to which a treatment works under real life conditions. An EFFECTIVENESS TRIAL is a trial that ‘is said to approximate reality (i. e. clinical practice). It is sometimes called a pragmatic trial’. An EFFICACY TRIAL ‘is a clinical trial that is said to take place under ideal conditions.’
In other words, an efficacy trial investigates the question, ‘can the therapy work?’, and an effectiveness trial asks, ‘does this therapy work?’ In both cases, the question relate to the therapy per se and not to the plethora of phenomena which are not directly related to it. It seems logical that, where possible, the first question would need to be addressed before the second – it does make little sense to test for effectiveness, if efficacy has not been ascertained, and effectiveness without efficacy does not seem to be possible.
In my 2007 book entitled UNDERSTANDING RESEARCH IN COMPLEMENTARY AND ALTERNATIVE MEDICINE (written especially for alternative therapists like Mel), I adopted these definitions and added: “It is conceivable that a given therapy works only under optimal conditions but not in everyday practice. For instance, in clinical practice patients may not comply with a therapy because it causes adverse effects.” I should have added perhaps that adverse effects are by no means the only reason for non-compliance, and that non-compliance is not the only reason why an efficacious treatment might not be effective.
Most scientists would agree with the above definitions. In fact, I am not aware of a debate about them in scientific circles. But they are not something alternative practitioners tend to like. Why? Because, using these strict definitions, many alternative therapies are neither of proven efficacy nor effectiveness.
What can be done about this unfortunate situation?
Simple! Let’s re-formulate the definitions of efficacy and effectiveness!
Efficacy, according to some alternative medicine proponents, refers to the therapeutic effects of the therapy per se, in other words, its specific effects. (That coincides almost with the scientific definition of this term – except, of course, it fails to tell us anything about the boundary conditions [optimal or real-life conditions].)
Effectiveness, according to the advocates of alternative therapies, refers to its specific effects plus its non-specific effects. Some researchers have even introduced the term ‘real-life effectiveness’ for this.
This is why, the authors of the study discussed in my previous post, could conclude that “aromatherapy massage and reflexology are simple… effective… interventions… to help manage pain and fatigue in patients with rheumatoid arthritis.” Based on their data, neither aromatherapy nor reflexology has been shown to be effective. They might appear to be effective because patients expected to get better, or patients in the no-treatment control group felt worse for not getting the extra care. Based on studies of this nature, giving patients £10 or a box of chocolate might also turn out to be “simple… effective… interventions… to help manage pain and fatigue in patients with rheumatoid arthritis.” Based on these definitions of efficacy and effectiveness, there are hardly any limits to bogus claims for any old quackery.
Such obfuscation suits proponents of alternative therapies fine because, using such definitions, virtually every treatment anyone might ever think of can be shown to be effective! Wishful thinking, it seems, can fulfil almost any dream, it can even turn the truth upside down.
Or can anyone name an alternative treatment that cannot even generate a placebo response when administered with empathy, sympathy and care? Compared to doing nothing, virtually every ineffective therapy might generate outcomes that make the treatment look effective. Even the anticipation of an effect alone might do the trick. How often have you had a tooth-ache, went to the dentist, and discovered sitting in the waiting room that the pain had gone? Does that mean that sitting in a waiting room is an effective treatment for dental pain?
In fact, some enthusiasts of alternative medicine could soon begin to argue that, with their new definition of ‘effectiveness’, we no longer need controlled clinical trials at all, if we want to demonstrate how effective alternative therapies truly are. We can just do observational studies without a control group, note that lots of patients get better, and ‘Bob is your uncle’!!! This is much faster, saves money, time and effort, and has the undeniable advantage of never generating a negative result.
To most outsiders, all this might seem a bit like splitting hair. However, I fear that it is far from that. In fact, it turns out to be a fairly fundamental issue in almost any discussion about the value or otherwise of alternative medicine. And, I think, it is also a matter of principle that reaches far beyond alternative medicine: if we allow various interest groups, lobbyists, sects, cults etc. to use their own definitions of fundamentally important terms, any dialogue, understanding or progress becomes almost impossible.
64 Responses to Effectiveness, efficacy and wishful thinking…if all else fails, invent your own definitions!
Leave a Reply to Mel Cancel reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Dear Professor Ernst,
I am beyond touched that you have dedicated a whole post to me. Thank you from the bottom of my heart. My reply is dedicated to you, and your continued misunderstanding of these fairly simple terms.
Now, there is no need for a reformulation of the definitions of “efficacy” and “effectiveness,” simply a greater understanding of what they mean. Perhaps it’s a language barrier issue (I certainly couldn’t have this debate in German). But you continue to make comments about the meaning of these terms that are fundamentally incorrect.
A couple of points:
1) “an efficacy trial investigates the question, ‘can the therapy work?’, and an effectiveness trial asks, ‘does this therapy work?’ In both cases, the question relate to the therapy per se and not to the plethora of phenomena which are not directly related to it.” That’s categorically incorrect. There are a number of factors understood to contribute to a patient’s improvement in a clinical setting: natural history and regression to the mean, the Hawthorne effect, elements of usual care, the placebo effect and the intervention itself. An efficacy trial attempts to control for all phenomena not related to the active ingredient of the intervention. A pragmatic trial on the other hand compares two packages of care without attempting to isolate and individually control for these different variables. As you say, there is little debate about these definitions and it’s really not that hard to understand.
2) As you troll pubmed for low quality studies to bash in an “unbiased” attempt to prove that CAM doesn’t work, you forget that most pragmatic studies compare an intervention to usual care and pharmaceutical treatments, which themselves have demonstrated efficacy. When a clinical trial asks whether or not something “works” it’s always relative to a particular control. So in a pragmatic trial, when acupuncture is demonstrated to be superior to pharmacological treatment and usual care for low back pain, migraines, tension headaches, IBS etc, it is compared to what is normally on offer, which represents the real choice that real patients are faced with. You and other skeptics “hypothesise” that acupuncture’s superiority has something to do with the placebo effect or other non-specific effects, but you have no evidence and frankly, those who are in the business of trying to help patients get better aren’t caught up on this. Objectively, the data clearly indicates which package of care achieves better outcomes and this is the question we are trying to answer. Arguing against the more effective treatment is unethical, particularly when it’s safer. Also, in this instance, real scientists ask questions about the phenomena (“Why is acupuncture superior to usual care?”) rather than assume a particular cause without testing that assumption (“It must be placebo, because that’s consistent with my worldview”).
3) Efficacy studies are designed to measure the effectiveness of pharmaceuticals, where the intervention is a chemical or another inanimate object taken with a presumed specific biological effect. They are not appropriately suited to measure complex interventions where the treatment is not an injested or administered object, but the administration of an intervention based on a trained practitioner’s skills. Interventions that fall into this category include surgery (zero demonstrated efficacy for heart transplants), talking therapies, physiotherapy, and acupuncture.
Studying these treatments in carefully controlled settings is appropriate to answer specific questions, but it is not possible to come up with a control analagous to a sugar pill unless the “mechanism of action” of the treatment if fully understood. If it’s not, you do not know if you are controlling for treatment effects or not. Since evidence based medicine is an attempt to study and measure what happens, rather than that which can be successfully studies using a DB RCT study design, that a biologically inert control for acupuncture doesn’t exist does not preclude its recommendation for rationally minded people.
4) “effectiveness without efficacy does not seem possible” That’s completely irrational, unscientific and more than a little delusional. You are arguing against objective data to support your biased worldview. Systematic reviews demonstrate that acupuncture results in greater analgesia, greater improvement in function and greater quality of life in those with low back pain than usual care and pharmaceuticals. The data consistently demonstrate its effectiveness. Your argument that “well, that just can’t be! I don’t believe it!! It goes against my beliefs!” is scientifically untennable.
EBM dictates that based on these data, the treatment should be recommended and made available. Why this package of care is more effective than drugs is an interesting question, but not one that precludes offering the treatment when unanswered. How paracetomol (acetaminophen) supposedly works still remains a mystery and yet it is one of the most widely used drugs on the planet, notwithstanding the fact that it kills thousands of people every year when used as recommended.
5) Please bear in mind that the only reason why it’s even worth discussing the evidence base for treatments like acupuncture is because for many conditions they outperform usual care. Your awkward grapple with the definitions of “efficacy” and “effectiveness” not withstanding, systematic reviews of effectiveness studies convincingly answer the question “If I want to get better, what treatment gives me the best chances?” For back pain, migraine, headache and others, a careful synthesis of the best evidence indicates that the answer is resoundingly “acupuncture.” If you feel more comfortable holding on to beliefs that are diametrically opposed to best evidence, that’s your prerogative but leave the people who are effectively helping patients get better out of it.
Dear Mel, maybe you are more the mathematical type.
A treatment effect can be modeled with following linear model:
TE = (ThE + epsilon) + P with TE being the observed effect, ThE being the effect due to Therapy, epsilon being an error term depending on treatment conditions and P being non-treatment related effects (e.g. placebo).
With regard to efficiency and efficacy following definitions apply:
Def1: Efficacy = ThE + epsilon1
Def2: Efficiency = ThE + epsilon2
Def3: epsilon2 > epsilon1
Def3 models the suboptimal settings.
With these definitions it follows immediately that your point 4 is moot. If the efficacy is 0 (ThE+epsilon1 = 0) an efficiency greater than zero is caused by the error term and not by the treatment which is an indication that the treatment actually may worsen the condition
As for this study, what did these Nurses measure ? They measured TE but failed to estimate P. I guess you know from highschool math, if you measure something consisting of two variables (in this case ThE+epsilon and P respectively) you have to estimate at least one to be able to determine the second. The Nurses failed to do so. That means the Efficacy/Efficiency is confounded with the placebo effect in a way no mathematical trick whatsoever could untangle. This further means that their conclusion, namely that the therapies they applied are effective is not valid as Prof. Ernst correctly points out. Mathematically speaking, they are claiming that ThE + epsilon > 0 , yet they have not estimated P. This simply does not fly as Prof. Ernst correctly points out.
regards,
Thomas – computational biologist with 25+ years experience under the belt.
@Thomas, thanks for offering your perspective.
“With these definitions it follows immediately that your point 4 is moot. If the efficacy is 0 (ThE+epsilon1 = 0) an efficiency greater than zero is caused by the error term and not by the treatment which is an indication that the treatment actually may worsen the condition.” No, point 4 is still valid as neither Professor Ernst, nor you, nor anyone else has provided evidence that ThE+epsilon1 = 0. Adequately powered studies provide ample evidence that efficacy is indeed great than 0. It’s great that you understand the mathematical models so well, but they need to be applied to the actual data.
As for the second point, I’m not sure which nurses you’re referring to, most of the authors go by “doctor” and this wasn’t a “study” but rather a Cochrane systematic review and meta-analysis. But you make my point for me so well, I really appreciate it because I couldn’t have put it better. What we have in the pragmatic section of the review is a comparison of two TEs – we don’t know P and we don’t know ThE. We can only compare which had the larger beneficial effect. In pragmatic effectiveness studies, the definition of a treatment being “effective” is one where it’s TE is significantly larger than usual care (or whatever is used as the pragmatic control). You are correct that in this design “efficacy/efficiency is confounded with the placebo effect in a way no mathematical trick whatsoever could untangle” but what we are aiming to answer by definition is which treatment has a greater observed effect. Thus, this study unequivocally supports acupuncture’s effectiveness, despite Professor Ernst’s confusion over what that term means.
First I am referring to the original paper regarding aroma therapy and massages. I thought this was obvious. Second, here we are not discussing acupuncture, but the definition of efficiacy and effectiveness. If you want to discuss acupuncture, you are on the wrong post.
Finally, in clinical trials efficiency and efficacy is very well defined (as outlined above) and does not change with trial type. If you don’t only adhere to alternative medicine but to alternative terminology that is your problem. According to standard terminology the authors can not draw the conclusion that massage and aroma therapy are efficient treatments. It could well be the placebo effect.
There are several other problems with the original paper critizised by Ernst. Pragmatic trials require (a) a preliminary trial to determine efficacy and (b) a large enough number of patients. A is cleraly not fulfilled and personally I do not think that an n of 17 per arm provides enough power to the study.
Apologies, Thomas, I didn’t realise you were referring to the low quality study that Professor Ernst “randomly” selected to make his point yesterday. I was referring to the Cochrane Review of Acupuncture for Headaches, mentioned previously (which I see now was lower in the thread, apologies).
http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD007587.pub2/full
Any comments on this? Based on their meta-analysis, the authors concluded that “acupuncture is effective” for headache. Any specific objections?
@Thomas,
Many thanks indeed for the linear model plus the definitions (and for your other comments). My background is in a few specialist fields of applied science that seem to be wholly irrelevant to the field of medicine, but you have shown me that the fundamental principles and the logic are the same — the only difference being the necessary field-specific precise terminology.
Well, many things can be sufficiently modeled in mathematics. However, deciding on the preponderance of evidence is sometimes a tricky undertaking that has to take into consideration how you got your information
Example 1: The frogs. Imagine you have a pond with many frogs, females to males at a ratio of 1:1 You draw a sample of two frogs, what is the probability that the other is female, given the fact you know that one is male ? If you know that one is male by having heard (but not seen) one of the frogs making a male specific sound ?
Example 2: Two people have left traces of their own blood at the scene of a crime. A suspect , Oliver, is tested and found to have type ‘O’ blood. The blood groups of the two traces are found to be of type ‘O’ (a common type in the local population, having frequency 60%) and type ‘AB’ (a rare type, with frequency 1%). Does this data (type ‘O’ and ‘AB’ blood were found at the scene) give evidence in favor of the proposition that Oliver was one of the two people present at the scene of the crime?”
This is the problem with the discussion with Mel here. Whereas she correctly applies very strict principles to Ernst et al she applies very lenient principles to pro-acupuncture research.
“This is the problem with the discussion with Mel here. Whereas she correctly applies very strict principles to Ernst et al she applies very lenient principles to pro-acupuncture research.”
The principles that I apply to the critical evaluation of all research are identical. Using a positive Cochrane Systematic review as evidence in support of acupuncture’s effectiveness for the treatment of headaches is not considered by any medical researcher to be lenient.
So here’s one for you, Thomas, let’s have a control group. You’re saying that the Cochrane Review does not support the effectiveness of acupuncture for the prevention of headaches? That using it as evidence is “lenient”? Show me stronger evidence for another treatment for the same population. Any treatment, pharmacological, psychological, whatever. Show me what strong evidence for a treatment for headaches looks like.
I take it you’re not a clinician, but at the end of the day when you’re done with all your mathematical models, one needs to look at the research and decide what to make available to real patients. If I understand you correctly, you’re saying that acupuncture should not be offered for the prevention and management of headaches, in spite of what is universally accepted to be high quality evidence of effectiveness and efficacy. So what would you make available to patients instead and what evidence would you base this on?
Mel, apparently you do not understand the scientific meaning of the wording of the conclusion of the Cochrane review. Literally it says “suggest” and recommends further studies.
Translated into plain English this means that
(a) there are some studies showing an effect,
(b) there are not enough robust studies for a final conclusion.
(c) data warrant a second look into that matter.
This is the meaning of the Cochrane review, nothing more. It did not show that acupuncture is many times more effective. This is what I mean with you being over-lenient. I understand this. I bet you came into the field by being healed by naturopathic medicine and that you are a personality who wants to help people. This would be a typical path. In such an scenario, observational bias can become huge. It would be interesting if you would give an estimate of the effectiveness of your treatment and then to do a statistics of your patients to see how effective your treatment really is. Alone for QC reasons this would be beneficial for yourself.
Anyway, the Cochrane Review is NOT high quality evidence, neither for effectiveness nor for efficacy. The authors themselves say this since they use terms such as “suggest” and “more studies are needed”. If Cochrane authors find evidence, they usually use wordings such as “there is some evidence”, “there is good evidence” etc.
Finally, I am not against somebody poking patients, but – they should be informed. However, telling them a treatment works or has good evidence when data are that controversial is highly unethical.
As for your challenge show me a treatment that is better, that is irrelevant here.
Mel, can you please name an alternative therapy that, by your definition od the terms, might NOT be effective?
First, I don’t think much of the term “alternative therapy” – like you say, there’s just medicine that works. My area of expertise is acupuncture, which enjoys a more robust evidence base than many currently used conventional treatments.
But to answer your question, you obviously need to define a population. I’d just go by the data and interpret it in context. In NICEs recent review of low back pain for example, which recommended cognitive/psychological therapy as a treatment for low back pain, according to their data cognitive therapy did not outperform usual care or waitlist control. Indeed, psychological therapy plus exercise didn’t even outperform waitlist control. So I would consider these treatments “ineffective” according to the published evidence – which is interesting since they involve attention, empathy, and positive expectation and didn’t even do better than the psychologically devastating effects of waiting for treatment.
On the other hand, when therapies do outperform usual care, I conclude that the evidence supports their “effectiveness” because that’s what effectiveness means.
thank you – so you cannot name a therapy [forget the ‘alternative’ for a moment] that, by your criteria is NOT effective, can you?
that means the term ‘effective’, as you define it, is meaningless. do you agree?
Huh?
“So I would consider these treatments “ineffective” according to the published evidence.”
Did you even read what I wrote before replying?
sorry, but I do not consider ‘psychological therapy plus exercise’ as an entity that we can discuss. there are dozens of both. can you please be a bit more specific and name a therapeutic entity that is, according to your definition, NOT effective?
Ok, looking through the Cochrane database, I note a review of Saline irrigation for chronic rhinosinusitis. There was no benefit found to using a low-volume (5 ml) nebulised saline spray over usual care. So this review supports the conclusion that saline irrigation is ineffective in the treatment of chronic rhinosinusitis.
are you sure? doesn’t it rather support the conclusion that this treatment is as effective as usual care?
Fair point, it rather depends on how the studies were designed statistically, whether or not they were testing for superiority or equivalence.
But then your question highlights a logical flaw in your argument – in order for the designation “effectiveness compared to usual care” to have meaning, one needn’t find an example of inferiority as you suggested, but merely an example of lack of superiority, which can include equivalence.
Thus, the term “effective” is not meaningless (well, perhaps it is to you, but that’s another story)
so you cannot name an ‘ineffective’ therapy ???
I think this gets to the heart of many of the problems with supporters of alternative therapies: their imprecise use of language and their willingness to go far beyond a study’s conclusions. Good papers couch what they say in carefully worded terms to make sure everything is logical and can be substantiated. Anything speculative is clearly so. In their conclusions, precise and measured language is used, making it clear what their results show. But CAM enthusiasts have no such qualms or self-control and are frequently seen to extrapolate wildly beyond the data, frequently adding their own spin, bias and prejudices – usually suiting their own interests, benefits and pockets.
@Alan, I couldn’t help but notice that your reply constitutes a series of sweeping statements about “supporters of alternative therapies” and what “good papers” do (is it your opinion that there are zero “good papers” on alternative therapies?) but you fail to provide a single specific example to support your arguments. Talk about imprecision of language!
So here’s an example, Cochrane recently updated their systematic review on the use acupuncture for the prevention of tension type headache. Compared to usual care, the RR for acupuncture was 2.5 for a 50% reduction in headaches and compared to quasi-“sham” acupuncture the RR was 1.3. The overall quality of evidence was moderate. The authors concluded: “The available results suggest that acupuncture is effective for treating frequent episodic or chronic tension-type headaches, but further trials – particularly comparing acupuncture with other treatment options – are needed.”
Do you have any specific issues with the authors’ conclusions of this particular meta-analysis? Based on their data, do you agree that it’s appropriate to conclude that acupuncture is effective for this condition?
http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD007587.pub2/full
Apologies, I did those results in rather a hurry and didn’t include CIs. Here were the results from the abstract:
Pragmatic studies:
“While effect size estimates of the two trials differed considerably, the proportion of participants experiencing at least 50% reduction of headache frequency was much higher in groups receiving acupuncture than in control groups (moderate quality evidence; trial 1: 302/629 (48%) versus 121/636 (19%); risk ratio (RR) 2.5; 95% confidence interval (CI) 2.1 to 3.0; trial 2: 60/132 (45%) versus 3/75 (4%); RR 11; 95% CI 3.7 to 35). Long-term effects (beyond four months) were not investigated.”
“Efficacy” studies:
“Acupuncture was compared with sham acupuncture in seven trials of moderate to high quality (low risk of bias); five large studies provided data for one or more meta-analyses. Among participants receiving acupuncture, 205 of 391 (51%) had at least 50% reduction of headache frequency compared to 133 of 312 (43%) in the sham group after treatment (RR 1.3; 95% CI 1.09 to 1.5; four trials; moderate quality evidence). Results six months after randomisation were similar.”
@Mel
I think it was perfectly clear that I was making a general statement – this very blog provides numerous examples.
Ok, but how about responding to the specific example I provided and the question I asked? Do you feel that the authors of this Cochrane systematic review articulated a conclusion consistent with the evidence? That the conclusions were “logical and substantiated”? Do you agree that this review demonstrates acupuncture’s effectiveness and efficacy for the prevention of tension type headache?
I think you misunderstood what I said, thus perhaps (inadvertently) providing a data point in my support, but I’ll wait until you’ve agreed with Prof Ernst on the meaning of those words you used.
Apologies if I misunderstood, I thought you were saying that “supporters of alternative therapies” make claims beyond what is supported by the research.
So my question to you is this: I have been known to say that according to the best evidence, “acupuncture is effective for the treatment of headaches” and I have also been known to support this statement using systematic reviews, such as the one recently published by the Cochrane Collaboration, which I referenced above.
Is it your opinion that the statement “acupuncture is effective for the treatment of headaches” is an evidence-based statement? Or is this an example of a statement that suggests a lack of “self-control” where I “extrapolate wildly beyond the data, frequently adding [my] own spin, bias and prejudices”?
Mel
Perhaps I wasn’t clear enough for you, but I thought my use of ‘frequently’ twice in the last sentence would be a clue.
Leaving a reply to one of your comments below.
In this review there are two trials with sham acupuncture a.k.a placebo, they display a a 95% CI boundary of the RR barely greater than 1 which means that the p-value is barely below 0.05.That, taken together with the low prior probability, indicates a possible false positive. To put it mildly I am astonished that apparently you do not know these connections.
Acupuncture *has* a low prior probability because the entire model it is based on (flux of Qi) is not probable. The probability to derive something correct from a false underlying model is indeed very low.
The conclusion that can be drawn from Cochrane reviews is:
Acupuncture is worth a second look in some very defined, mainly pain related scenarios. Your claim, however, “acupuncture works” is vastly overblown and NOT supported by Cochrane.
@Thomas
“In this review there are two trials with sham acupuncture a.k.a placebo” – If you would like to equate sham acupuncture to a “placebo” control, you need to demonstrate that sham acupuncture is biologically inert and does not provide any of the specific treatment effects of the acupuncture itself. There is a large literature (which I have repeatedly referenced in this site and on my own) that demonstrates that sham acupuncture spectacularly fails these criteria. If you would like to persist with that line of thinking, then you will need to show me your evidence. Otherwise, we can recognise the reality that the sham intervention is controlling for part of the effect of the intervention. It is an active control and thus sample sizes need to be larger in order to test for superiority.
“Acupuncture *has* a low prior probability because the entire model it is based on (flux of Qi) is not probable. The probability to derive something correct from a false underlying model is indeed very low.” That line of argument is irrational and unsupportable. Many drugs and procedures are used under the assumption that they have one model of action only to find that the therapeutic effect comes from a different model of action (statins and antidepressants both come to mind). The effectiveness doesn’t change. Similarly, using a biochemical/neuro-endocrinological model of acupuncture’s effects has strong documentation and gives a much stronger prior plausibility. The established effectiveness of acupuncture as evidenced by high quality systematic review exists independently of the explanatory model.
I know about the problems of designing a placebo for surgical interventions. However, if a sham intervention is equally effective as the acupuncture, it can be done by a taylor poking you with needles right ?
With regard to your refutation of the probability argument you have overlooked a minor detail. The underlying model of drugs is that they have a point of action leading to a chain of events that results in a therapeutic effect. This model is correct in *any* drug. With acupuncture this is completely different. It is derived from a model that violates basic theories of nature, much like homeopathy.
With regard to your proposed model you might be interested in this review:
“Gorski, David H. (2014). “Integrative oncology: really the best of both worlds?”. Nature Reviews Cancer. doi:10.1038/nrc3822. ISSN 1474-175X. PMID 25230880.”
or this one:
MacPherson, Hugh; Maschino, Alexandra C; Lewith, George; Foster, Nadine E; Witt, Claudia; Vickers, Andrew J; Acupuncture Trialists’ Collaboration (2013). Eldabe, Sam, ed. “Characteristics of Acupuncture Treatment Associated with Outcome: An Individual Patient Meta-Analysis of 17,922 Patients with Chronic Pain in Randomised Controlled Trials”. PLoS ONE 8 (10): e77438.
The last one is particularly of interest, because it finds that the success of acupuncture does not depend on the points chosen, or the technique or experience of the acupuncturist (needle punching taylor ?) but on the number of needles and the number of sessions.
Finally here we have your quote that acupuncture indeed works: “The established effectiveness of acupuncture as evidenced by high quality systematic review exists independently of the explanatory model.”
@Thomas: “However, if a sham intervention is equally effective as the acupuncture, it can be done by a taylor poking you with needles right ?” Except that evidence shows that it’s not, acupuncture outperforms active sham control in adequately powered studies. Also, whether or not the active intervention is shown to be “equally effective” to the sham control depends on how the study is designed and how it’s powered. Your comment is meaningless in the absence of references to support it. In the Cochrane review on acupuncture for headaches, acupuncture significantly outperformed sham.
“With regard to your refutation of the probability argument you have overlooked a minor detail. The underlying model of drugs is that they have a point of action leading to a chain of events that results in a therapeutic effect. This model is correct in *any* drug.”
Um, hold on, let me see if I understand this. Because ingestion of a chemical can lead to a therapeutic effect, any drug has biological plausibility? for any condition? Beta blockers for herpes? NSAIDs for dwarfism? Antibiotics for erectile dysfunction? That a chemical compound can result in physiological and clinical changes, does not mean that any drug has biological plausibility for any condition. And yet this dangerous line of thinking is common in medical practice: http://archinte.jamanetwork.com/article.aspx?articleid=410250
On the other hand, acupuncture stimulation has many well-documented mechanisms including altered transcription via mechano-transduction, neuro-endocrine modulation, and mast cell degranulation leading to anti-inflammatory effects in a route not disimilar to capsaicin.
“With regard to your proposed model you might be interested in this review.” I’m familiar with this review and many others produced by Gorski and his colleagues. This is a heavily biased narrative review that does not constitute a source of reliable information. Just as a small example, he cites a 2010 mouse study on the role of adenosine in acupuncture’s mechanisms and makes some lame, unsubstantiated comment about the size of the mouse’s leg and how the needle would be relatively bigger, etc. And yet he completely fails to mention a 2012 study that found the same effect in humans, probably because it completely invalidates his point. If you’re aiming to read material that supports your current views, read Gorski. If you’re interested in actually understanding the literature, go elsewhere.
“The last one is particularly of interest, because it finds that the success of acupuncture does not depend on the points chosen, or the technique or experience of the acupuncturist (needle punching taylor ?) but on the number of needles and the number of sessions.”
So success of acupuncture depends on the number of sessions e.g. adequate dosing. Makes sense. Just out of curiosity, what percentage of the trials included in this meta-analysis would be considered to have used adequate dosing? It looks like the minority. Perhaps the point location and technique only become significant when the treatment is adequately dosed? Would seem logical.
@ Mel on Tuesday 07 June 2016 at 10:26
“@Thomas: “However, if a sham intervention is equally effective as the acupuncture, it can be done by a taylor poking you with needles right ?” Except that evidence shows that it’s not, acupuncture outperforms active sham control in adequately powered studies. Also, whether or not the active intervention is shown to be “equally effective” to the sham control depends on how the study is designed and how it’s powered.
So, “real” acupuncture outperforms “sham” acupuncture? Do either of them do anything of tangible benefit in any medical sense? Does either reduce or nullify the need for dialysis for renal failure, for example, or perhaps chemo for glioblastoma grade four from which my best friend died, or maybe, a ruptured disc at L5-S1 from which I suffered, or even palliative care for methesioloma from which Warren Zevon died? Take your silly needles and give yourselves a needle enema; probably the best use for your metaphysical nonsense.
“Your comment is meaningless in the absence of references to support it.”
Your comment is meaningless in the absence of any evidence to support acupuncture. You have no evidence yet you demand evidence from others. You are the king/queen/non-gender-specific-monarch of trolls. Either way, you are full of crap. Put up one thing your nonsense has cured and then discussion will be meaningful.
“In the Cochrane review on acupuncture for headaches, acupuncture significantly outperformed sham.”
Neither cured anything, so who f@cking cares?
“Um, hold on, let me see if I understand this. Because ingestion of a chemical can lead to a therapeutic effect, any drug has biological plausibility? for any condition? Beta blockers for herpes? NSAIDs for dwarfism? Antibiotics for erectile dysfunction? That a chemical compound can result in physiological and clinical changes, does not mean that any drug has biological plausibility for any condition. And yet this dangerous line of thinking is common in medical practice: http://archinte.jamanetwork.com/article.aspx?articleid=410250”
Really, is this how your mind works? It all falls into place.
“On the other hand, acupuncture stimulation has many well-documented mechanisms including altered transcription via mechano-transduction, neuro-endocrine modulation, and mast cell degranulation leading to anti-inflammatory effects in a route not disimilar to capsaicin.”
https://en.wikipedia.org/wiki/Mechanotransduction
No mention of acupuncture here.
https://www.google.com.au/search?q=neuro-endocrine+modulation&oq=neuro-endocrine+modulation&aqs=chrome..69i57&sourceid=chrome&ie=UTF-8
I couldn’t find any mention of acupuncture being a credible mechanism in any of these. Maybe I didn’t search hard enough (aka known as scraping the bottom of the barrel for anything that may faintly look supportive).
https://en.wikipedia.org/wiki/Degranulation
Nothing here either.
Surely you wouldn’t be making extravagant claims devoid of evidence? Naahh, forget that, you have history.
“So success of acupuncture depends on the number of sessions e.g. adequate dosing. Makes sense. Just out of curiosity, what percentage of the trials included in this meta-analysis would be considered to have used adequate dosing? It looks like the minority. Perhaps the point location and technique only become significant when the treatment is adequately dosed? Would seem logical.”
When your “adequate dosing” does anything meaningful, let me know. By-the-way. meaningful does not mean relieving your gullible muppets of cash for your benefit. It also does not include some deluded person saying, “Oh, I feel so much better. my broom-ride home will be so much more bearable”.
You faff about the edges and provide nothing useful. Who cares whether sham or “real” is more successful at taking money from gullible people.
Here is a challenge; cure a disease and let us know? (Therefore, goodbye forever.)
Perhaps the wrong place to discuss this metastudy but here goes.
The authors admit that one trial was the main influencer in their results and that two large trials were deficient for methodological reasons.
Half the selected trials were at risk of bias because of blinding/randomisation issues.
The authors also admit that they changed the Cochrane criteria for selective reporting because nearly all trials would have had to be rejected.
All of the authors of this study are involved in CAM or whatever the latest market speak term and there lies a problem. It is obvious that at the intersection of alternative therapies and science there is an unbridgeable chasm. Scientists and doctors want to find out what to works and what doesn’t. Alt medders only want confirmation of their particular career choice.
There are some other problems too. First, the placebo controlled study with 300 something patients barely met the 0.05 threshold. What does that mean ?
A p-value starts with the hypothesis that there is no difference between two experiments (the NUll-hpoythesis) A p-value tells us that the probability of my observed data fitting to the null hypothesis. However, “acupuncture works” is only *one* alternative hypothesis out of maybe hundreds. That means, in order for the data to support a particular alternative hypothesis, the prior probability of this hypothesis has to be considered. Low prior probability means low probability even several with p-values < 0.05. One needs a lot of p-valkues to overcome that or to put if with Carl Sagan: extraordinary claims require extraordinary proof.
So, even with two trials or so showing somewhat of an acceptable p-value only suggest a second look into the matter, nothing more. Mels claims that the Cochrane review shows acupuncture works are definitely overblown.
@Thomas, sorry but the whole “low prior probability” is invalid when it’s judged from the perspective of people who have systematically ignored the mechanistic data establishing biological plausibility. You are perpetuating a common (and severely flawed) circular argument, which goes as follows:
– acupuncture is effective, but since according to skeptics has a low prior plausibility, and the results are inversely correlated with this subjective prior plausibility number, acupuncture is not effective
– in the oft quoted words of Colquoun and Novella “There is no point in discussing surrogate outcomes such as fMRI studies or endorphine release studies until such time as it has been shown that patients get a useful degree of relief.” In other words, no point looking at all the data that supports biological mechanisms, we really don’t want to read anything that informs us of prior plausibility. But since we haven’t read it, we’ve decided that there’s a low plausibility. Very lame, very flawed and very unsupportable scientifically.
So the body of evidence that provides prior plausibility is systematically ignored (how very scientific) and the effectiveness demonstrated by meta-analysis is arbitrarily discounted on the basis of lack of knowledge of these studies. This argument is completely irrational.
“First, the placebo controlled study with 300 something patients barely met the 0.05 threshold. What does that mean ?” I don’t know, what power calculations are you referring to? How does 300 subjects compare to this calculation for a 0.05 threshold? Do you think 300 is big or small? How did you arrive at this assumption? And what intervention are you referring to as “placebo”? Did the control group receive a sugar pill?
@Acleron, completely valid to identify weaknesses in this review, as no published research is above critical appraisal.
However your choice of language in describing the bias evaluated in these studies belies a lack of familiarity with systematic review design. “Blinding issues” are built in to pragmatic study design, as they are for injections, surgery, physio etc. Your comments do not invalidate the results. I invite you to look at the Cochrane Handbook for Systematic Reviews, for a discussion of how to conduct and critically assess systematic reviews.
“The authors also admit that they changed the Cochrane criteria for selective reporting because nearly all trials would have had to be rejected.” First of all, the selective reporting issues were due to the nature of headache trials and how outcomes are reported, not because of the intervention. This is an issue SR/MAs of all interventions in the treatment of headaches, nothing to do with acupuncture. Secondly, these studies would not have been “rejected,” they would have had a reduction in GRADE score.
This SR/MA constitutes a high quality, systematic appraisal of the evidence acupuncture’s effectiveness in the treatment of headache. Your comments do not suggest otherwise. Also be mindful that if you insist on criticising this type of high quality review and denying that it constitutes good evidence, you will find that pretty much nothing meets that criteria in either CAM or conventional medicine.
Blinding issues are just ignored by alt-medders by using terms such as pragmatic. If you are testing a method which appears to be rather implausible then it behoves you to produce exemplary data.
‘If the strict Handbook guidelines had been applied, almost all trials would have been rated ‘unclear’ for the ‘selective reporting’ item. ‘
So rather than reject, they changed the criteria. It is telling that you cannot see the huge problem with this approach.
I notice that elsewhere you opine that 300 is a sufficiently high number of participants. The authors of this metastudy report that none of the included trials were of sufficient power.
So there are blinding issues, reporting issues and methodological issues but your opinion is that this is high quality.
The issue of medical trials of insufficient quality is already being addressed but that is irrelevant to this modality except for one important point. The Cochrane foundation was established to highlight the concerns of researchers and the self criticism of many trials over lack of reproducibility, inadequate controls, inadequate methodology, compromised authors and selective publication are publicised so action can be taken. No such self criticism comes from alt medders who will use rhetoric and false logic to justify their treatments.
I will repeat, alt medders are incapable of doing the science because they approach the problem from a belief or a statement that it does work without ever considering that it doesn’t.
@Acleron
“Blinding issues are just ignored by alt-medders by using terms such as pragmatic.” I repeat, blinding issues are part and parcel of any intervention where a person is involved in giving the treatment. There are zero double-blind studies of any practitioner-led treatments including heart transplants or brain surgery. That active interventions cannot be studied using identical methods to passive interventions does not invalidate them.
“If you are testing a method which appears to be rather implausible then it behoves you to produce exemplary data.” There is a direct correlation between how implausible one judges acupuncture and how much they know or have read about the subject. Plenty of practitioners use a neuro-endocrinological model for understanding acupuncture’s effectiveness and the literature is awash with studies documenting this. Read them and then comment.
In terms of exemplary data, the Cochrane review demonstrated that acupuncture was many times more effective than usual care in the prevention of headaches. I consider this exemplary. If you disagree, show me an intervention that performs better. Put your money where your mouth is. It’s all well and good to sit at your keyboard and repeat that acupuncture doesn’t work. But then tell us what does work, show me your evidence-based treatment for the prevention of headaches and the research you support your claim with. Enlighten us.
“‘If the strict Handbook guidelines had been applied, almost all trials would have been rated ‘unclear’ for the ‘selective reporting’ item. ‘
So rather than reject, they changed the criteria. It is telling that you cannot see the huge problem with this approach.”
@Acleron, again, they are referring to the simple fact that researchers use different reporting outcomes for headaches, there’s no uniformity. This applies to trials for treating headaches with paracetomol, anti-depressants, exercise, or riboflavin. If you’re saying that this lack of uniformity in reporting somehow invalidates the inclusion of these studies, then you’re basically saying that systematic reviews on headache trials should never be performed, ever. It’s completely irrational and demonstrates a poor understanding of review methodology.
“I notice that elsewhere you opine that 300 is a sufficiently high number of participants.” I certainly did not, 300 is way too few subjects for a trial with an active comparator. Care to show me where you got that from? I’ll repeat for the sake of all skeptics who enjoy misquoting me that this and all websites are searchable. No need to guess at what I wrote, all of the words are on this very page.
“The authors of this metastudy report that none of the included trials were of sufficient power.” Again, I beseech you, if you have a strong desire to critically appraise meta-analyses in public, please read up on it. Combining smaller studies for meta-analysis increases their power. This is a key part of their raison d’etre. This is meta-analysis 101. Honestly, the Cochrane manual is a really helpful book if you’re interested in discussing meta-analysis methodology.
Trying compare plausible treatments with implausible ones is a rhetorical trickle that doesn’t work. Because it is impossible to introduce controls in certain interventions does not give permission to ignore them in all.
Plausibility in establishing a plausible mechanism. Why should anything produced by puncturing the skin always produce beneficial results no matter what the disease states?
Exemplary data is nothing to do with results from a statistical package, it is all to do with the quality of the data. No statistic will correct for bias and uncontrolled trials with high risk of bias are present in this dataset.
The reporting bias is just another indicator of the poor quality of these trials. Just because the disease state is difficult to study is no excuse for ignoring this point.
Apologies, I re-read your comment about power and am now pleased you accept that these trials were underpowered. However I will repeat that combining these trials with all their problems will not improve the conclusion, rather the reverse. Until you have had to handle such data and produce recommendations based on that data, these points will obviously be beyond you. Reading the Cochrane handbook and even some simple stats books will not prepare you for the real problems in data analysis.
one indisputable truth about systematic reviews/meta-analyses is: RUBBISH IN, RUBBISH OUT. Therefore responsible authors of such articles always stress that any conclusions have to be taken with a pinch of salt.
Dear Professor Ernst
Since you are a professor why don’t you start acting like one? Please stick to the topic and stop your ad hominem attacks on Mel, CAM supports and anyone else who dares disagree with you regardless of who you think started with them first. Your comments are unprofessional and condescending.
I am so sorry – what exactly did you deem to be an ad hom? and what is unprofessional and condescending? I am truly interested to hear what grieves you.
As an example, in this post I would say that your comment about Mel: “who (thinks she) knows a lot about research” because she has several higher qualifications (but was unable to show us any research published by herself)” are directed at her personally and are irrelevant. The condescending nature of theses comments are what I find unprofessional and condescending. Are you saying that only people who have published research can understand research methodology?
What grieves me is that others in the scientific community might find the way that you refer to and sometimes interact with people who may not agree with you acceptable.
it grieves me to hear that statement of proven facts are, in your mind, ad homs and condescending. perhaps Mel doesn’t? I would dispute that these statements are irrelevant in this context.
So what is your assessment of these comments made to Edzard by Mel, Doug?
“This post affirms yet again a rather poor understanding of clinical trial methodology.”
“As you troll pubmed for low quality studies to bash in an “unbiased” attempt to prove that CAM doesn’t work,”
“Your argument that “well, that just can’t be! I don’t believe it!! It goes against my beliefs!”…”
Would you call these ad homs? Condescending? Unprofessional? If not, why not?
Not just in my mind. You could have made your point just as well by just stating that the comments were posted by an acupuncturist. We can disagree about the relevance of your statements.
so we do, it seems.
Deary, deary me. Every time I think Edzard has outdone himself, he comes up with something better. Your response to Mel is both unwarrantedly rude and incorrect. She clearly does know a lot about research, and judging from some of your replies it seems she might know a bit more than you, or at least is more meticulous in her argument. As for the idea that a non-researcher cannot comment on research, where on earth did you get this principle from? I am not a trained in research methodology and so cannot comment on its niceties, but reading these blogs it is clear that Edzard (and his acolytes) a. have made up their minds and won’t change them whatever the evidence to the contrary, and b. have been unwilling or unable to respond to Mel’s deconstruction of NICE’s withdrawal of support for acupuncture for back pain. I would say that an evidence-based approach to life is characterised by an open mind and a degree of humility – one that recognises that what seems certain today, may not seem so tomorrow. I don’t find this mindset modelled to any degree by Edzard Ernst and many others who frequent his blogs.
“Your response to Mel is both unwarrantedly rude and incorrect” – please enlighten me: what response precisely?
“She clearly does know a lot about research” – I am not ware that I said she did not.
“As for the idea that a non-researcher cannot comment on research, where on earth did you get this principle from?” – I think you just invented it; I certainly did not.
” Edzard (and his acolytes) a. have made up their minds and won’t change them” – you are demonstrably wrong: I have always changed my mind with the evidence. btw, this is why today I am more sceptical about acupuncture than when I published the positive Cochrane review which I cited earlier on this blog.
“unwilling or unable to respond to Mel’s deconstruction of NICE’s withdrawal of support for acupuncture for back pain” – this is off topic; I think we dealt with this fully.
” I don’t find this mindset modelled to any degree by Edzard Ernst and many others who frequent his blogs.” – an ad hom, don’t you think?
Peter, Mel has doubtlessly read a lot, she also knows a lot, but in science the only question we answer is “Is our hypothesis supported by the arguments or not ?” This evaluation process involves (a) exact definitions ans (b) a lot of probabilistics and this is a tricky area where she is not really firm (see for instance the Monty Hall Problem, it would be interesting to hear Mels solution to it).
There are two reasons why I am saying that. First her assessment about the low grade study was clearly wrong. The authors conclusion that aromatherapy and massages are an effective therapy can not be drawn from their data as Prof. Ernst correctly pointed out. Second the conclusion that acupuncture works can not be drawn from the Cochrane Review as she claims. The conclusion of the Cochrane review is acupuncture is worth a second look in very specific settings, which as I remember Edzard Ernst has pointed out earlier.
@Thomas, funny thing, I never wrote here that “acupuncture works.” You know how I know? Because we’re communicating in a digital format, so I ran a search of this page for “acupuncture works” and I got four hits – all of them attributable to you mis-quoting me in your comments. If you’d like to argue for precision in language and “exact definitions” then at least attempt to be accurate in attributions.
What I said was that the Cochrane Review provided evidence that acupuncture is effective for the prevention and treatment of tension type headaches and other variations of that sentiment. I specified a condition and specified that the review lent support to its effectiveness. The hypothesis is firmly supported.
Quote: “What I said was that the Cochrane Review provided evidence that acupuncture is effective for the prevention and treatment of tension type headaches and other variations of that sentiment.”
That is from here.
Lets contrast that with the review itself: “The available results suggest that acupuncture is effective for treating frequent episodic or chronic tension-type headaches, but further trials – particularly comparing acupuncture with other treatment options – are needed.” In science there is a HUGE difference between the wording “suggests” and “provided evidence”.
Quote: “We use acupuncture to treat a wide variety of conditions [….]” (From your facebook page) I don’t think you would treat people with non-working treatments, would you ? That would be highly unethical, wouldn’t it ?
Edward, you mystify me. You said “Those who followed the recent discussions about acupuncture on this blog will probably know her; she is an acupuncturist who (thinks she) knows a lot about research because she has several higher qualifications (but was unable to show us any research published by herself).”
The statement “who (thinks she) knows” is clearly meant to demean. The “because she has several higher qualifications (but was unable to show us any research published by herself)” is clearly meant to suggest that someone who has not done research is less qualified to analyse research than someone who has.
I am paying you the respect of having mastered English as a second language and therefore conclude that you know exactly what you are saying, even though you seem to want to deny it.
And the idea that you dealt with the NICE issue is simply inaccurate. You really didn’t respond to any of Mel’s key points or the overall conclusion that acupuncture was more effective than any other modality considered.
why don’t you just take the words as they are written, not as you interpret their meaning?
Peter Deadman,
The data in the study of aromatherapy and reflexology show that reflexology is more effective for rheumatoid arthritis than is aromatherapy — which has sod all to do with acupuncture, its practitioners, and its apologists, until such time that the sCAM empire [aka: alternative medicine; integrative medicine; integrative health] produces the long overdue evidence-based effectiveness ranking of its plethora of modalities for each and every health condition for which it claims to be effective.
In the meantime, either provide us with the list of sCAM modalities that are *ineffective* for: low back pain; migraines; tension headaches; IBS; rheumatoid arthritis; and persistent allergic rhinitis; etc. — or desist from pretending to know things that you don’t know.
And we all need to note (though not necessary agree with) the US National Center for Complementary and Integrative Health New Strategic Plan ([email protected];) which states as its “Objective 2 :
To Improve Care for Hard-to-Manage Symptoms:
*Develop and improve complementary health approaches and integrative treatment strategies for managing symptoms such as pain, anxiety, and depression.
*Conduct studies in “real world” clinical settings to test the safety and efficacy of complementary health approaches, including their integration into health care.”
So, here we have ‘real world’ studies testing ‘efficacy’. The term ‘efficacy’ here refers to ‘pragmatic’ studies – not ‘under ideal conditions’ as some have it!
What are we to make of that?
Just asking!
Pretty sure ‘efficacy’ in an NCCIH context simply means the efficiency with which mark and money are separated.
Heck, NCCIH alone is a billion-dollar centrifuge, so they’re clearly doing something right.
Mel says “Do you agree that this review demonstrates acupuncture’s effectiveness and efficacy for the prevention of tension type headache”
and later says ” I have been known to say that according to the best evidence, “acupuncture is effective for the treatment of headaches” and I have also been known to support this statement using systematic reviews, such as the one recently published by the Cochrane Collaboration, which I referenced above.”
But Mel herself quotes in a response upthread http://edzardernst.com/2016/06/effectiveness-efficacy-and-wishful-thinking-if-all-else-fails-invent-your-own-definitions/#comment-78099
“The available results **suggest** that acupuncture is effective…” (my emphasis)
As Thomas Mohr said http://edzardernst.com/2016/06/effectiveness-efficacy-and-wishful-thinking-if-all-else-fails-invent-your-own-definitions/#comment-78168 “suggest” has a definite meaning in a scientific study such as a Cochrane review – it means there is not enough evidence to *show* the result, just a non-definite ambiguously positive result that make it worth investigating.
So that study does not “demonstrate” effectiveness, it does not meaningfully “support” it.
This suggests to me that mel’s understanding of the exact terms used in research and clinical trials may not be good or else she places excessive confidence in studies which give limited support to the claim, perhaps due to confirmation bias.
“This suggests to me that mel’s understanding of the exact terms used in research and clinical trials may not be good or else she places excessive confidence in studies which give limited support to the claim, perhaps due to confirmation bias.”
“As Thomas Mohr said “suggest” has a definite meaning in a scientific study such as a Cochrane review”
Fascinating, can you provide a reference to this definite meaning for how this “term” is used in Cochrane reviews?? I just searched the Cochrane handbook and while it uses the word “suggest” 59 times in a way entirely consistent with how I’ve interpreted it, no where does it define it as a term, having a special or specific meaning as you guys suggest. So I’d love to see your reference or some objectively validated tool that grades the subjective verbiage used by authors in their conclusions section.
The systematic review I’ve referenced supports my assertions in its forest plots and GRADE tables, which is what researchers look at when they want to understand what a MA/SR shows. An obsessive over-reliance on the conclusion section of the abstract (which would have you flunking any research skills class), suggests that a) you didn’t read the review or b) you don’t understand review methodology in general.
The interpretation of the conclusion is important in this area because many pseudoscientists will only read that.
Glad to see you consider the GRADE values important, changing the evaluation to arrive at a higher score should not have happened.
Concentrating on derived statistics is what no research scientist does. You might pass some low grade exam by doing so but once you are in the real world of scientific research you want the actual data. The actual data in this study, as far back as is possible from the published work, shows so many problems such that even the use of the word ‘suggests’ is problematic. The authors would have been safer with keeping to the tired old ‘more work required’.
Ernst, you’re such a tool! The only reason they wheel you out to comment is because the media know you’re going to keep their sponsors happy. What I can’t work out about you is whether you are a genuine fool who accidentally hit the big time… or you’re just a paid up member of team Pharma. Ps breast cancer screening
http://edzardernst.com/2012/12/ad-hominem-attacks-are-signs-of-victories-of-reason-over-unreason/
@ Dan on Tuesday 07 June 2016 at 20:50
“Ernst, you’re such a tool! The only reason they wheel you out to comment is because the media know you’re going to keep their sponsors happy.”
You seem to have some inside knowledge about the prof. Would you care to share it, or are you mouthing off without any substance? Don’t worry, the answer is clear.
“What I can’t work out about you is whether you are a genuine fool who accidentally hit the big time… or you’re just a paid up member of team Pharma. Ps breast cancer screening”
This raises some questions, none for which you will have an answer. The first is, of course, how the prof’s appointment to Exeter was an accident? Again, if you have any inside knowledge, please spill your guts? The second is; how does one become “a paid up member of team Pharma”? Is there an online application page, is there a minimum qualification, what is the membership fee, what is the application and approval process, and what are the commitments if admitted?
I suspect, however, there is a tool among us, and it isn’t the prof.