While looking up an acupuncturist who has recently commented on this blog trying to teach me how to do science and understand research methodology, I was impressed that he, Dr John McDonald, PhD, has just published a clinical trial. Not many acupuncturists do that, you know, and I very much applaud this action, which even seems to have earned him his PhD! McDonald is understandably proud of his achievement – all the more because the study arrived at positive conclusions. This is what he wrote about it:
…So, in a nutshell, acupuncture is a safe, effective and cost-effective treatment for allergic rhinitis which produces lasting changes in the immune system and hence improvements in symptoms and quality of life. Dr John McDonald
Fascinating! I quickly looked up the paper. Here it is:
This new study was designed as a randomized, sham-controlled trial of acupuncture for persistent allergic rhinitis in adults investigated possible modulation of mucosal immune responses. A total of 151 individuals were randomized into real and sham acupuncture groups (who received twice-weekly treatments for 8 weeks) and a no acupuncture group. Various cytokines, neurotrophins, proinflammatory neuropeptides, and immunoglobulins were measured in saliva or plasma from baseline to 4-week follow-up.
Statistically significant reduction in allergen specific IgE for house dust mite was seen only in the real acupuncture group. A mean (SE) statistically significant down-regulation was also seen in pro-inflammatory neuropeptide substance P (SP) 18 to 24 hours after the first treatment. No significant changes were seen in the other neuropeptides, neurotrophins, or cytokines tested. Nasal obstruction, nasal itch, sneezing, runny nose, eye itch, and unrefreshed sleep improved significantly in the real acupuncture group (post-nasal drip and sinus pain did not) and continued to improve up to 4-week follow-up.
The authors concluded that acupuncture modulated mucosal immune response in the upper airway in adults with persistent allergic rhinitis. This modulation appears to be associated with down-regulation of allergen specific IgE for house dust mite, which this study is the first to report. Improvements in nasal itch, eye itch, and sneezing after acupuncture are suggestive of down-regulation of transient receptor potential vanilloid 1.
These conclusions seem to be based on the data of the study. But they are oddly out of line with the above statement made by McDonald about his trial. What could be the reason for this discrepancy? Could it be that he behaves ‘scientifically’ correct when under the watchful eye of numerous co-authors from the School of Medicine, Menzies Health Institute, Griffith University, Queensland, Australia, the School of Medicine, Menzies Health Institute, Griffith University, Queensland, Australia, the National Institute of Complementary Medicine, Western Sydney University, Sydney, Australia, the Health Innovations Research Institute and School of Health Sciences, RMIT University, Melbourne, Victoria, Australia, and the Stanford University, Palo Alto, California? And could it be that he is a little more ‘liberal’ when on his own? A mere speculation, of course, but it would be nice to know.
Anyway, the trial itself raises a number of questions – unfortunately I have no access to the full paper – which I will post here in the hope that my acupuncture friend, who are clearly impressed by this paper, might provide the answers in the comments section below:
- Which was the primary outcome measure of this trial?
- What was the power of the study, and how was it calculated?
- For which outcome measures was the power calculated?
- How were the subjective endpoints quantified?
- Were validated instruments used for the subjective endpoints?
- What type of sham was used?
- Are the reported results the findings of comparisons between verum and sham, or verum and no acupuncture, or intra-group changes in the verum group?
- Was the success of patient-blinding checked, quantified and successful?
- What other treatments did each group of patients receive?
- Does anyone really think that this trial shows that “acupuncture is a safe, effective and cost-effective treatment for allergic rhinitis”?
70 Responses to Acupuncture for allergic rhinitis? A new study and plenty of old questions
Leave a Reply to DavidP Cancel reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Having worked at the National Institute of Complementary Medicine for a couple of years I noticed that their modus operandi is to be extremely careful with what they write in a scientific paper – if they are caught out even once for falsification or fabrication they know that they will be gone. For them the only solution for this problem is to go and sell the results of their paper, which is usually neutral or negative, in a very positive way by being extremely liberal with the truth. Simple reason for this is that they know the public will not read the scientific publication, either because the paper is not accessible or they just don’t care. And they also know that there is currently no rules on how you convey your research results to the public – you can basically say what you want. Below is a link where a rather neutral result was conveyed to the public via a national radio interview as being extremely positive – the next big thing in science.
https://frankvanderkooy.wordpress.com/2016/02/22/misleading-the-public-complementary-medicine-at-universities-part-1/
Thank you for the information Frank van der Kooy.
No worries George
Question is; Do you know of any rules or regulations that “control” what and how a scientist translate their research results to the public? Furthermore is there any penalties if you clearly mislead the public? I guess it is a grey area.
I have not read this PhD. So I’m talking in general.
There are reports of scientific fraud in the literature. (Of full impact trials not PhDs.) Several professors were discussing the identification of the fabrication of trials. In this instance they have been removed. It is considered an offence. Misconduct.
However, there is a difference between knowingly and unknowingly commiting an offence. I suspect again this PhD aside, that many people may reflect on their past work, and identify areas for improvement. For example Professor Edzard PhD – He may acknowledge in hindsight areas for improvement, mistakes made as part of an ongoing learning process.
Also, it begs the question regarding the extent of supervision a “student” receives. Also the Quality of scientific education. Who should conduct trials or who are better placed. Teamwork.
What is very important I would suggest is the authors discussion, acknowledging limitations, and making accurate recommendations. Rather than only “findings”. I’d like to see advances in these sections with more weight provided to sectional critical thinking along with more active promotion and presence in abstracts.
Bests
true! we are all on different learning-curves; the question is: how low are we allowed to start?
I fully agree with you. If I read some of the things that I’ve written 10 or more years ago I would be able to identify a number of things that I would like to improve – with age comes wisdom! Key question, in general, is the important point of knowingly or unknowingly inflating a result. My experience tells me that in most cases, at my former employer at least, that they knowingly provide the public with one-sided overly positive results while that is not reflected in the publication itself. How do I know this: Because I usually told them this – you cannot present only an overly positive result without providing the limitations of your experimental design, and also give a balanced unbiased view in your recommendations. Unfortunately none of my pleas had much of an impact – but I am working on it!
Yes, we can all learn. In the scientific community it is accomplished by criticism. Nobody likes their work being criticised but if you reject it, you will not improve.
It is totally fair to acknowledge every scientist’s personal learning curve, but — as Edzard asks — how low are we allowed to start? I have always appreciated having my papers rejected by a good journal. One can learn swiftly by digesting strong criticisms from effective peer reviewers. But from my (considerable) experience on the editorial side of scientific publishing, I’m aware there are quite a few professional scientists who routinely protest referees’ criticisms and come back again and again with a barely altered manuscript. They try to argue their work into print. This is not how good science should be done: it’s surely a case of confirmation bias for whatever conclusions the protesting author wishes to see in print.
Sadly, the current rate of increase of new journals in all fields must make it easier for poor quality work to be accepted for publication: editors have to fill their pages somehow!
Edzard, so let’s revisit.
It is designed as a placebo randomised controlled trial of acupuncture (insertion of an acupuncture needle into specific locations on the body according to acupuncture theory vs sham acupuncture (insertion of needles into non specific locations on the body in opposition to acupuncture theory).
So in essence, it was a randomised needling technique trial.
If the trial was comparing “acupuncture” then you would at least expect theoretical comprehension of the condition in acupuncture terms. Including (at least) an acupuncture treatment consensus which includes rationale for the acupuncture needling sites considered “acupuncture” similarly rationale for the acupuncture sham no acupuncture point sites considered no acupuncture or pretend acupuncture site.
Is there reporting of the acupuncture interventions? Or is that limited.
Bests
So really this was bizare experiment on needling?
as McDonald is now responding again, you better ask him all these question, I think.
This trial was registered with the Australian New Zealand Clinical Trials Registry, Identifier: ACTRN 12610001052022.
The title of the paper is: “Effect of acupuncture on house dust mite specific IgE, substance P, and symptoms in persistent allergic rhinitis ” yet the title of the registration is less specific: “The effect of acupuncture on mucosal immunity in the upper respiratory tract in allergic rhinitis patients”.
According to the registration:
The outcomes were defined as:
Note the mention of measurements at 3, 6, 9 and 13 weeks, but that the parameters to be measured are not specified.
The trial registration is dated 26/05/2009 but was ‘Retrospectively registered’ 18 months later on 30/11/2010. It took another five years to publish the paper.
Like Prof Ernst, I don’t have the full paper, but I think several more questions need to be added to prof Ernst’s list.
What are the additional several more questions to be added to Professor E. Ernst’s list Alan?
Best wishes
I’m not able to access the paper from my institution, either. I refuse to pay $35 to read an article. Dr John McDonald, PhD might think in future to submit his work only to journals that provide open access, or seek funding to pay for open-access.
Edzard generously omitted the tedious numerical details from his version of the abstract, but I’d like to draw attention to the following points.
“Statistically significant reduction in allergen specific IgE for house dust mite was seen only in the real acupuncture group, from 18.87 kU/L (95% CI, 10.16–27.58 kU/L) to 17.82 kU/L (95% CI, 9.81–25.83 kU/L) (P = .04).” Anyone who can measure IgE to four-digit precision deserves an award. (The same does not apply to the journal, its editor or the referees of the paper for allowing this basic error to appear in print.) The mean difference may reach a p-value of 0.04, but it is naive to imagine this demonstrates statistical significance, or — most certainly — anything clinically meaningful. Just look at those ranges: 2.5-fold in both groups!
In this post, David Colquhoun patiently explains how we fool ourselves with p-values. Take home message for Dr John McDonald, PhD is “If you observe a P value close to 0.05, your false discovery rate will not be 5%. It will be at least 30% and it could easily be 80% for small studies.”
“A mean (SE) statistically significant down-regulation was also seen in proinflammatory neuropeptide substance P (SP) 18 to 24 hours after the first treatment from 408.74 (299.12) pg/mL to 90.77 (22.54) pg/mL (P = .04). No significant changes were seen in the other neuropeptides, neurotrophins, or cytokines tested.” Here we have five-figure precision in quoted means for a cytokine measurement. Take another award, Dr John, PhD. And once again, p=0.04.
For these measurements we have no ranges quoted. Dr John, PhD instead resorts to the cheap trick of quoting standard errors of the mean (SE) instead of standard deviations (SD) to make the huge ranges of measured variation look smaller. The critical question to be asked about these data is: how many cytokines, neurotrophins and proinflammatory neuropeptides were measured and at how many timepoints? The results show a barely significant difference (probably a false discovery, as explained above) in a single parameter at a single time point. The kits used to measure these things typically allow for dozens of parameters to be measured, and the implication is given that the measurements were repeated on samples at other time points. Never mind false-discovery, we have a distinct possibility of a type 1 statistical error here.
Finally, we are given no details or p-values for the clinical measurements.
I think the abstract provides enough detail for me to draw a conclusion about Dr John McDonald, PhD’s assertion that “So, in a nutshell, acupuncture is a safe, effective and cost-effective treatment for allergic rhinitis which produces lasting changes in the immune system and hence improvements in symptoms and quality of life.” In fact, this study shows that acupuncture in no way modulates the mucosal immune response or produces “lasting changes in the immune system”. The study does not even address issues of safety or cost-effectiveness. It is a shining example of the statistical incompetence that mars so much of the biomedical literature. Further studies are almost certainly not warranted.
I’d certainly like to see the data that produces any sort of significance with those ranges of confidence.
I note that although sham controls are mentioned, nothing is said about blinding.
Don’t NEED to have a mention of blinding.
Precise blinding is important, were the acupuncturists aware of which needles they were using? Were the assessment clinicians aware of the patient groups? Were the technicians performing the serology aware of which groups their samples came from? The use of the term blinding rather than double or even triple blinding indicates the answers for each is yes.
“The use of the term blinding rather than double or even triple blinding indicates the answers for each is yes.”
It depends on a trial design. A pragmatic randomised control trial may not blind participants or the person delivering the intervention. So, one is better to apply considered judgement regarding blinding.
Of course, the data collectors, or assessor may be blind to allocation. Yet, no blinding doesn’t necessarily mean that the results are always positive or significant.
This is because there are non-blinded trials with non significant results.
My point was that there are various design options. The double blinded placebo trial is one design. We don’t always require a double blinded placebo randomised controlled trials to evaluate the effectiveness of an intervention. There are other options.
Moreover, the term “blind” identified via a title or abstract is questionable whether it would be there in the first place. In other words, no mention of “blind” in the title or abstract would not be unusual or indicative that blinding was or was not conducted. One would need the full report.
Bests
Acupuncturists are most likely to be aware whether they are inserting an acupuncture needle into their patient or not.
Were the technicians performing the serology aware of which groups their samples came from….
Good, specific and reasonable question. Better if data was reported in the abstract, failing that in the full report. Or if not, rationale for why not. Yet maybe that was a given anyway.
Bests
Having seen the paper it is clear that the acupuncturist was fully aware of which group he was treating. This is a serious failing considering the claims made in the conclusions. That others have done the same does not make this paper immune from criticism.
The paper mentions that others eg the serologists were masked from the groups. This implies that results and samples were supplied in groups and it is easy to see that methodological biases can creep in between groups.
“Having seen the paper it is clear that the acupuncturist was fully aware of which group he was treating.”
Um. I am not sure I agree as to the relevance of this… Say for example subjects were randomised into two intervention arms by a third party. It should not matter that the therapist knew he was delivering intervention a) to the randomly allocated subjects.
“This is a serious failing considering the claims made in the conclusions.”
I am still not convinced that this is a or the serious failing regarding conclusions.
“That others have done the same does not make this paper immune from criticism.”
No paper is immune to criticism. If people take the time to comment and apply their expertise -this is something to appreciate.
That others have done the same is not because that is a flaw, it is the use of various design and methodological procedures that are acceptable and necessary (topic depending). The aim is however, to design and conduct a robust trial. No blinding doesn’t mean that a trial is not robust. You could have blinding in a trial but the incorrect utilisation of statistical analyses.
“The paper mentions that others eg the serologists were masked from the groups.”
“This implies that results and samples were supplied in groups..”
Not necessarily. It could be that the serologists were blind to allocation intervention arms.
Nonetheless, better reporting of methodology is required. Sometimes it is difficult to even ascertain who recruited subjects, who randomised subjects, how was this done.
Bests
“Say for example subjects were randomised into two intervention arms by a third party. It should not matter that the therapist knew he was delivering intervention”
I disagree, it matters a great deal.
Me: “Say for example subjects were randomised into two intervention arms by a third party. It should not matter that the therapist knew he was delivering intervention”
Edzard: “I disagree, it matters a great deal.”
Me: That is fair enough. If you discuss why a non-blinded therapist/teacher/other delivering the treatment/particular intervention matters a great deal – that would be useful.
Bests
it’s in the post: the therapist cannot help to influence the outcome by verbal and non-verbal communication with the patient. this effect can be particularly powerful, if [like in this trial] there is just one therapist with a strong vested interest in the outcome.
“it’s in the post: the therapist cannot help to influence the outcome by verbal and non-verbal communication with the patient. this effect can be particularly powerful, if [like in this trial] there is just one therapist with a strong vested interest in the outcome”.
But you said yourself there was not much difference between the intervention arms. So, that kind of demonstrates limited power not particularly powerful power.
The similarities and differences of the interventions should always be explicitly described.
In my view, a pragmatic randomised controlled trial would have been a better design. If there are therapist effects, this can be accounted for in the sample size and analysis.
Bests
not very logical, I am afraid.
there is a potential for influence on the outcome; this is why one needs to control for it as good as possible.
perhaps the use of medication would not have caused a difference between groups; but because it MIGHT, one tries to control for this and other confounding factors.
“not very logical, I am afraid.
there is a potential for influence on the outcome; this is why one needs to control for it as good as possible.
perhaps the use of medication would not have caused a difference between groups; but because it MIGHT, one tries to control for this and other confounding factors.”
It depends. We are talking about therapist interaction. What is the design of this randomised controlled trial?
Edzard, after reviewing the full paper e.g. did it set out to be a placebo trial? How would you describe the actual trial design?
Does the full report indicate that the only difference between the two groups was that the needles were placed in non-acupuncture points? Which were?
I am waiting for access to the full paper.
Bests
it was a placebo-controlled trial; placebos mimicking physical interventions like acupuncture are called ‘Sham’
“Does the full report indicate that the only difference between the two groups was that the needles were placed in non-acupuncture points?” yes; there is little other reason to use sham
“Take home message for Dr John McDonald, PhD is “If you observe a P value close to 0.05, your false discovery rate will not be 5%. It will be at least 30% and it could easily be 80% for small studies.”
Can you explain this further please in context? Apologies.
Please read the article at the link I posted: it explains things more clearly than I can.
Ok.
You think a lot. At least read the report first!
“I think the abstract provides enough detail for me to draw a conclusion about Dr John McDonald, PhD’s assertion”
“In fact, this study shows that acupuncture in no way modulates the mucosal immune response or produces”
“The study does not even address issues of safety or cost-effectiveness.”
“It is a shining example of the statistical incompetence that mars so much of the biomedical literature.”
“Further studies are almost certainly not warranted.”
Oh. Ok. I’ll take your word for it shall I!
Flattery!
“Edzard generously omitted the tedious..”
After you have read the full text and answered most of your questions for yourself, it might then be a more appropriate time to engage in any meaningful discussion, if that is in fact your intent.
yes, please post it or email it to me
As you are no doubt aware, the paper is published by Annals of Allergy, Asthma and Immunology and is available from their website.
as you doubtlessly have seen, my next post is also about your study
why the silence all of a sudden? have you mislaid your own article?
In the same way as Edzard Ernst has asked Mel to hold fire till the final NICE guidelines appear?
I have now had sight of the full article. I have limited time today to devote to this matter. A full critique of the paper would run to great length. To summarize the main points, the primary outcome measure was any modulation in upper respiratory tract as measured by total and allergen specific IgE, salivary Substance P and VIP, and 9 pro-inflammatory cytokines, neurotrophins or neuropeptides. Secondary outcomes were changes in symptoms, quality of life and nasal airway clearance.
When you are close to a study you are often blinded to what an outsider can see clearly. Your paper is a mass of paired t-tests, Friedmann tests and p values, but the actual numbers all tend to fall within a fairly comparable range. Comparing a series of measurements done over a period of time, for example the Substance P data and the symptom scores, with baseline by means of separate paired tests strikes me as statistically suspect. If, by chance, the baseline value for a parameter was unusually high or low, then the separate tests of paris will almost certainly show a significant difference from baseline, but that’s different from biological significance.
You omit the baseline (week 0) values for salivary Substance P. Why is this? For every other parameter you compare subsequent measurements with week 0, but for Substance P (which you measured at week 0 — see Fig. 1 ) the comparisons are with week 1.
Throughout the paper data are presented to ridiculous levels of precision. Readers can make what they will of weekly means for daily symptom scores ranging (in all 3 groups) from 15–24 with standard deviations ranging from 10–19, but sure as eggs, paired comparisons only for the real acupuncture group reach statistical significance, with p-values from 0.001 to 0.02.
The paper presents a single Table 3, results for the Mini-Rhinoconjunctivitis Quality of Life Questionnaire yet the text also refers us to a (non-existent) Table 3 for the results of the Instantaneous total nasal symptom score.
Sorry, Dr. MccDonald, PhD. You told us proudly your study showed all sorts of things it just doesn’t show, it is statistically highly suspect and does not do the reputation of acupuncture research much good.
Had you read the full text of the paper you would see that an allergy specialist screened all patients for the study, and that skin prick tests and RAST tests were performed in the rooms of the allergy specialist under his supervision.
There is a lot of data which was not included in the final draft which was included in earlier edits. My apologies for the omission of the Instantaneous Nasal Symptoms scores table. This table was apparently removed in the editing process as we were over the permitted limit of tables and figures, and I did not notice the remaining reference to it in the text. I am happy to provide the missing table if I can find a way to post it here. The Week 0 substance P data I am also happy to provide. I think it important to clearly separate what I said in my blog and what was contained in the report of the study. In my blog I drew on the whole body of acupuncture research into allergic rhinitis which included cost-effectiveness studies and follow-up period of up to six months whereas we only measured up to four weeks post-treatment. This was an early exploration of potential immune modulation underpinning the symptomatic improvements which have been reported consistently. In truth we did not find very much apart from a modest decrease in dustmite specific IgE and a dramatic decrease in Substance P which however did not correlate with the time course of improvements in signs and symptoms. Hence it is unlikely that either IgE changes or substance P changes alone could account for the improvements in signs and symptoms. There appears to be a lasting modulation produced by an adequate course of acupuncture treatments (a minimum of 12 treatments at least twice weekly as referred to in the paper) as all studies published to date have reported that symptoms continue to improve for at least three months after treatment ceases. The nature of this apparent lasting modulation is unknown, and the role that short term modulations in pro-inflammatory neuropeptides, cytokines and neurotrophins might play in this lasting modulation are also unknown. Some symptoms improved in this study while others did not and this may provide some clues for further studies in terms of suggesting what may have changed. Clearly there is still much to learn about how acupuncture works.
Here is the data on iTNSS you requested:
Table 11. Instantaneous total nasal symptom scores (iTNSS): group means with standard errors
No acupuncture (n = 37) Sham acupuncture (n = 37) Real acupuncture(n = 36)
Week Mean (±SE) Mean (±SE) Mean (±SE)
0 4.8±0.5 5.0±0.4 5.4±0.5
9 4.3±0.3 4.2±0.5 3.9±0.5*
12 4.3±0.4 4.3±0.5 3.6±0.5*
Abbreviations: n – number of subjects, SE – standard error, SD – standard deviation
* p < 0.05 compared to Week 0
Table 12. Instantaneous total nasal symptom scores (iTNSS): time-point differences within groups
Week 0 vs Week 9 Week 0 vs Week 12
Group n Difference(95% CI) P value n Difference(95% CI) P value
Real acupuncture 36 1.5 (0.4 to 2.7) 0.010* 36 1.9 (0.7 to 3.1) 0.003*
Sham acupuncture 37 0.8 (-0.3 to 2.0) 0.147 37 0.7 (-0.5 to 1.9) 0.257
No acupuncture 37 0.5 (-0.4 to 1.4) 0.259 37 0.5 (-0.4 to 1.5) 0.244
Abbreviations: n – number of subjects, 95% CI – 95% confidence interval
Unfortunately the table formatting which I re-edited here disappeared on posting, making the tables hard to read.
I have just reviewed the original data. The week 0 substance P was not analysed by the laboratory technician as the Week 1 pre-treatment sample was taken as the baseline as treatment had not yet commenced (and it is unlikely to have been any major differences between a pre-treatment sample taken in Week 1 and another pre-treatment sample taken a week earlier in Week 0).
@Dr John McDonald, PhD
Thank you for supplying the data. Please see the next post in this blog for many criticisms of your paper from Prof. Ernst and others. I regret I am not prepared to waste further time analysing these extra data. Your study is a clunker.
Dear John,
I d like to let you know I thought your paper was interesting and that I can imagine the amount of hard work that went into your paper. I have to say as well that I have felt very uncomfortable reading the comments, almost like anyone’s own personal nightmare. I’d hate to think that you have been stressed out about this, or that you have felt humiliated. Hence the reason for this response. It’s easy to critise yet harder to embrace and welcome a person and teach, encourage and guide them. Sometime academia can be torturous, other times you take away with you some of the best skills and techniques one could only but dream of. I personally wish you a promising future in research. If most people reading this had not experienced naming and shaming before, I dread to think of their experiences hence behaviour as a result. Best wishes George Boc Scientist
George,
I tend to agree with you but also disagree with you. I agree that John, as the inexperienced author, should not be named and shamed because he might have unintentionally inflated these results publicly (although this is even difficult to believe) The blame should go to the senior authors who, based on their track record, have inflated these results intentionally. The main problem is that this trial and all the other trials of the senior author is designed and publicly reported on in such a way that it creates public trust in a pseudo scientific medical intervention. One can argue that acupuncture for hay fever is a trivial thing but it is the public trust in acupuncture that is the problem. People with a more serious medical condition can now think that acupuncture (and other CM interventions) might work for them, because it has been shown to “work” in this study (and all the other studies of the senior author), and this can cause them harm and dare I say even death (due to delaying effective treatments). This is the problem and in my view this is serious enough to respond in the way that people have responded on this blog post – although I think the focus should have been on the senior author.
Frank,
“I think the focus should have been on the senior author.” Perhaps, but it was the first author who chose to publicize it on his website with a no-holds-barred, inflated and unjustifiable conclusion: “So, in a nutshell, acupuncture is a safe, effective and cost-effective treatment for allergic rhinitis which produces lasting changes in the immune system and hence improvements in symptoms and quality of life.”
In any case, the convention in science publishing is that all authors hold individual and joint responsibility for the entire paper. Any criticisms I made about the study therefore relate to the authors jointly, notwithstanding which the first author clearly took principal ownership and responsibility for the study, as seen by his reponses on this blog, which is why I addressed criticisms to him personally. He could have sought support and advice from his co-authors, but chose not to.
Yes, you are right. It is indeed the public statement that has done the damage here.
It is just a pity that the experienced people, who should know better, seems to be getting away with it. It happens quite a lot – I am speaking in general now- that once (un)intentional mistakes are highlighted that the senior authors keep quiet or they blame it on the inexperience of the junior author. Which begs the question if Prof Smith has responded or provided any answers yet?
Just to mention that there maybe better ways to make your points against complementary medicine. Critique, but if anyone is to take matters seriously, a great deal of compassion and professionalism means that not only would the public take your comments on board so would those comp med practitioners. Of course this isn’t the deal there would be extreme cases. But it’s better to make judgement appropriately, which includes an appropriate acceptable manner. In my day, if I had responded with a letter to the editor in this way, I’d be a laughing stock. So, while I think debate is good there is a point where one could use some reflexitivity and demonstrate their skills in a less provocative, one sided manner.
However, what’s the probability of that?!
Bests to all
Hi Frank,
I understand, I just had this awful feeling for example someone who was vulnerable, perhaps already snowed under could take these type of public comments which could be enough or let’s say result in unintended consequences.
I understand Edzard perspective, why should for an example complementary medicine make claims that are not evidenced based? Why does complementary medicine do this? What is acupuncture? How do the effects of acupuncture contribute to healing? What, why, when, how? But this type of naming and shaming is enough to cause unintended consequences. It’s equilivant to professional humiliation. No one is to blame, but it just crossed my mind. I know of some of the best researchers query there work with sleepless nights. It is what it is and I really don’t think John PhD would have commented John PhD if he had been aware of the possible problems and issues with his work, that no one else with better skills was able to take a look until now and say hang on a min. It shows more issues with education system than anything else.
It would be good to state what trial designs one expects, what to aim for
Hope you all have a nice day.
George.
McDonald claims a PhD which used to be by public examination. He published his work and garnered publicity by making misleading statements to support a business that makes plenty of claims but somehow fails to back up those claims with evidence. He obviously has no trouble exposing his work but he must be protected from criticism? How about the public should be protected from his claims?
hear, hear
It’s not about “protection” of the author from “criticism” or “protection” of the “public.” It regards the way in which communication occurs. The setting for which issues raised have been able to occur in the first place.
Your comment regarding “misleading” is fair enough, as we know there are two sides to the meaning, but as we also know this is the twentieth century, people don’t get “stoned” to “death” anymore. There are alternate, more effective, and academic methods of communication. To be honest it was just my own opinion, I felt concerned for the welfare of this gentleman misleading unintentionally or not. It reminded me of a nurse who committed suicide following an incident. You never know that’s all I’m pointing out. I think the public would be upset that a man had had his life’s hard work ripped apart publically in an unkind manner, one that set out to degrade him in the start. Why search for the article? We have every right but it begs the question of motivation, incentives of the critics. As I said no one is to blame, it’s how it is. We are all responsible for our own actions. The gentleman could come back with interesting information. He at least tried – but someone told him not to bother! That it’s worthless. If educational standards and support have reduced that’s the main issue and I can’t be certain about the business intentions only acts of humility.
Bests
I for one do understand. but I don’t think that a man who posts stuff like this is going to suffer from some reasonable criticism: John McDonald “Edzard Ernst has just invented a new level of evidence. Edzard has blogged about a blog I wrote about a published paper which he has not read (by his own admission). So where does a blog about a blog about an unread research trial sit on the EBM pyramid? In his blog Edzard manages to “raise questions” about the unread paper which are covered in the published paper (all coated in the usual patronising and supercilious sarcasm). I have suggested he might like to read the full text before attempting to engage in a discussion.”
George wrote: “but as we also know this is the twentieth century”.
Perhaps that explains his asinine comments.
Edzard, thank you for the comment. I’m pleased to hear that JM appears “ok”, it is a shame to read of his reaction… Although I do wonder what’s a man supposed to do? Sometimes we all act in ways we don’t mean to make ourselves feel better. But you make a valid point. I have noted that.
I better get back to work.
Bests
@george
Last time I looked it was the twenty first century but I’ve always been accused of living in the past.
The public are misled by these conclusions dressed as science and suffer real harm. You may not think that taking money from someone for a treatment that doesn’t work isn’t harmful, let us put the facts to society and let them judge if that is what they want.
The author may indeed and probably is ill educated in scientific practice but to maintain he is some wilting flower after publicly misrepresenting his results is too much. He is a long term acupuncturist, just what did he do in original research to gain a PhD, this paper certainly isn’t a scientific exercise?
There are not two sides to the meaning of this exercise. The balance lies entirely with evidence based medicine which shows that acupuncture is a theatrical placebo. This paper shows nothing, the grandiose conclusions of the author are wrong.
You question my motives? They are totally and utterly selfish. I don’t want to waste my time trying to persuade friends and relatives to get proper treatment. Also, I want to have conversations with anyone and not have to cringe inwardly when someone makes the same basic error that this author made.
As to MacDonald’s silence, I would hope that he is in deep discussion with his co-authors about the deficiencies found in this study. However I suspect that a studious indifference to the criticism will be maintained and if the pressure becomes too great that he will be sacrificed on the altar of his peer’s business plans.
Cue the next quack to say that science shows that acupuncture can cure your allergies.
Dear Gentle Goerge,
I agree with your sentiments, but long experience tells me that reasoned education, however stated, will not convert people who have an entrenched financial interest in altmed. They have undergone ‘training’ in their particular ‘speciality’, often investing years of time and a lot of money to do so. Like all adherents to a faith-based position they usually remain steadfast, no matter in how subtly persuasive a manner the arguments against them are expressed.
In the present specific case of Dr McDonald, he is a medically qualified individual who has convinced himself of the therapeutic value of acupuncture. He ran a clinical study which effectively showed acupuncture was of no value in treatment of allergic rhinitis but he chose to spin the results as positive. He was let down, successively, by his PhD supervisor, his examiner(s), the referees of his paper and the editor of the journal who accepted the paper, none of whom pointed out to him the flaws that have been picked up in many comments on this blog.
Go to any respectable scientific meeting and you will often see the work of professional scientists torn apart publicly in Q&A sessions; occasionally in a very unpleasant manner. The world is not a kind place, and science is no exception to this rule. The persistent lobbying of camists of all persuasions to gain public acceptance for their practices in the absence of genuine supportive evidence is also unkind and represents a danger to some vulnerable patients. This particular blog is but one attempt to light a candle of reason in an area beset by many distorted and confabulated messages.
(emphasis mine)
Is “George Boc” your real name? A quick search is unable to find any mention of that name in connection with science or publications. NLM for example, turns up Zero in all databases on “Boc G [Author]”; LinkedIn has one George Boc, apparently a Romanian auditor.
Perhaps misspelled?
Are acupuncturists qualified to diagnose persistent allergic rhinitis, and are they authorised to appropriately prescribe the vital medicines that are available to those whose persistent allergic rhinitis sometimes/often results in an acute asthma attack?
Pete. If you have a point just make it. Why all this nonsense.
I take your response to be a resounding “No!” to both questions.
The fundamental question is: Are acupuncturists practising medicine without a license and/or without having in place an adequate reporting mechanism? My original questions were, of course, “nonsense” to *all* vendors of quackery.
Do you have the full report?
It is difficult to determine what is the primary outcome for this trial. It is not clear. However, it doesn’t mean that there was not a primary outcome. It is not the first and only trial to not have a clear primary outcome reported in the “abstract”. It is also not clear what are secondary outcomes or ad-hoc – although one could make a judgement. Better to read the actual full report.
“What was the power of the study, and how was it calculated?”
“For which outcome measures was the power calculated?”
The two questions do appear to be in fact one question. However, we would be able to determine the primary outcome (or other relevant data) having information regarding sample size determination or analysis. These problems have been raised before i.e. to what extent do trials adequately report sample size assumptions and components – this is not new to research conduct (as you highlight).
“How were the subjective endpoints quantified?”
I am not able to understand this sentence. Can you elaborate or be more specific please?
“Was the success of patient-blinding checked, quantified and successful?”
It is now a recommendation not to “check” the success of patient blinding. See CONSORT for rationale. (If you are interested.) Blinding is not really a big issue as you make out, you may be surprised to know. You can have pragmatic trials no blinding and no “positive” results. Blinding depends upon the trial design.
There is no mention of a blinded trial and there does not necessarily need to be patient blinding.
More important considerations such as who generated the random allocation sequence, and was this person different to the person who enrolled participants is one example.
“Does anyone really think that this trial shows that”: “acupuncture is a safe, effective and cost-effective treatment for allergic rhinitis?”
I have not accessed the full report. Can you post it here? One would expect an economic evaluation alongside the trial (or other method) for a statement. It depends on what is meant by “cost-effective” what perspective was chosen and the comparison(s). Also how cost-effectiveness was determined (data used). The author is likely to have undergone some form of evaluation regarding cost. Be interesting to find out more.
Bests GB
My first thought was “How many immune parameters did they measure?” The answer is lots, and the abstract claims only one dropped statistically (but not clinically) significantly. That’s p-hacking unless they corrected for multiple tests.
A single result dropping only at “18 to 24 hours after the first treatment” is just a random fluctuation, especially since the trial specification does not include any tests 18 to 24 hours after the first treatment – again p-hacking.
The subjectively assessed nasal obstruction etc. certainly raise the question of blinding – the treating acupuncturists certainly knew whether the patient was in the control or real acupuncture group and that can affect their interactions and thus patient’s survey responses. The abstract doesn’t actually claim the subjective results didn’t improve in the other groups. Assessing blinding effectiveness and dealing with its impact is a whole extra level of metathinking that is beyond a naive technologist like me.
As usual, an abstract without the full paper is almost worthless. It would be helpful if Dr McDonald could send a copy to Prof. Ernst.
The full text has been published by Annals of Allergy, Asthma and Immunology and is available from their website.
Thank you John. Have a good weekend.
Bests
Dr John McDonald PhD of the Southport Acupuncture Clinic (Queensland) correctly states “the American Academy of Otolaryngology Head and Neck Surgery Foundation updated their Clinical Practice Guidelines for allergic rhinitis”, but he is wrong to state “they recommended acupuncture as a treatment option for patients preferring a non-pharmacologic therapy. (April 2015: http://oto.sagepub.com/content/152/1_suppl/S1.full).
It is worth checking those guidelines in full:
“Action Statements The development group made a strong recommendation that clinicians recommend intranasal steroids for patients with a clinical diagnosis of AR whose symptoms affect their quality of life. The development group also made a strong recommendation that clinicians recommend oral second-generation/less sedating antihistamines for patients with AR and primary complaints of sneezing and itching. The panel made the following recommendations: (1) Clinicians should make the clinical diagnosis of AR when patients present with a history and physical examination consistent with an allergic cause and 1 or more of the following symptoms: nasal congestion, runny nose, itchy nose, or sneezing. Findings of AR consistent with an allergic cause include, but are not limited to, clear rhinorrhea, nasal congestion, pale discoloration of the nasal mucosa, and red and watery eyes. (2) Clinicians should perform and interpret, or refer to a clinician who can perform and interpret, specific IgE (skin or blood) allergy testing for patients with a clinical diagnosis of AR who do not respond to empiric treatment, or when the diagnosis is uncertain, or when knowledge of the specific causative allergen is needed to target therapy. (3) Clinicians should assess patients with a clinical diagnosis of AR for, and document in the medical record, the presence of associated conditions such as asthma, atopic dermatitis, sleep-disordered breathing, conjunctivitis, rhinosinusitis, and otitis media. (4) Clinicians should offer, or refer to a clinician who can offer, immunotherapy (sublingual or subcutaneous) for patients with AR who have inadequate response to symptoms with pharmacologic therapy with or without environmental controls.
The panel recommended against (1) clinicians routinely performing sinonasal imaging in patients presenting with symptoms consistent with a diagnosis of AR and (2) clinicians offering oral leukotriene receptor antagonists as primary therapy for patients with AR.
The panel group made the following options: (1) Clinicians may advise avoidance of known allergens or may advise environmental controls (ie, removal of pets; the use of air filtration systems, bed covers, and acaricides [chemical agents formulated to kill dust mites]) in patients with AR who have identified allergens that correlate with clinical symptoms. (2) Clinicians may offer intranasal antihistamines for patients with seasonal, perennial, or episodic AR. (3) Clinicians may offer combination pharmacologic therapy in patients with AR who have inadequate response to pharmacologic monotherapy. (4) Clinicians may offer, or refer to a surgeon who can offer, inferior turbinate reduction in patients with AR with nasal airway obstruction and enlarged inferior turbinates who have failed medical management.
(5) Clinicians may offer acupuncture, or refer to a clinician who can offer acupuncture, for patients with AR who are interested in nonpharmacologic therapy. The development group provided no recommendation regarding the use of herbal therapy for patients with AR.”
It is clear no ‘recommendation’ (‘strong’ or otherwise) was offered for the use of acupuncture, but rather an option was suggested that acupuncture ‘may be offered’ specifically “for patients with AR who are interested in nonpharmacologic therapy.” In other words, specifically for patients who want alternatives to conventional orthodox medicine.
Well, of course, any implausible treatment ‘may be offered’ – and might be that thereby that specific group of patients achieve some satisfaction. But such reasonable medical practice (to offer TLC and engage placebo effects) is a world away from suggesting to any patient that acupuncture may have an effect on their immune system/allergic rhinitis. There simply is no evidence of such effectst, and the AAOHNS Foundation does not say there is.
It is precisely because acupuncturists have a tendency to use dissembling hyperbole which must mislead patients, that evidence-based practitioners are riled.
Please Dr McDonald, check your data, answer Professor Ernst’s questions, avoid quoting medical authorities selectively and rewrite your paper so that we all may benefit.
Thank you.