Much of so-called alternative medicine (SCAM) is used in the management of osteoarthritis pain. Yet few of us ever seem to ask whether SCAMs are more or less effective and safe than conventional treatments.
This review determined how many patients with chronic osteoarthritis pain respond to various non-surgical treatments. Published systematic reviews of randomized controlled trials (RCTs) that included meta-analysis of responder outcomes for at least 1 of the following interventions were included: acetaminophen, oral nonsteroidal anti-inflammatory drugs (NSAIDs), topical NSAIDs, serotonin-norepinephrine reuptake inhibitors (SNRIs), tricyclic antidepressants, cannabinoids, counselling, exercise, platelet-rich plasma, viscosupplementation (intra-articular injections usually with hyaluronic acid ), glucosamine, chondroitin, intra-articular corticosteroids, rubefacients, or opioids.
In total, 235 systematic reviews were included. Owing to limited reporting of responder meta-analyses, a post hoc decision was made to evaluate individual RCTs with responder analysis within the included systematic reviews. New meta-analyses were performed where possible. A total of 155 RCTs were included. Interventions that led to more patients attaining meaningful pain relief compared with control included:
- exercise (risk ratio [RR] of 2.36; 95% CI 1.79 to 3.12),
- intra-articular corticosteroids (RR = 1.74; 95% CI 1.15 to 2.62),
- SNRIs (RR = 1.53; 95% CI 1.25 to 1.87),
- oral NSAIDs (RR = 1.44; 95% CI 1.36 to 1.52),
- glucosamine (RR = 1.33; 95% CI 1.02 to 1.74),
- topical NSAIDs (RR = 1.27; 95% CI 1.16 to 1.38),
- chondroitin (RR = 1.26; 95% CI 1.13 to 1.41),
- viscosupplementation (RR = 1.22; 95% CI 1.12 to 1.33),
- opioids (RR = 1.16; 95% CI 1.02 to 1.32).
Pre-planned subgroup analysis demonstrated no effect with glucosamine, chondroitin, or viscosupplementation in studies that were only publicly funded. When trials longer than 4 weeks were analysed, the benefits of opioids were not statistically significant.
The authors concluded that interventions that provide meaningful relief for chronic osteoarthritis pain might include exercise, intra-articular corticosteroids, SNRIs, oral and topical NSAIDs, glucosamine, chondroitin, viscosupplementation, and opioids. However, funding of studies and length of treatment are important considerations in interpreting these data.
Exercise clearly is an effective intervention for chronic osteoarthritis pain. It has consistently been recommended by international guideline groups as the first-line treatment in osteoarthritis management. The type of exercise is likely not important.
Pharmacotherapies such as NSAIDs and duloxetine demonstrate smaller but statistically significant benefit that continues beyond 12 weeks. Opioids appear to have short-term benefits that attenuate after 4 weeks, and intra-articular steroids after 12 weeks. Limited data (based on 2 RCTs) suggest that acetaminophen is not helpful. These findings are consistent with recent Osteoarthritis Research Society International guideline recommendations that no longer recommend acetaminophen for osteoarthritis pain management and strongly recommend against the use of opioids.
Limited benefit was observed with other interventions including glucosamine, chondroitin, and viscosupplementation. When only publicly funded trials were examined for these interventions, the results were no longer statistically significant.
Adverse events were inconsistently reported. However, withdrawal due to adverse events was consistently reported and found to be greater in patients using opioids, SNRIs, topical NSAIDs, and viscosupplementation.
Few of the interventions assessed fall under the umbrella of so-called alternative medicine (SCAM):
- some forms of exercise,
It is unclear why the authors did not include SCAMs such as chiropractic, osteopathy, massage therapy, acupuncture, herbal medicines, neural therapy, etc. in their review. All of these SCAMs are frequently used for osteoarthritis pain. If they had included these treatments, how do you think they would have fared?
My new book has just been published. Allow me to try and whet your appetite by showing you the book’s introduction:
“There is no alternative medicine. There is only scientifically proven, evidence-based medicine supported by solid data or unproven medicine, for which scientific evidence is lacking.” These words of Fontanarosa and Lundberg were published 22 years ago. Today, they are as relevant as ever, particularly to the type of healthcare I often call ‘so-called alternative medicine’ (SCAM), and they certainly are relevant to chiropractic.
Invented more than 120 years ago by the magnetic healer DD Palmer, chiropractic has had a colourful history. It has now grown into one of the most popular of all SCAMs. Its general acceptance might give the impression that chiropractic, the art of adjusting by hand all subluxations of the three hundred articulations of the human skeletal frame, is solidly based on evidence. It is therefore easy to forget that a plethora of fundamental questions about chiropractic remain unanswered.
I wrote this book because I feel that the amount of misinformation on chiropractic is scandalous and demands a critical evaluation of the evidence. The book deals with many questions that consumers often ask:
- How well-established is chiropractic?
- What treatments do chiropractors use?
- What conditions do they treat?
- What claims do they make?
- Are their assumptions reasonable?
- Are chiropractic spinal manipulations effective?
- Are these manipulations safe?
- Do chiropractors behave professionally and ethically?
Am I up to this task, and can you trust my assessments? These are justified questions; let me try to answer them by giving you a brief summary of my professional background.
I grew up in Germany where SCAM is hugely popular. I studied medicine and, as a young doctor, was enthusiastic about SCAM. After several years in basic research, I returned to clinical medicine, became professor of rehabilitation medicine first in Hanover, Germany, and then in Vienna, Austria. In 1993, I was appointed as Chair in Complementary Medicine at the University of Exeter. In this capacity, I built up a multidisciplinary team of scientists conducting research into all sorts of SCAM with one focus on chiropractic. I retired in 2012 and am now an emeritus professor. I have published many peer-reviewed articles on the subject, and I have no conflicts of interest. If my long career has taught me anything, it is this: in the best interest of consumers and patients, we must insist on sound evidence; not opinion, not wishful thinking; evidence.
In critically assessing the issues related to chiropractic, I am guided by the most reliable and up-to-date scientific evidence. The conclusions I reach often suggest that chiropractic is not what it is often cracked up to be. Hundreds of books have been published that disagree. If you are in doubt who to trust, the promoter or the critic of chiropractic, I suggest you ask yourself a simple question: who is more likely to provide impartial information, the chiropractor who makes a living by his trade, or the academic who has researched the subject for the last 30 years?
This book offers an easy to understand, concise and dependable evaluation of chiropractic. It enables you to make up your own mind. I want you to take therapeutic decisions that are reasonable and based on solid evidence. My book should empower you to do just that.
He is biased against it!
He cherry-picks the evidence!
He does not understand homeopathy!
If you are one of the many who believe such notions, please read on.
The website of the NHS England has a fairly detailed account of homeopathy. Here is the section entitled ‘What can we conclude from the evidence?‘ – but I recommend reading the full text:
There have been several reviews of the scientific evidence on the effectiveness of homeopathy.
The House of Commons Science and Technology Committee said there’s no evidence that homeopathy is effective as a treatment for any health condition.
There’s no evidence behind the idea that substances that cause certain symptoms can also help treat them.
Nor is there any evidence behind the idea that diluting and shaking substances in water can turn those substances into medicines.
The ideas that underpin homeopathy aren’t accepted by mainstream science, and aren’t consistent with long-accepted principles on the way the physical world works.
The Committee’s 2010 report on homeopathy said the “like cures like” principle is “theoretically weak”, and that this is the “settled view of medical science”.
For example, many homeopathic remedies are diluted to such an extent that it’s unlikely there’s a single molecule of the original substance remaining in the final remedy. In cases like these, homeopathic remedies consist of nothing but water.
Some homeopaths believe that, as a result of the succussion process, the original substance leaves an “imprint” of itself on the water. But there’s no known mechanism by which this can occur.
The 2010 report said: “We consider the notion that ultra-dilutions can maintain an imprint of substances previously dissolved in them to be scientifically implausible.”
Some people who use homeopathy may see an improvement in their health condition as the result of a phenomenon known as the placebo effect.
If you choose health treatments that provide only a placebo effect, you may miss out on other treatments that have been proven to be more effective.
Since 1948, homeopathy had been part of the NHS, there were 5 homeopathic NHS hospitals, and the costs for homeopathy were covered. Why would the NHS decision makers suddenly turn against it? They must have loved homeopathy for at least 4 reasons:
- It is inexpensive.
- It has support in high places.
- It did not cause any direct harm.
- It had many supporters who fought tooth and nail for it.
It is therefore hardly reasonable to assume that the NHS is biased against homeopathy. But, why do they now say that it is
- not effective beyond placebo,
- and can cause harm by making people miss out on effective therapies?
The answer is simple: BECAUSE THESE STATEMENTS ARE IN ACCORDANCE WITH THE OVERWHELMING MAJORITY OF THE BEST EVIDENCE AVAILABLE TO DATE.
So, here you are: the NHS now confirms what I (and many other experts) have been saying since years. And we all insist on the fact that this not because we are biased, stupid, uninformed, paid by BIG PHARMA, or want to deprive anyone of anything. We do it for one reason only:
BECAUSE IT’S THE TRUTH!
I have long cautioned that chiropractic overuse of X-rays is a safety problem. Is this still an issue? A recent paper was aimed at finding out.
The objective of this review was to determine the diagnostic and therapeutic utility of routine or repeat radiographs (in the absence of red flags) of the cervical, thoracic or lumbar spine for the functional or structural evaluation of the spine. Investigate whether functional or structural findings on repeat radiographs are valid markers of clinically meaningful outcomes. The research objectives required that the researchers determine the validity, diagnostic accuracy and reliability of radiographs for the structural and functional evaluation of the spine.
The investigators searched MEDLINE, CINAHL, and Index to Chiropractic Literature from inception to November 25, 2019. They used rapid review methodology recommended by the World Health Organization. Eligible studies (cross-sectional, case-control, cohort, randomized controlled trials, diagnostic and reliability) were critically appraised. Studies of acceptable quality were included in our synthesis.
Twenty-three papers were critically appraised. No relevant studies assessed the clinical utility of routine or repeat radiographs (in the absence of red flags) of the cervical, thoracic or lumbar spine for the functional or structural evaluation of the spine. No studies investigated whether functional or structural findings on repeat radiographs are valid markers of clinically meaningful outcomes. Nine low risk of bias studies investigated the validity (n = 2) and reliability (n = 8) of routine or repeat radiographs. These studies provided no evidence of clinical utility.
The authors’ conclusions are clear: We found no evidence that the use of routine or repeat radiographs to assess the function or structure of the spine, in the absence of red flags, improves clinical outcomes and benefits patients. Given the inherent risks of ionizing radiation, we recommend that chiropractors do not use radiographs for the routine and repeat evaluation of the structure and function of the spine.
In the paper, the authors provided further valuable information and background:
In the United States in 2010, the rate of spine radiographs within 5 days of presenting to a chiropractor was 204 per 1000 new patients. An analysis of national trends in the United States suggests that the rate of spinal radiography by chiropractors and podiatrists increased by 14.4% between 2003 and 2015. This increase occurred despite the publication of several evidence-based clinical practice guidelines and clinical prediction rules to assist chiropractors in determining the indication for spine radiographs to assist with diagnosing a pathology. Overall, guidelines suggest that radiographs are indicated when signs and symptoms of potentially serious underlying pathology (red flags) are identified through the clinical history and physical examination. However, on its own, an isolated “red flag” may have a high false positive rate for the diagnosis of underlying spinal pathology, such as cancer. For example, the presence of a solitary “red flag” such as age over 50 years may not be sufficient to warrant taking spine radiographs. Therefore, clinicians are encouraged to combine sound clinical judgement and the assessment of red flags when ordering radiographs.
In the absence of “red flags”, the use of spinal radiographs is not recommended. Nevertheless, factions of chiropractors, including the International Chiropractic Association promote the use of routine or repeat radiographs to assess the structure and function of the spine. This practice which dates back to 1910 was initiated when no evidence was available to guide the judicious use of spine radiographs. Historically, these groups of chiropractors have argued that radiographs are helpful to measure postural abnormalities, identify vertebral misalignment or subluxation and guide treatment with spinal manipulative therapy. The belief that radiographs are useful to detect and correct spine structure and function provides the foundation for many chiropractic technique systems that are still in use today. To our knowledge, approximately 23 chiropractic techniques use spine radiography (including full spine radiography) to guide the clinical management of patients. These include the Gonstead, Chiropractic BioPhysics®, Toggle-Recoil, and National Upper Cervical Chiropractic Association (NUCCA) techniques. Proponents of these techniques claim that the use of routine and repeat radiographs is supported by scientific evidence and have published a guideline to assist clinicians with the biomechanical assessment of spinal subluxation in chiropractic clinical practice using radiography. However, these claims have not yet been evaluated for their clinical utility, the benefit a patient gains from a test or treatment. This was a particular concern for the College of Chiropractors of British Columbia (CCBC) which regulates the practice of chiropractic in the province of British Columbia, Canada. The mission of the CCBC is to protect the public by regulating British Columbia’s doctors of chiropractic to ensure safe, qualified and ethical delivery of care.
The references from these two paragraphs can be found in the original paper. One reference the authors did not include was my article of 1998 which, at the time, received plenty of angry responses from chiropractors. Here is its conclusion: DATA SUGGEST AN OVERUSE OF RADIOGRAPHY BY THE CHIROPRACTIC PROFESSION. THIS CONSTITUTES A SAFETY PROBLEM THAT DESERVES TO BE TAKEN SERIOUSLY AND REQUIRES FURTHER RESEARCH.
Twenty-two years later, do I get the impression that the chiropractic profession might not be the fastest in getting its act together?
I have discovered ‘Google Scholar’!
Yes, of course, I knew about it, but I never used it much. In particular, I did not know it has a huge page just on me. So I had a good look at it (who would be able to resist?) and found many things of interest – for instance, the fact that (as of yesterday) my papers have been cited a total of 86 759 times, and that 4 of them have been cited more that a thousand times.
Here they are:
|Interactions between herbal medicines and prescribed drugs AA Izzo, E Ernst
Drugs 61 (15), 2163-2175
|Fibrinogen as a cardiovascular risk factor: a meta-analysis and review of the literature
E Ernst, KL Resch
Annals of internal medicine 118 (12), 956-963
|Influence of context effects on health outcomes: a systematic review
Z Di Blasi, E Harkness, E Ernst, A Georgiou, J Kleijnen
The Lancet 357 (9258), 757-762
|The prevalence of complementary/alternative medicine in cancer: a systematic review
Cancer: Interdisciplinary International Journal of the American Cancer …
Two things are perhaps noteworthy here, I feel:
- Only 2 of the 4 papers are on research in so-called alternative medicine (SCAM).
- In the 4th paper, they forgot to add Barrie Cassileth who was its co-author.
Scanning my own articles, the real revelation was how much I owe to others, how many co-workers I have had, how many of them I had completely forgotten about, and how many have already gone forever.
So, allow me to take this opportunity to honour those who have passed away (in the order they appear on the page).
- ARPAD MATRAI was a brilliant scientist, Olympic swimmer for Hungary, and close friend. He came to London in 1980 to work in my lab. After I had left, I attracted him to Munich where we had several hugely productive years together – until he died of leukaemia in 1988.
- JOHN DORMANDY see here.
- VERONIKA FIALKA was my senior registrar in Vienna and became a good friend. After I had left Vienna, she took over my position as head of the department. We then somehow lost contact and, one day, I received the sad news of her early death.
- NASSIM KANJI was my PhD student at Exeter. She did very well, and we published several papers on autogenic training together.
- PETER FISHER see here.
- GEORGE LEWITH see here.
- CHRIS SILAGY was a brilliant GP and researcher. We did not have much contact except for one paper we had together.
- JOHN GARROW see here.
- ANDREW HERXHEIMER see here.
- WALLACE SAMPSON was a famous and brilliant US sceptic. We had various contacts and shared one paper.
- P T FLUTE was head of haematology at St George’s Hospital, London while I worked there. I remember him as kind and supportive.
I owe more gratitude to these (and all my other) co-authors than I will ever be able to express.
In response to the recommendation, the health ministry announced the formation of a working group of medical experts to suggest proposals for further regulation of homeopathy. A spokesman said that medicines whose efficiency is not clinically proven should not be procured using public funds, nor prescribed to treat the sick.
Russia has proved a profitable market for foreign suppliers of homeopathic medicine such as French company Boiron, which opened its Russian subsidiary in 2005. “Today, the Russian market is our company’s fourth largest in terms of turnover, after France, the US and Italy. Russia has always been interesting for Boiron because of the large population, and a relatively high incidence of illness and lower level of medicine consumption in comparison with Europe,” general director of Boiron in Russia Irina Nikulina said.
According to figures from Russian pharmaceutical market analysts DSM Group, Boiron sold 35 percent of all the homeopathic medicine sold in Russia last year, or 2.88 billion rubles (USD 49.5 million) worth of medicine. Boiron produces Russia’s most popular homeopathic medicine, called Oscillococcinum, which is marketed to relieve flu symptoms and accounted for 18.98 percent of all homeopathic medicines sold in 2016.
The many international initiatives aimed at minimising the harm done by homeopathy are slowly beginning to yield results. It took many years for politicians to realise that the supposedly harmless homeopathy is, in fact, not harmless at all. Homeopathy causes harm by:
- wasting people’s money,
- distracting patients from effective treatments,
- the ill-conceived advice homeopaths give to patients,
- making a mockery of evidence-based medicine,
- violating the principles of medical ethics,
- undermining rational thinking in society.
One therefore has to applaud Russia’s ‘Commission against Pseudoscience’, hope that the working group does produce robust advice, and support similar initiatives in other countries.
In 2015, the Hungarian Academy of Sciences stated that “Homeopathic remedies don’t meet the criteria of evidence based medicine” and that homeopathic products should follow the same strict scientific standards as conventional drugs. In 2017, the Scientific Advisory Board of European Academies (EASAC) concluded that there is no substantial evidence that homeopathy works and may even be harmful to our health.
Now, Hungary is about to act. New regulation is tightening the marketing of homeopathic products in Hungary. From Wednesday this week, homeopathic remedies can only be distributed in Hungary without a therapeutic indication or claim. The reason for this move is that none of the products’ efficacy have been adequately confirmed by rigorous clinical trials.
In a statement, the Hungarian National Institute of Pharmacy and Nutrition (OGYÉI) said the changes are due to a law amendment that came into effect last year. The new regulation only allows homeopathic medicines with therapeutic indications authorized before Hungary’s accession to the EU (2004), to be marketed after July 1, 2020, if they have complied with the EU regulations on the marketing of these medicines.
Currently, Hungary has no homeopathic product with therapeutic efficacy proven in clinical trials. The product license of homeopathic products – in compliance with the legislation of the European Union – can be obtained by two different procedures in Hungary. The so-called simplified procedure can be used for “high-dilution products” marketed without a therapeutic indication, in which case the effectiveness of the product does not need to be certified.
The “normal” procedure is applicable to homeopathic medicinal products marketed for a therapeutic indication, in which case, just as any other medicinal products, therapeutic efficacy must be clinically proven. OGYÉI emphasized that from July 1st, the advertising of marketable homeopathic remedies may only contain the label text of the product, no additional information.
The move by the Hungarian authorities is, of course, most welcome. It brings Hungary finally in line with the rest of the EU. The many enthusiasts of homeopathy will no doubt suspect a worldwide conspiracy against homeopathy. If so, they merely disclose how far they have put their heads into the sand. Such measures are nothing but the long overdue actions towards abolishing double standards that have existed far too long and have helped nobody except the homeopathic industry.
Michael Dixon LVO, OBE, MA, FRCGP has been a regular feature of this blog (and elsewhere). He used to be a friend and colleague until … well, that’s a long story. Recently, I came across his (rather impressive) Wikipedia page. To my surprise, it mentions that Dixon
“has been criticised by professor of complementary medicine and alternative medicine campaigner Edzard Ernst for advocating the use of complementary medicine. Ernst said that the stance of the NHS Alliance on complementary medicine was “misleading to the degree of being irresponsible.” Ernst had previously been sympathetic to building a bridge between complementary and mainstream medicine, co-writing an article with Michael Dixon in 1997 on the benefits of such an approach. Ernst and Dixon write “missed diagnoses by complementary therapists giving patients long term treatments are often cited but in the experience of one of the authors (MD) are extremely rare. It can also cut both ways. A patient was recently referred back to her general practitioner by an osteopath, who was questioning, as it turned out quite correctly, whether her pain was caused by metastates. Good communication between general practitioner and complementary therapist can reduce conflicts and contradictions, which otherwise have the potential to put orthodox medicine and complementary therapy in an either/or situation.”
31) February 2009, 24. “Academics and NHS Alliance clash over complementary medicine”. Pulse Today.
32) ^ Update – the journal of continuing education for General Practitioners, 7th May 1997
I have little recollection of the paper that I seem to have published with my then friend Michael, and it is not listed in Medline, nor can I find it in my (usually well-kept) files; the journal ‘Update’ does not exist anymore and was obviously not a journal good enough for keeping a copy. But I do not doubt that Wiki is correct.
In fact, it is true that, in 1997, I was still hopeful that bridges could be built between conventional medicine and so-called alternative medicine (SCAM). But I had always insisted that they must be bridges built on solid ground and with robust materials.
Put simply, my strategy was to test SCAM as rigorously as I could and to review the totality of the evidence for and against it. Subsequently, one could consider introducing those SCAMs into routine care that had passed the tests of science.
Dixon’s strategy differed significantly from mine. He had no real interest in science and wanted to use SCAM regardless of the evidence. Since the publication of our paper in 1997, he has pursued this aim tirelessly. On this blog, we find several examples of his activity.
And what happened to the bridges?
I’m glad you ask!
As it turns out, very few SCAMs have so far passed the test of science and hardly any SCAM has been demonstrated to generate more good than harm. The material to build bridges is therefore quite scarce, hardly enough for solid constructions. Dixon does still not seem to be worried about this indisputable fact. He thinks that INTEGRATED MEDICINE is sound enough for providing a way to the future. I disagree and still think it is ‘misleading to the degree of being irresponsible’.
Who is right?
Dixon or Ernst?
Opinions about this differ hugely.
Time will tell, I suppose.
The objective of this survey was to assess the prevalence and types of so-called alternative medicine (SCAM) usage as well as the participants’ spirituality/religiousness in an outpatient department for endocrinology and metabolic diseases. All individuals visiting the outpatient department at a German university hospital from April to June 2009 were offered a standardized questionnaire on the use of dietary supplements and other SCAMs as well as their religiousness/spirituality. Demographic and clinical data of 428 respondents were taken from the electronic health record.
Of the respondents, 16.4% (n = 66) classified themselves to be religious/spiritual and 67.9% (n = 273) as not religious/spiritual. The results show that:
- 41.4% of the respondents used supplements and 27.4% additional therapies;
- the use of supplements and other SCAMs was more frequent in people with higher religiousness/spirituality (p = 0.005 and p = 0.01,resp.);
- there were no associations between religiousness/spirituality and the number of consultations, costs for drugs, appraisal of the physicians treatment methods, the perceived effectiveness of prescribed drugs, fear of late complications or of side effects.
The authors concluded that a higher religiousness/spirituality is associated with a more frequent use of supplements or additional therapies in individuals with endocrinopathies or metabolic diseases. As SCAM has been shown to be associated with worse outcome, addressing religiousness/spirituality which stresses the responsibility of the person for his life might offer an additional resource and should be further studied.
This survey has a dismal sample size and even worse response rate and must therefore be taken with more than a pinch of salt. Yet vaguely similar associations have been shown before. For instance, analysing data from the 1995-1996 National Survey of Midlife Development in the United States (n=3032), researchers examined the correlations between four aspects of spirituality/religiousness-i.e., spiritual only, religious only, both spiritual and religious, and neither spiritual nor religious-and six measures of SCAM. Compared with spiritual only persons, the odds of using energy therapies were 86% lower for spiritual and religious persons, 65% lower for religious only persons, and 52% lower for neither spiritual nor religious persons. Compared to spiritual only persons, spiritual and religious individuals were 43% more likely to use body-mind therapies in general; however, when this category did not contain prayer, meditation, or spiritual healing, they were 44% less likely. Religious only individuals were disinclined toward SCAM use.
There might be considerable cultural and national differences, of course, but if it is true that religiousness/spirituality is associated with a more frequent use of SCAM, we ought to ask what the nature of the link between the two might be. There are, as far as I can see, three possibilities:
- religiousness/spirituality causes SCAM use;
- SCAM use causes religiousness/spirituality;
- the two are related via one or several other factors.
I see no reason why 1 or 2 should be true. More likely there is a common denominator. The obvious one might be that both religiousness/spirituality and SCAM use are somewhat irrational, more a matter of belief than evidence, and revealing a lack of scepticism or critical thinking. In this case, religiousness/spirituality and SCAM use would simply be two different expressions of the same frame of mind.
What do you think?
In the wake of both the NEJM and the LANCET withdrawing two potentially influential papers due unanswered questions about the source and reliability of the data, one has to ask how good or bad the process of peer review is.
Peer review is the evaluation of work by one or more people with similar competences as the producers of the work (peers). It functions as a form of self-regulation by qualified members of a profession within the relevant field. It normally involves multiple steps:
- Authors send their manuscript to a journal of their choice for publication.
- The journal editor has a look at it and decides whether to reject it straight away (for instance, because the subject area is not of interest) or whether to send it out to referees for examination (often to experts suggested by the authors of the submission).
- The referees (usually 2 or 3) have the opportunity to reject or accept the invitation to review the submission.
- If they accept, they review the paper and send their report to the editor (usually following a deadline).
- The editor tries to come to a decision about publication; often the referees are not in agreement, and a further referee has to be recruited.
- Even if the submission is potentially publishable, the referees will have raised several points that need addressing. In such cases, the editor sends the submission back to the original authors asking them to revise the article.
- The authors do their revision (often following a deadline) and re-submit their paper.
- Now the editor can decide to either publish it or send it back to the referees asking them whether they feel their criticisms have been adequately addressed.
- Depending on the referees’ verdicts, the editor makes the final decision and informs all the involved parties accordingly.
- If the paper was accepted, it then goes into production.
- When this process is finished, the authors receive the proofs for final a check.
- Eventually, the paper is published and the readers of the journal may scrutinise it.
- Often this prompts comment which may get published.
- In this case, the authors of the original paper may get invited to write a reply.
- Finally the comments and the reply are published in the journal side by side.
The whole process takes time, sometimes lots of time. I have had papers that took almost two years from submissions to publications. This delay seems tedious and, if the paper is important, unacceptable (if it is not important, it should arguably not be published at all). Equally unacceptable is the fact that referees are expected to do their reviewing for free. The consequence is that many referees do their reviewing less than well.
When I was still at Exeter, I had plenty of opportunity to see the problems of peer review from the reviewers perspective. At a time, I accepted about 5 reviews per week, and in total I surely have reviewed over 1000 papers. I often recommended inviting a statistician to do a specialist review of the stats. Only rarely were such suggestions accepted by the journal editors. Very often I recommended rejecting a submission because it was rubbish, and occasionally, I told the editor that there was a strong suspicion of the paper being fraudulent. The editors very often (I estimate in about 50% of cases) ignored my suggestions and comments and published the papers nonetheless. If the editor did follow my advice to reject a paper, I regularly saw it published elsewhere later (usually in a less well-respected journal). Several times, an author of a submission contacted me directly after seeing my criticism of his paper. Occasionally this resulted in unpleasantness, once or twice even in threats. Eventually I realised that improving the publications in the realm of SCAM was a Sisyphean task, became quite disenchanted with all this and accepted less and less reviews. Today, I do only very few.
I had even more opportunity to see the peer review process from the author’s perspective. All authors must have suffered from unfair or incompetent reviews and most will have experienced the frustrations of the endless delays. Once (before my time in alternative medicine) a reviewer rejected my paper and soon after published results that were uncannily similar to mine. In alternative medicine, researchers tend to be rather emotional about their subject. Imagine, for instance, the review you might get from Dana Ullmann of a trial of homeopathy that fails to show what he believes in.
Finally, since 40 years, I have also had the displeasure of experiencing peer review as an editor. This often seemed like trying to sail between the devil and the deep blue sea. Editors want to fill their journals with the best science they can find. But all too often, they receive the worst science they can imagine. They are constantly torn by tensions pulling them in opposite directions. And they have to cope not just with poor quality submissions but also with reviewers who miss deadlines and do their work badly.
So, peer review is fraught with problems! The trouble is that there are few solutions that would keep a better check on the reliability of science. Peer review, it often seemed to me, is the worst idea, except for all others. If peer review is to survive (and I think it probably will), there are a few things that could, from my point of view, be done to improve it:
- Make it much more attractive for the referees. Payment would be the obvious thing – and by Jove, the big journals like the LANCET and NEJM could afford it. But recognising refereeing academically would be even more important. At present, academic careers depend largely of publications; if they also depended on reviewing, experts would queue up to do it.
- The reports of the referees should get independently evaluated according to sensible criteria. These data could be conflated an published as a criterion of academic standing. Referees who fail to to a good job would spoil their chances to get re-invited for this task.
- Speed up the entire process. Waiting months on months is hugely counter-productive for all concerned.
- Today many journals ask authors for the details of experts who are potential reviewers of their submission and then send the paper in question to them for review. I find this ridiculous! No author I know of has ever resisted the temptation to name people who are friends or owe a favour. Journals should afford the extra work to find who the best independent experts on any particular subject are.
None of this is simple or fool-proof or even sure to work well, of course. But surely it is worth trying to get peer-review right. The quality of future science depends on it.