MD, PhD, MAE, FMedSci, FRSB, FRCP, FRCPEd.

This post has an odd title and addresses an odd subject. I am sure some people reading it will ask themselves “has he finally gone potty; is he a bit xenophobic, chauvinistic, or what?” I can assure you none of the above is the case.

Since many years, I have been asked to peer-review Chinese systematic reviews and meta-analyses of TCM-trials submitted to various journals and to the Cochrane Collaboration for publication, and I estimate that around 300 such articles are available today. Initially, I thought they were a valuable contribution to our knowledge, particularly for the many of us who cannot read Chinese languages. I hoped they might provide reliable information about this huge and potentially important section of the TCM-evidence. After doing this type of work for some time, I became more and more frustrated; now I have decided not to accept this task any longer – not because it is too much trouble, but because I have come to the conclusion that these articles are far less helpful than I had once assumed; in fact, I now fear that they are counter-productive.

In order to better understand what I mean, it might be best to use an example; this recent systematic review seems as good for that purpose as any.

Its Chinese authors “hypothesized that the eligible trials would provide evidence of the effect of Chinese herbs on bone mineral density (BMD) and the therapeutic benefits of Chinese medicine treatment in patients with bone loss. Randomized controlled trials (RCTs) were thus retrieved for a systematic review from Medline and 8 Chinese databases. The authors identified 12 RCTs involving a total of 1816 patients. The studies compared Chinese herbs with placebo or standard anti-osteoporotic therapy. The pooled data from these RCTs showed that the change of BMD in the spine was more pronounced with Chinese herbs compared to the effects noted with placebo. Also, in the femoral neck, Chinese herbs generated significantly higher increments of BMD compared to placebo. Compared to conventional anti-osteoporotic drugs, Chinese herbs generated greater BMD changes.

In their abstract, the part on the paper that most readers access, the authors reached the following conclusions: “Our results demonstrated that Chinese herb significantly increased lumbar spine BMD as compared to the placebo or other standard anti-osteoporotic drugs.” In the article itself, we find this more detailed conclusion: “We conclude that Chinese herbs substantially increased BMD of the lumbar spine compared to placebo or anti-osteoporotic drugs as indicated in the current clinical reports on osteoporosis treatment. Long term of Chinese herbs over 12 months of treatment duration may increase BMD in the hip more effectively. However, further studies are needed to corroborate the positive effect of increasing the duration of Chinese herbs on outcome as the results in this analysis are based on indirect comparisons. To date there are no studies available that compare Chinese herbs, Chinese herbs plus anti-osteoporotic drugs, and anti-osteoporotic drug versus placebo in a factorial design. Consequently, we are unable to draw any conclusions on the possible superiority of Chinese herbs plus anti-osteoporotic drug versus anti-osteoporotic drug or Chinese herb alone in the context of BMD.

Most readers will feel that this evidence is quite impressive and amazingly solid; they might therefore advocate routinely using Chinese herbs for the common and difficult to treat problem of osteoporosis. The integration of TCM might avoid lots of human suffering, prolong the life of many elderly patients, and save us all a lot of money. Why then am I not at all convinced?

The first thing to notice is the fact that we do not really know which of the ~7000 different Chinese herbs should be used. The article tells us surprisingly little about this crucial point. And even, if we manage to study this question in more depth, we are bound to get thoroughly confused; there are simply too many herbal mixtures and patent medicines to easily identify the most promising candidates.

The second and more important hurdle to making sense of these data is the fact that most of the primary studies originate from inaccessible Chinese journals and were published in Chinese languages which, of course, few people in the West can understand. This is entirely our fault, some might argue, but it does mean that we have to believe the authors, take their words at face value, and cannot check the original data. You may think this is fine, after all, the paper has gone through a rigorous peer-review process where it has been thoroughly checked by several top experts in the field. This, however, is a fallacy; like you and me, the peer-reviewers might not read Chinese either! (I don’t, and I reviewed quite a few of these papers; in some instances, I even asked for translations of the originals to do the job properly but this request was understandably turned down) In all likelihood, the above paper and most similar articles have not been properly peer-reviewed at all.

The third and perhaps most crucial point can only be fully appreciated, if we were able to access and understand the primary studies; it relates to the quality of the original RCTs summarised in such systematic reviews. The abstract of the present paper tells us nothing at all about this issue. In the paper, however, we do find a formal assessment of the studies’ risk of bias which shows that the quality of the included RCTs was poor to very poor. We also find a short but revealing sentence: “The reports of all trials mentioned randomization, but only seven described the method of randomization.” This remark is much more significant than it may seem: we have shown that such studies use such terminology in a rather adventurous way; reviewing about 2000 of these allegedly randomised trials, we found that many Chinese authors call a trial “randomised” even in the absence of a control group (one cannot randomise patients and have no control group)! They seem to like the term because it is fashionable and makes publication of their work easier. We thus have good reason to fear that some/many/most of the studies were not RCTs at all.

The fourth issue that needs mentioning is the fact that very close to 100% of all Chinese TCM-trials report positive findings. This means that either TCM is effective for every indication it is tested for (most unlikely, not least because there are many negative non-Chinese trials of TCM), or there is something very fundamentally wrong with Chinese research into TCM. Over the years, I have had several Chinese co-workers in my team and was invariably impressed by their ability to work hard and efficiently; we often discussed the possible reasons for the extraordinary phenomenon of 0% negative Chinese trials. The most plausible answer they offered was this: it would be most impolite for a Chinese researcher to produce findings which contradict the opinion of his/her peers.

In view of these concerns, can we trust the conclusions of such systematic reviews? I don’t think so – and this is why I have problems with research of this nature. If there are good reasons to doubt their conclusions, these reviews might misinform us systematically, they might not further but hinder progress, and they might send us up the garden path. This could well be in the commercial interest of the Chinese multi-billion dollar TCM-industry, but it would certainly not be in the interest of patients and good health care.

19 Responses to Do Chinese reviews summarising Chinese clinical trials of TCM send us up the garden path?

  • It is a bit bizarre that the herbs themselves wouldn’t be identified in the meta-analysis. “Chinese herbs” isn’t very useful to anyone. The references do list the Chinese formulas at least (Yigu, Fufang, etc.) but those will be equally obscure unless you are familiar with TCM. I guess the formula itself rather than individual herbs should be considered the “active drug” anyway but it would be helpful to know what the formula is. The 0% negative findings in Chinese TCM clinical trials is more worrying when it comes to whether or not you can trust them to be a fair assessment. There’s obviously a lot of layers requiring more transparency before they can be trusted to be providing accurate or useful information.

  • Actually – are you sure the 0% negative findings in TCM is correct?
    A 2 min search brought me this for example: http://www.cmjournal.org/content/7/1/5
    Which is a politely phrased negative review result; I’m sure there are more. Maybe research into TCM is improving in this respect?

  • in the sentence before i wrote: “very close to 100% of all Chinese TCM-trials report positive fingings.”

  • You’re raising an interesting topic.

    There’s obviously a strong argument considering chinese research when doing medical decisions. After all, we should remember that China is the most populated country in the world and there’s probably a lot of research. It sounds kind of stupid not to make use of that information.
    On the other hand it’d be also stupid to use medical information that’s most likely just a large collection of publication bias and low quality studies.
    Sounds like a difficult dilemma.

    One thing that probably should happen: We need more researchers learning Mandarin. Or at least we need access to good translators for researchers. After all, Mandarin is in many aspects THE most important language in the world. I was surprised to learn that this is not only the case if you count native speakers. If you count second- and third-language knowledge, English is still far behind Mandarin.
    I know that this has some practicality issues – learning a language is hard. I’m trying it myself (but I’m not doing medical research).
    And obviously this doesn’t help against a large pile of biased research. But it would be a start.

    • yes, being multi-lingual helps. i speak 3 languages but none of the Chinese ones. my solution for many years was to have 1 or 2 Chinese co-workers in my team. as you say, it does not avoid the bias in the original papers but it sometimes help identifying it.

  • There is a problem with lauding peer review in journals. When the journal is of the crank variety the peers tend to be cranks. It also seemed that the “integrative” portions of formerly reputable seem to use a different set of peers when evaluating alternative papers. Finding valid sources of information is becoming increasingly difficult. The term peer review is gradually losing its meaning.

    • i agree, but the problem is not confined to minor or cranl journals. even many Cochrane reviews suffer from the difficulties i describe in the post.

  • I do systematic reviews of TCM treatments. I alway be doubful about the results reported in the Chinese papers, because they lack a system/authority to check whether their data and research procedure are properly done. Thus, it is not know that whether their research is honestly reported.

    I have found 2 papers with identical results (same mean and SD in age, duration of illness, F:M ratio, and result %), but they were published by different groups of people, and the subjects were sampled from 2 different places (then how come the charaterisitcs would be the same?? it is likely that one of them is plagiarised). I have encountered such situation no less than 10 times.

  • I am only just starting out after finishing my degree at uni and I am currently doing my honours which looks at traditional Chinese medicine. The amount of articles I have read which either sound wonderful in the abstract and then have massive failings in the method/result is amazing. I agree with you when you state that the idea of peer-review Chinese systematic reviews are sending the wrong message, I have read so many that are just a bunch of waffle and that have no really scientific meaning at all. Not to even mention the bias within such publications. I know I still have a very long way to go till I can even think to call myself a researcher/scientist but this blog was such a nice breath of fresh air in the middle of my studies, thanks for the post!

  • Oh so you mean that their clinical trial is not all true? Isn’t it scary if that’s the case? We are hoping for better cure with clinical trials and then we have false results, that’s sad.

Leave a Reply to Edzard Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe via email

Enter your email address to receive notifications of new blog posts by email.

Recent Comments

Note that comments can be edited for up to five minutes after they are first submitted but you must tick the box: “Save my name, email, and website in this browser for the next time I comment.”

The most recent comments from all posts can be seen here.

Archives
Categories