MD, PhD, FMedSci, FSB, FRCP, FRCPEd

One of the questions I hear frequently is ‘HOW CAN I BE SURE THIS STUDY IS SOUND’? Even though I have spent much of my professional life on this issue, I am invariably struggling to provide an answer. Firstly, because a comprehensive reply must inevitably have the size of a book, perhaps even several books. And secondly, to most lay people, the reply would be intensely boring, I am afraid.

Yet many readers of this blog evidently search for some guidance – so, let me try to provide a few indicators – indicators, not more!!! – as to what might signify a good and a poor clinical trial (other types of research would need different criteria).

INDICATORS SUGGESTIVE OF A GOOD CLINICAL TRIAL

  • Author from a respected institution.
  • Article published in a respected journal.
  • A clear research question.
  • Full description of the methods used such that an independent researcher could repeat the study.
  • Randomisation of study participants into experimental and control groups.
  • Use of a placebo in the control group where possible.
  • Blinding of patients.
  • Blinding of investigators, including clinicians administering the treatments.
  • Clear definition of a primary outcome measure.
  • Sufficiently large sample size demonstrated with a power calculation.
  • Adequate statistical analyses.
  • Clear presentation of the data such that an independent assessor can check them.
  • Understandable write-up of the entire study.
  • A discussion that puts the study into the context of all the important previous work in this area.
  • Self-critical analysis of the study design, conduct and interpretation of the results.
  • Cautious conclusion which are strictly based on the data presented.
  • Full disclosure of ethics approval and informed consent,
  • Full disclosure of funding sources.
  • Full disclosure of conflicts of interest.
  • List of references is up-to-date and includes also studies that contradict the authors’ findings.

I told you this would be boring! Not only that, but each bullet point is far too short to make real sense, and any full explanation would be even more boring to a lay person, I am sure.

What might be a little more fun is to list features of a clinical trial that might signify a poor study. So, let’s try that.

WARNIG SIGNALS INDICATING A POOR CLINICAL TRIAL

  • published in one of the many dodgy CAM journals (or in a book, blog or similar),
  • single author,
  • authors are known to be proponents of the treatment tested,
  • author has previously published only positive studies of the therapy in question (or member of my ‘ALT MED HALL OF FAME’),
  • lack of plausible rationale for the study,
  • lack of plausible rationale for the therapy that is being tested,
  • stated aim of the study is ‘to demonstrate the effectiveness of…’ (clinical trials are for testing, not demonstrating effectiveness or efficacy),
  • stated aim ‘to establish the effectiveness AND SAFETY of…’ (even large trials are usually far too small for establishing the safety of an intervention),
  • text full of mistakes, e. g. spelling, grammar, etc.
  • sample size is tiny,
  • pilot study reporting anything other than the feasibility of a definitive trial,
  • methods not described in sufficient detail,
  • mismatch between aim, method, and conclusions of the study,
  • results presented only as a graph (rather than figures which others can re-calculate),
  • statistical approach inadequate or not sufficiently detailed,
  • discussion without critical input,
  • lack of disclosures of ethics, funding or conflicts of interest,
  • conclusions which are not based on the results.

The problem here (as above) is that one would need to write at least an entire chapter on each point to render it comprehensible. Without further detailed explanations, the issues raised remain rather abstract or nebulous. Another problem is that both of the above lists are, of course, far from complete; they are merely an expression of my own experience in assessing clinical trials.

Despite these caveats, I hope that those readers who are not complete novices to the critical evaluation of clinical trials might be able to use my ‘warning signals’ as a form of check list that helps them to tell the chaff from the wheat.

21 Responses to How to differentiate good from bad research

Leave a Reply

Your email address will not be published. Required fields are marked *

Please answer the following: *

Recent Comments

Note that comments can now be edited for up to five minutes after they are first submitted.


Click here for a comprehensive list of recent comments.

Categories