Recently, I had the pleasure to give a lecture about bias in research to medical students at my former medical school in Vienna. This led to interesting discussions with the audience. They prompted me to think more than usual about ‘the biased researcher’, a phenomenon that, in my opinion, seriously plagues the field of so-called alternative medicine (SCAM). The way I see it, we can differentiate 4 overlapping categories of researchers (male or female; for simplicity, I here use only the masculine form) investigating the effectiveness of various forms of SCAM:
THE TRUE SCIENTIST
The true scientist is adequaately trained in all aspects of his work. Therefore, he knows that he research consists of testing hypotheses. He does his job without emotional or ideological baggage. All he aims at doing is answering the research question at hand in the most rigorous fashion. He is not influenced by outside pressures, does not care about the direction of his results, and merely wants to conduct the best science possible that his particular situation allows. In other words, he does what he can to minimize all sources of bias.
THE SLOPPY RESEARCHER
The sloppy researcher is either less well trained or he is less focussed and somewhat careless. He tends to cut corners, and is thus prone to make mistakes. His miskakes can introduce bias in his research which is unintentional because he has no axe to grind. In other words, the sloppy researcher is not biased but might easily produce biased results. As the sloppyness is unintentional, the resulting bias can go in either direction; the sloppy researcher might therefore generate false-positive and false-negative findings at random.
THE BIASED RESEARCHER
The biased researcher does have an axe to grind. Typically, he has a strong positive opinion about the treatment he is testing. For him, the concept of falsifying his beloved hypothesis is an abomination – he might know that this is how science out to work, but he can simply not bring himself to doing it. His mission is to confirm his prior conviction that the therapy in question is effective. This conviction is so strong that he does not feel that he is doing anything wrong. Obviously, the biased researcher would introduce bias into his research at multible levels. His bias will then compell him to hide the flaws in his research as much as he can. Consequently, his published papers will not easily disclose his bias and will therefore have the power to mislead the public.
THE DISHONEST RESEARCHER
The dishonest researcher is out to cheat. He wants to generate resuts of a certain type, usually showing that the therapy in question is effective. He is usually motivated by money and/or ambition. He may be sufficiently well trained to be able to hide his dishonesty from detection. Like the biased researcher’s papers, his fraudulent publications will not disclose his fabrications and will therefore have the power to mislead the public.
You will, of course, realize that, in my attempt to create these 4 categories, I have exaggerated and created caricatures of the real-life situations. However, I feel the the distinction between the 4 categories might be helpful to understand medical research and its pitfalls. As I pointed out in the introduction, the categories overlap. In reality, most researchers are hybrids of two or more categories. For instance, nobody can entirely be free of bias and everybody makes mistakes occasionally.
The question arises as to which type of category might dominate SCAM. I am not aware of reliable research that would answer it. However, my experience tells me that, in SCAM, we have a regrettable void of true scientists combined with an abundance of biased researchers (see, for instance, the growing list of researchers in my ALTERNATIVE MEDICINE HALL OF FAME). What is worse, the latter category is bringing SCAM research more and more into disrepute which, in turn, demotivates true scientists to consider SCAM as a serious subject.
11 Responses to A few toughts and ideas about ‘the biased researcher’
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
This is indeed an interesting subject. I’d be tempted to say that the True Scientist does not exist, at least not as the caricature sketched here, fully detached from emotions and truly objective in their observation of reality.
Most likely, every researcher has biases and an agenda, i.e. is trying to prove or support something that they have a hunch about, or even already believe; or else they think they have stumbled upon something new and special, and want to prove that this is indeed the case, regardless of the actual outcome. In other words: all researchers at the very least care about what they do for one reason or another – and that reason is almost never purely ‘Doing Science to the Best of my Ability’.
I think that this is pretty much inevitable in order to maintain the motivation to do scientific work in the first place. This is also why we have all sorts of rules, checks and practices in place which are supposed to make science self-correcting. Unfortunately, this doesn’t always work, and there appear to be pretty large areas where flawed research is the norm rather than the exception – and not just in SCAM.
I think that we can mostly rule out the True Scientist – I know of only one SCAM researcher who comes close to this category, and his name is Edzard Ernst.
If I should guess, it’s mostly the second and third categories, so the sloppy (as well as incompetent) and severely biased researchers – and quite often a combination of the two.
E.g. I’ve seen lots of ‘studies’ by homeopaths that are ridiculously bad in almost every aspect, such as a chikungunya ‘study’ from India based on just one 2 x 2 table with columns ‘Treated’ and ‘Untreated’ and rows ‘Disease Symptoms’ and ‘No Disease Symptoms’, with no attempts whatsoever to control for any of the many possible confounders. No doubt these people thought that they were doing scientific research, and with a positive outcome for their favourite SCAM, but no real scientist would touch this with a 10-foot pole.
Then again, this is all just personal speculation based on anecdotal observations and even guesswork of yours truly (i.e. category 2 at best), and more rigorous scientific research is certainly warranted.
“True Scientist does not exist, at least not as the caricature sketched here, fully detached from emotions and truly objective in their observation of reality.”
I entirely agree
” this is all just personal speculation based on anecdotal observations and even guesswork”
In my case, it is more than guesswork; it is based on 30 years of knowing many of the people involved and working with many of them.
“I know of only one SCAM researcher who comes close to this category, and his name is Edzard Ernst.”
THANKS FOR THE COMPLIMENT!!!
But I readily admit that I do have biases too – interestingly, they changed over time; 30 years ago they were in favour of SCAM!
@Edzard
I think that the compliment is well-deserved: as you say, you were initially biased in favour of SCAM, and mostly set out to give it a better scientific foundation. However, your research, your results and your conclusions came up negative most of the time, which strongly suggests that your research methods did not suffer from this bias. And, of course, you were also willing to accept these negative outcomes, even though this eventually caused quite a bit of trouble. I’d say these are pretty strong hallmarks of a True Scientist.
Nowadays, you appear to exhibit a rather strong bias against SCAM (as do most people who agree with you here, me included). Then again, we could of course reframe this scientifically as a Bayesian prior: with almost no form of SCAM proven effective even after decades of proper scientific research, it is only sensible to assume that any SCAM research will have a negative outcome, and work from there to see if this assumption can be proven wrong.
that’s kind of you to say;
alas, Dana probably has a different perspective on this?
And forty or so years before that – in favour of faries!
Scientists use research to get as close as possible to the ‘truth’ of an issue.
Camists use ‘research’ as a Cloak of Intellectual Respectability.
But that is a wrong use, demonstrates lack of integrity and is to be deprecated.
@Richard Rawlins
As the old saying goes: a quack uses science like a drunkard uses a lamp post: for support, not for illumination.
In my version the quack uses science like a dog uses a lamppost.
@Socrates
Spot on, even better than the drunkard analogy …
RR said:
“,,,I think that this is pretty much inevitable in order to maintain the motivation to do scientific work in the first place.”
There is no doubt about that: all research has to be hypothesis-driven because there is no alternative (that is sensible).
***********
“Crossing the floor” shows the integrity of EE.
***********
My “blackbox” argument against homeopathy (from complete ignorance of it) would be that if plain water can be affected by absent molecules (of the “cure”), then that water must also have “memory” of everything else that was there before, so repeatability would be impossible.
@Old Blob
Have you been smoking something more psychoactive than ordinary tobacco?
The “true scientist” sounds a bit like the definition of the “reasonable man” given by Marrow LJ in Fardell v Potts (1927) Herbert’s Misleading Cases in the Common Law 8 at 10: