Large language models (LLMs) can generate false facts, often referred to as “hallucinations”. The technique is often used for generating, plausible-sounding, but completely fabricated references in scientific papers. Apparently, it is happening at an alarmingly high rate. One study revealed that ~ 47% of references generated by LLMs like ChatGPT, were completely made up, and only a small percentage (around 7%) were both authentic and accurate.
The fake references are highly convincing, making the fabricated entries look legitimate. Such fraudulent papers can thus easily pass through peer-review. Some researchers and ‘paper mills’ knowingly use AI to generate fraudulent papers or references to inflate their publication record.
A recent study found that “LLMs are highly susceptible to adversarial hallucination attacks, frequently generating false clinical details that pose risks when used without safeguards. While prompt engineering reduces errors, it does not eliminate them”. Several medical journals and institutions have therefore taken to investigating “hallucination”and are issuing retractions or correction notices for such papers.
The case of a US govermnent document previously discussed on this blog can serve as an apt example of “hallucination”:
The Trump administration released a report billed as a “clear, evidence-based foundation” for action on a range of children’s health issues. But the report, from the presidential “Make America Healthy Again Commission”, cited studies that did not exist, as reported by the NEW YORK TIMES and several further publications. These included fictitious studies on direct-to-consumer drug advertising, mental illness and medications prescribed for children with asthma.
“It makes me concerned about the rigor of the report, if these really basic citation practices aren’t being followed,” said Katherine Keyes, a professor of epidemiology at Columbia University who was listed as the author of a paper on mental health and substance use among adolescents. Dr. Keyes has not written any paper by the title the report cited, nor does one seem to exist by any author.
Dr. Ivan Oransky, co-founder of Retraction Watch, said the errors in the report were characteristic of the use of generative artificial intelligence. He said that, while he did not know whether the government had used A.I. in producing the report or the citations, “we’ve seen this particular movie before, and it’s unfortunately much more common in scientific literature than people would like or than really it should be.”
Asked at a news conference on Thursday whether the report had relied on A.I., the White House press secretary, Karoline Leavitt, deferred to the Department of Health and Human Services. Emily Hilliard, a spokeswoman for the department, did not answer a question about the source of the fabricated references and downplayed them as “minor citation and formatting errors.”
It should, however, be clear to anyone concerned that “hallucination” is far from a minor error. It is outright fraud that has the potential to do huge harm.
3 Responses to “Hallucination”, the phenomenon of AI-generated, plausible-sounding misinformation
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
This problem with AI rears its ugly head not just in science, but in every place where it is used. And in hindsight, Id say that this is an inevitable feature of AI, because its core mechanism is literally making up information by estimating how a linguistic thread is most likely to proceed, NOT starting out with and hanging on to factual (and hopefully vetted) information. So it’s literally style over substance: never mind if something is true or not, as long as it sounds nice and plausible, it’s dumped on your screen.
And it gets even worse: even if LLMs are forced to carry out sanity checks and fact checks on the information they’re dreaming up, these checks are based on information gleaned from the Internet – a fast growing part of which is, you guessed it, produced by AI. Because it is far cheaper and quicker to have AI produce texts, reports and even images and videos than to have humans do it. According to recent estimates, already more than half the information on the Internet is AI-generated.
So basically, AI appears to be training itself with the crap it produced itself. Something tells me that this will not end well.
I have read that 99% of all species that ever existed on earth are extinct today. This may sound pessimistic but for severaI reasons, I suspect that unfortunately this new AI technology is another step in this direction for our own species.
QUOTE Extinction, Wikipedia
As long as species have been evolving, species have been going extinct. It is estimated that over 99.9% of all species that ever lived are extinct. The average lifespan of a species is 1–10 million years,[38] although this varies widely between taxa. A variety of causes can contribute directly or indirectly to the extinction of a species or group of species. “Just as each species is unique”, write Beverly and Stephen C. Stearns, “so is each extinction … the causes for each are varied—some subtle and complex, others obvious and simple”.[39] Most simply, any species that cannot survive and reproduce in its environment and cannot move to a new environment where it can do so, dies out and becomes extinct. Extinction of a species may come suddenly when an otherwise healthy species is wiped out completely, as when toxic pollution renders its entire habitat unliveable; or may occur gradually over thousands or millions of years, such as when a species gradually loses out in competition for food to better adapted competitors. Extinction may occur a long time after the events that set it in motion, a phenomenon known as extinction debt.