OpenAI researchers say they have identified a cause of why artificial intelligence systems produce fabricated information and outlined a possible solution. The problem, known as “hallucination,” occurs when AI assistants generate false claims, fictional sources or fabricated quotes with apparent confidence.
In a paper published Sept. 5, Adam Kalai, Santosh Vempala of Georgia Tech, and other researchers argue that hallucinations are driven less by sloppy model design and more by the way performance is measured. “Hallucinations aren’t caused by sloppy writing, but by the way current evaluation metrics are set up,” the paper states. Current benchmarks tend to reward confident guesses and penalize expressions of uncertainty, much like a multiple-choice test that favors guessing over leaving answers blank.
The study highlights that leaderboards ranking AI systems prioritize accuracy while overlooking error rates and uncertainty. Researchers recommend that scoreboards instead penalize confident mistakes more strongly and provide some credit when a model chooses not to answer, in order to encourage greater honesty in AI outputs.
One example cited in the paper shows the trade-off between accuracy and caution. In the SimpleQA benchmark, one model abstained from answering more than half of the questions but was wrong only 26% of the time when it did respond. Another model answered almost everything but hallucinated in about 75% of cases.
The researchers conclude that rewarding models for acknowledging uncertainty could reduce fabricated outputs and provide users with responses that are less misleading, even if less complete.
Source: OpenAI