The Hidden Incentives Driving AI Hallucinations
IBM, September 18,2025
Artificial intelligence has a confidence problem. The same large language models (LLMs) that generate fluent text for millions of users can also invent facts with equal poise, a flaw researchers call hallucination. And despite steady improvements in model accuracy, this tendency to produce wrong but plausible answers has proven stubbornly hard to fix.
A new study by OpenAI suggests the problem is not a mysterious glitch deep in the code, but a side effect of how researchers measure progress in AI. Benchmarks that rank models by accuracy can push them to guess rather than hold back, rewarding confident errors over admissions of uncertainty. It is a subtle incentive with wide consequences: the very scoreboards that drive competition in the field may be teaching systems to bluff.
'Evaluations are really at the heart of it, similar to how KPIs incentivize humans,' Ayhan Sebin, an AI Ecosystem and Partnership Development Executive at IBM, told IBM Think in an interview. 'If the scoring system rewards guesses, then the models will learn to guess.'