Suppose a quiz asks for the birthday of someone the model has never read about. A model that guesses has a one-in-365 chance of being right. A model that says “I don’t know” scores zero. Over thousands of questions, the guesser comes out ahead. That is the example OpenAI uses to explain a good part of why chatbots invent things.
A guess that sounds like knowledge
In September 2025, researchers from OpenAI and Georgia Tech published a paper titled “Why Language Models Hallucinate.” Their central claim is that the “training and evaluation procedures reward guessing over acknowledging uncertainty.” Most benchmarks grade a model on accuracy alone, so a model that bluffs when unsure scores better than one that abstains. Models are tuned to do well on such tests, and the habit carries over into everyday answers.
The paper also traces an earlier cause. During pretraining, a model learns to predict the next word from enormous amounts of text, with no labels marking which statements are true. For patterns that recur, such as grammar, that works well. For arbitrary facts that appear rarely and follow no pattern, a person’s birthday for instance, there is little for the model to learn, and errors are close to inevitable. The authors describe hallucinations as ordinary errors in a classification problem: when a model cannot tell a false statement from a true one, plausible falsehoods follow.
What NIST calls it
The US National Institute of Standards and Technology uses a different word. Its generative AI risk profile, published in July 2024 as a companion to its AI Risk Management Framework, calls the problem confabulation: confidently stated but erroneous or false content. It notes that the term is colloquially known as hallucinations or fabrications, and says confabulations have been shown to be pervasive in current systems.
NIST also makes a point that is easy to miss. Confabulation, it says, is not an inherent flaw of language models themselves but a result of pre-training built around predicting the next word. The scope is broader than invented facts, too. It includes mistaking people, places or details of historical events, and output that contradicts the source text it was given or an earlier statement in the same conversation.
Where errors cluster
The research and the NIST profile point to the same weak spots. Rare facts, specific names, exact dates, figures and references are the most likely to be wrong, because they are arbitrary and thinly covered in training data. A fluent explanation of a well-known concept is usually more reliable than a precise detail about an obscure one.
Confidence offers no protection. A model’s tone stays assured whether or not the content is right, so how certain an answer sounds says nothing about whether it is true.
Will it go away?
The OpenAI paper argues the problem can be reduced rather than eliminated, and that the fix is partly social. It proposes modifying the scoring of existing benchmarks that dominate leaderboards, so that a confident error costs more than an honest “I don’t know.” Developers are also pairing models with document search, as described in our guide to retrieval-augmented generation, which gives a model something to quote instead of recall.
Both help. Neither means a model can be assumed to be right. Systems that act on their own conclusions, such as the AI agents now being built into business software, raise the stakes, because a false premise can turn into an action.
How to check an answer
Verification is mostly a habit of going to the original source.
Check specifics against a primary source: an official website, a published paper, a regulator’s page, a company’s own filing. If a chatbot offers a citation, open it, because references can be invented or can exist without saying what the answer claims. Asking the same question in different words and comparing the details is a cheap test. If the specifics change, treat them as unreliable.
Match the effort to the stakes. Summarizing a document you supplied or rewording your own draft carries little risk. Medical, legal, financial and safety questions should be settled by a qualified professional, with the chatbot at most a starting point for what to ask.
