Why AI accepts false premises

Ask a model a question that assumes a false fact, and it may repeat that fact as though it were true. A study of this “false premise” failure asks why models miss facts they can sometimes retrieve in a direct question.

A false claim can steer the answer

The researchers define false-premise hallucination as generating an unsupported answer when a question embeds an incorrect assumption. Their example asks why Albert Einstein received the Nobel Prize in Physics in 1920, though he received it in 1921. The model may accept the date in the question rather than correct it.

The authors construct evaluation data using questions designed around known facts, then compare direct factual questions with versions containing a false premise. They examine both the model's answers and internal information flow. In Llama-2-13B, the paper reports 100% accuracy on 4,004 direct-asking questions, compared with 24.3% on their corresponding false-premise questions (introduction, PDF p. 2). This striking gap illustrates how the wording of an input can change performance even when relevant knowledge is available to the model.

Their analysis identifies a small set of attention heads—components that distribute a model's attention among input tokens—as influential when the model processes the false object in a premise. They call these “false premise heads.” The authors propose FAITH, which constrains the identified heads during inference. Table 2 (PDF p. 8) reports results across four question templates. The paper finds that constraining roughly 1% of attention heads can produce nearly a 20% improvement in model performance, as summarized in the abstract; results vary by question template and model. Table 3 (PDF p. 8) explores generalization across templates and knowledge sets.

What this reveals, and what it does not

The study examines a particular form of factual failure, rather than every kind of hallucination. The analysis uses models up to 13 billion parameters because of computing limits. It evaluates the influence of individual attention heads, not every possible combination of heads, as the authors explain in their limitations section (PDF p. 9). The benchmark design and model scale therefore bound how broadly the mechanism can be generalized.

For managers and practitioners, the finding offers a reminder that an AI response can be shaped by assumptions built into a question. A model's ability to answer a direct fact-check does not guarantee that it will challenge the same incorrect premise in a natural conversation. The paper's intervention operates on model internals, so it is not a user-facing correction feature by itself.

Possible implications for practice

The results suggest that evaluation should include prompts that contain misleading assumptions, particularly where people may treat fluent answers as confirmation. FAITH provides evidence that specific internal signals can be studied to reduce this failure in tested models. Whether that approach will transfer to other architectures and everyday use remains open.

Bibliography & sources

1. Hongbang Yuan; Pengfei Cao (鹏飞 曹); Zhuoran Jin; Yubo Chen; Daojian Zeng; Kang Liu; Jun Zhao (2024). “Whispers that Shake Foundations: Analyzing and Mitigating False Premise Hallucinations in Large Language Models.” Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2670–2683. DOI: 10.18653/v1/2024.emnlp-main.155. Source paper