When should an AI answer or refuse?

An AI that answers every question can sound helpful while making avoidable factual mistakes. Learn to Refuse (L2R) tests an alternative: define the system's knowledge through a structured source and let it abstain when it lacks evidence.

A limited knowledge base and a choice to abstain

The paper separates a language model from an external knowledge base. The base begins empty and can be filled with verified facts. When a question arrives, the system checks whether its evidence supports an answer. It can then give a traceable response or refuse. The authors also introduce Automatic Knowledge Enrichment, which uses the model's own knowledge to suggest candidate additions and asks the model to validate them before they enter the base. This validation is intended to help populate the system, though it does not make generated knowledge infallible.

L2R uses “soft” and “hard” refusal mechanisms in its question-answering setup. The study compares it with baseline models and retrieval-augmented generation on TruthfulQA, CommonsenseQA and MedQA. Table 2 (PDF p. 7) reports results across these benchmarks. On TruthfulQA's MC1 task, GPT-3.5-Turbo achieved 46.6% accuracy, while L2R-GPT reached 65.1%; it answered 654 rather than 817 questions. The paper notes that this improvement came with 163 fewer answered questions (Table 1, PDF p. 7 and accompanying text). The results indicate a tradeoff between answer coverage and accuracy.

The system also gains on CommonsenseQA and MedQA in the reported comparisons (Table 2, PDF p. 7). These results come from research benchmarks, not a live service with a broad, continuously updated knowledge base.

Abstention changes how performance should be read

A refusal can prevent a wrong answer, but it also leaves a user's question unresolved. Accuracy among answered questions and the number of questions answered describe different aspects of service quality. L2R's tables report answer counts alongside accuracy so readers can see this difference. Its structured base also aims to make evidence traceable, though knowledge entry and quality control create their own work.

The authors describe the paper as a demonstration in question-answering scenarios and note that the knowledge bases contain hundreds of knowledge items in some experiments. They do not claim the system eliminates hallucinations or that scaling to millions of facts is settled (limitations, PDF p. 9). The refusal function itself is simple and the tested knowledge scope is limited.

Possible implications for practice

The findings suggest that a system's reliability can improve when it has a meaningful option to decline unsupported answers. This shifts evaluation toward both accuracy and coverage. For organizations, the model raises questions about who verifies knowledge, how refusals are handled, and whether an unanswered request is acceptable in a given service.

Bibliography & sources

1. Lang Cao (2024). “Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism.” Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3628–3646. DOI: 10.18653/v1/2024.emnlp-main.212. Source paper