Generative AI in organizations and society
Multi-turn AI chats expose weaknesses hidden by single-turn tests
MT-Eval compares single- and multi-turn conversations and finds that models often lose ground as dialogue history grows.
A model may answer a standalone question well and still lose track of a conversation. MT-Eval tests that difference by comparing multi-turn interactions with single-turn versions of the same queries. Its results show that standard one-question benchmarks can miss weaknesses in remembering and using context.
Four patterns of conversation
The authors identify four common interaction types from human–assistant conversations: recollection, where the assistant must remember earlier information; expansion, where the discussion explores related topics; refinement, where a request changes or becomes more specific; and follow-up, where a new question depends on an earlier answer. The benchmark contains 1,170 multi-turn queries and creates single-turn counterparts so model performance can be compared under both conditions (task design and dataset statistics, PDF pp. 1–4; proceedings pp. 20153–20156).
The researchers evaluate ten language models, including open- and closed-source systems. They measure response quality across the interaction types and the change between a multi-turn query and its single-turn equivalent. That distinction matters: a model’s general knowledge may be adequate while its ability to keep track of prior turns is not (evaluation, PDF pp. 5–7; proceedings pp. 20157–20159).
History can help and hinder
The results show that most models perform worse in multi-turn settings than in corresponding single-turn settings. Closed models generally lead overall, while some open models exceed GPT-3.5-Turbo on particular tasks. The authors identify distance from relevant information and error propagation as important factors: an answer can become less reliable when the needed detail is far back in the dialogue, or when a previous mistake shapes later turns (main results, Tables 2–3, PDF pp. 7–9; proceedings pp. 20159–20161).
“Error propagation” means that an incorrect earlier response can become part of the context a model relies on later. Longer conversations therefore pose a different test from simply placing more text in a prompt. Models must select which pieces of history remain relevant as the user’s goals shift.
What the benchmark does not represent
MT-Eval covers selected conversation patterns and a finite set of generated or adapted examples. Human review was used in constructing new items, but the benchmark cannot capture every real-world dialogue or measure all qualities users value. The authors discuss these limits and possible extensions in their limitations section (PDF pp. 13–14; proceedings pp. 20165–20166).
Possible implications for AI assistants
For organizations assessing assistants used across ongoing tasks, a single-turn score may not predict how well a system follows an evolving conversation. MT-Eval offers a way to examine memory distance, follow-ups and correction after a wrong answer. Broader testing could help clarify which interaction designs support dependable context use.
Bibliography & sources
- Wai Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. “MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 20153–20177 (2024). https://doi.org/10.18653/v1/2024.emnlp-main.1124.