---
title: "Multi-turn AI chats expose weaknesses hidden by single-turn tests"
description: "MT-Eval compares single- and multi-turn conversations and finds that models often lose ground as dialogue history grows."
url: "https://digitvibe.com/blog/mt-eval-multi-turn-conversation-tests/"
---

[All blog posts](https://digitvibe.com/blog/)

Generative AI in organizations and society

# Multi-turn AI chats expose weaknesses hidden by single-turn tests

MT-Eval compares single- and multi-turn conversations and finds that models often lose ground as dialogue history grows.

By DigitVibe blog · Published November 12, 2026

*   Peer reviewed · 2024

A model may answer a standalone question well and still lose track of a conversation. MT-Eval tests that difference by comparing multi-turn interactions with single-turn versions of the same queries. Its results show that standard one-question benchmarks can miss weaknesses in remembering and using context.

![Complete abstract from MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models.](https://digitvibe-blog.workspace-scripts.workers.dev/media/68707ea44f19e88563f12b08e68733ef0d0d195a42698204d0083e581ca7cf30.webp)

Complete abstract from the original paper. Wai Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong (2024), [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). [Source paper, page 1](https://doi.org/10.18653/v1/2024.emnlp-main.1124) [View full-size excerpt](https://digitvibe-blog.workspace-scripts.workers.dev/media/68707ea44f19e88563f12b08e68733ef0d0d195a42698204d0083e581ca7cf30.webp)

### Four patterns of conversation

The authors identify four common interaction types from human–assistant conversations: recollection, where the assistant must remember earlier information; expansion, where the discussion explores related topics; refinement, where a request changes or becomes more specific; and follow-up, where a new question depends on an earlier answer. The benchmark contains 1,170 multi-turn queries and creates single-turn counterparts so model performance can be compared under both conditions (task design and dataset statistics, PDF pp. 1–4; proceedings pp. 20153–20156).

The researchers evaluate ten language models, including open- and closed-source systems. They measure response quality across the interaction types and the change between a multi-turn query and its single-turn equivalent. That distinction matters: a model’s general knowledge may be adequate while its ability to keep track of prior turns is not (evaluation, PDF pp. 5–7; proceedings pp. 20157–20159).

### History can help and hinder

The results show that most models perform worse in multi-turn settings than in corresponding single-turn settings. Closed models generally lead overall, while some open models exceed GPT-3.5-Turbo on particular tasks. The authors identify distance from relevant information and error propagation as important factors: an answer can become less reliable when the needed detail is far back in the dialogue, or when a previous mistake shapes later turns (main results, Tables 2–3, PDF pp. 7–9; proceedings pp. 20159–20161).

“Error propagation” means that an incorrect earlier response can become part of the context a model relies on later. Longer conversations therefore pose a different test from simply placing more text in a prompt. Models must select which pieces of history remain relevant as the user’s goals shift.

### What the benchmark does not represent

MT-Eval covers selected conversation patterns and a finite set of generated or adapted examples. Human review was used in constructing new items, but the benchmark cannot capture every real-world dialogue or measure all qualities users value. The authors discuss these limits and possible extensions in their limitations section (PDF pp. 13–14; proceedings pp. 20165–20166).

### Possible implications for AI assistants

For organizations assessing assistants used across ongoing tasks, a single-turn score may not predict how well a system follows an evolving conversation. MT-Eval offers a way to examine memory distance, follow-ups and correction after a wrong answer. Broader testing could help clarify which interaction designs support dependable context use.

## Bibliography & sources

1.  Wai Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. “MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 20153–20177 (2024). [https://doi.org/10.18653/v1/2024.emnlp-main.1124](https://doi.org/10.18653/v1/2024.emnlp-main.1124).

---

DigitVibe. Index of public content: https://digitvibe.com/llms.txt