Generative AI and automation
Training LLM agents through multi-turn preference
Direct Multi-Turn Preference Optimization adapts preference learning to sequential agents and tests robustness on three simulated tasks.
A language agent makes a sequence of choices as it interacts with an environment. Training methods that compare only single answers can miss how small errors accumulate across those turns. Direct Multi-Turn Preference Optimization (DMPO) adapts preference training to that sequence.
Comparing whole interaction paths
Direct Preference Optimization (DPO) trains a model from pairs of preferred and dispreferred responses. For an agent, those responses may be full trajectories: sequences of observations and actions. The paper explains that applying DPO directly to multi-turn tasks creates a mathematical difficulty because the normalization term in the preference model cannot be canceled in the same way as in single-turn settings.
The authors derive DMPO by changing the constraint used in the reinforcement-learning objective and adding length normalization. The result is intended to let the method compare preferred and dispreferred paths without the same partition-function obstacle. The experiments test three agent datasets and examine both noisy and clean preference settings. In the noisy setting, some dispreferred trajectories contain lower-quality actions; in the clean setting, the comparison trajectories are filtered for quality.
Table 2 (PDF p. 7) reports average rewards for two base models across three agent tasks in the noisy setting. DMPO outperforms DPO on all unseen test sets and most seen test sets for both models, according to the authors' analysis. In the clean comparison using Llama-2-7B-Chat, Table 3 (PDF p. 7) shows DMPO above the ETO baseline across WebShop and ScienceWorld. The paper also tests robustness to different lengths of dispreferred trajectories (discussion around PDF p. 8).
These results are about simulated tasks and reward scores. They do not show the effect of DMPO on real customer interactions, workplace productivity or the risks of giving an agent access to external tools.
Why trajectory quality matters
The approach is designed to address compounding errors: when an agent's early action is mistaken, later actions may build on the resulting state. Its length normalization also aims to limit an imbalance caused by preferred and dispreferred paths containing different numbers of steps. Together, those design choices make the preference comparison more specific to multi-turn behavior.
The limitations section (PDF p. 9) says the study focuses on turn-wise task formulation, uses 7-billion-parameter models and tests simulated datasets. The authors point to larger models and datasets, and token-wise formulations, as areas for further investigation.
Possible implications for practice
DMPO suggests that feedback for automated agents may need to reflect the quality of a complete action sequence. This could be relevant when a system performs a multi-step task, though benchmark gains alone do not determine whether deployment would be reliable. The paper offers a method for studying how preferences shape agent behavior across turns.
Bibliography & sources
- Wentao Shi; Mengqi Yuan; Junkang Wu; Qifan Wang; Fuli Feng (2024). “Direct Multi-Turn Preference Optimization for Language Agents.” Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2312–2324. DOI: 10.18653/v1/2024.emnlp-main.138. Source paper