--- title: "Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs" description: "Simulations with full access to every character’s goals look more successful than interactions where each agent has realistic private information." url: "https://digitvibe.com/blog/misleading-success-llm-social-simulations/" --- [All blog posts](https://digitvibe.com/blog/) AI and social simulation # Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs Simulations with full access to every character’s goals look more successful than interactions where each agent has realistic private information. By DigitVibe blog · Published November 11, 2026 * Peer reviewed · 2024 ![Complete abstract from Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs.](https://digitvibe-blog.workspace-scripts.workers.dev/media/6c16cbe0de949d425fb9c4548fa5c96533e0522d92fea45a9a09fc81f76c494f.webp) Complete abstract from the original paper. Source: Association for Computational Linguistics and Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, Maarten Sap. Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, Maarten Sap. 2024. “Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs.” In \*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing\*, 21692–21714. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.1208. Reproduced under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); no third-party material is included. [Source paper, page 1](https://doi.org/10.18653/v1/2024.emnlp-main.1208) [View full-size excerpt](https://digitvibe-blog.workspace-scripts.workers.dev/media/6c16cbe0de949d425fb9c4548fa5c96533e0522d92fea45a9a09fc81f76c494f.webp) ### A social simulation can look successful for the wrong reason A language model can write a smooth conversation when it knows what every person wants. Real conversations are different: people hold private information, misunderstand each other and infer intentions from what is said. Zhou and colleagues compare these two settings and find that simulations with an omniscient narrator make language models appear more capable than turn-by-turn interactions where each agent knows only its own role. The researchers define SCRIPT mode as one model generating both sides of an interaction while seeing all characters’ profiles, relationships and goals. In AGENTS mode, each model plays a single character and receives only that character’s information. A third condition, MINDREADERS, gives agents access to each other’s private goals. The team tested GPT-3.5 and Mixtral-8x7B in scenarios drawn from Sotopia, including cooperative tasks and bargaining. They ran 450 simulations per model and setting and scored whether agents achieved their goals. ### Information asymmetry changes the result SCRIPT and MINDREADERS simulations achieved significantly higher goal-completion scores than AGENTS mode (Figure 2, pages 21694–21695). This pattern suggests that access to other characters’ goals can make a task much easier. In cooperative scenarios, MINDREADERS performed much like SCRIPT. In competitive bargaining, simply revealing the other side’s information did not fully reproduce the results of omniscient scripting. In SCRIPT mode, interactions tended to end in agreement, while AGENTS characters more often left when a deal seemed unlikely. The difference also appeared in conversational quality. Human evaluators judged SCRIPT dialogues as more natural than AGENTS dialogues. Agent-mode turns were longer on average: about 29.83 tokens versus 16.02 for SCRIPT in one comparison, and 33.72 versus 13.36 in another (Figure 4, page 21696). The authors also fine-tuned GPT-3.5 on 1,252 omnisciently generated episodes. That training offered limited improvement when the model later had to participate in AGENTS-style interactions. Training on idealized scripts did not fully transfer to the information gaps of a real interaction. The result challenges a common interpretation of fluent dialogue generation. A plausible conversation produced from a complete view of all participants may reveal more about a model’s ability to author a scene than its ability to act as a participant. However, the study uses generated scenarios, two model families and model-based scoring of goal completion. Automatically judging success is only a proxy, and changing from one omniscient writer to two separate agents changes several features at once. The experiments do not directly compare AI interactions with human conversations. ### Evaluate the social conditions, not just the script For teams using simulations to study negotiation, collaboration or customer interactions, the paper makes a case for reporting who can see what information. Evaluation that omits information access may overstate social capability. A more realistic setting can reveal where agents fail to coordinate, infer intentions or manage uncertainty. The broader implication is that simulation fidelity depends not only on whether the dialogue sounds human, but also on whether the conditions of interaction resemble those people face. The paper recommends describing simulation perspectives and information access so readers can judge what a reported success score actually represents. In particular, observed success within a modeled social setting should not be mistaken for validated prediction of behavior outside that environment or for evidence of broad deployment safety. ## Bibliography & sources 1. Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, Maarten Sap. 2024. “Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs.” In \*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing\*, 21692–21714. Association for Computational Linguistics. [https://doi.org/10.18653/v1/2024.emnlp-main.1208](https://doi.org/10.18653/v1/2024.emnlp-main.1208). --- DigitVibe. Index of public content: https://digitvibe.com/llms.txt