An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Making

Who gets to decide when AI agents disagree?

Many multi-agent AI systems rely on one “judge” agent or a simple vote to settle competing answers. But the method used to combine those answers can shape the outcome. Zhao, Wang and Peng examine this neglected design choice through social choice theory—the study of how individual preferences are aggregated into a group decision—and introduce GEDI, a modular toolkit for testing election-style rules among language-model agents.

Their survey of 52 recent multi-agent systems found that 23 used a dictatorial decision-maker, 13 used plurality voting, one used a utilitarian method and 15 did not specify a decision rule or did not require one (Figure 1, page 2712). The authors note that each approach has trade-offs. A dictator is efficient but creates a single point of failure. Plurality counts only first-choice votes, ignoring other preferences. More sophisticated ranked methods can capture broader preferences but also have their own limitations. No voting system satisfies every desirable criterion in every situation.

Ranking answers changes benchmark results

GEDI collects each agent’s ordering of answer choices and applies a selected voting rule. The authors compare 10 approaches across MMLU, MMLU-Pro and ARC-Challenge, using six open models and two proprietary systems. Each question is answered by a group of 10 agents built on the same model; accuracy is averaged across five runs.

On MMLU, several ranked voting rules generally matched or exceeded a blind dictator. For GPT-4, Borda Count—the method assigning points by each option’s rank—reached 82.5% accuracy, compared with 75.6% for blind dictatorship, a 6.9-point difference. GPT-3.5 reached 65.9% under Borda against 60.8% under blind dictatorship. Other models showed smaller changes. On MMLU-Pro and ARC-Challenge, effects likewise depended on the model, benchmark and rule. The authors report that Minimax and Ranked Pairs produced generally robust positive effects across their tested combinations (Table 2, pages 2717–2718).

The findings also highlight costs and limits. Informed dictatorship, where the designated agent sees the other agents’ ballots, used more computation than voting-based aggregation but did not consistently outperform it. With only three agents, some voting methods already showed gains in some settings, though the main reported comparison uses ten. The agents share the same underlying model, so their outputs can share biases and errors; a ballot profile from AI agents is not equivalent to a diverse group of independent people. Multiple-choice tests offer clear answer labels, but real organizational decisions involve competing values, uncertain outcomes and no single gold answer.

The aggregation rule is part of the system

For practitioners evaluating AI teams, the result suggests that “multi-agent” is not enough detail to explain how a system reaches its conclusion. The decision rule is an active component: it can affect accuracy, robustness and the visibility of minority preferences. GEDI helps make those choices explicit and comparable. The paper does not establish a universally superior election method; instead, it shows why the aggregation step deserves its own evaluation rather than being treated as a minor implementation detail.

These outcomes should be read as evidence about this evaluation and its tested alternatives, not as a universal ranking of social choice methods. Other group sizes, voting rules, interaction patterns and model versions may change which procedure performs best. The work is valuable in making those tradeoffs measurable.

Bibliography & sources

1. Xiutian Zhao, Ke Wang, Wei Peng. 2024. “An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Making.” In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, 2712–2727. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.158.