Multiple experts, more reliable AI answers

One expert can produce a confident answer that overlooks a legitimate perspective. Multi-expert Prompting asks a language model to generate several expert viewpoints, combine them and choose a response through a structured decision process.

A structured process for combining perspectives

The method uses a model to create several expert roles suited to a question. Each role answers independently. The model then aggregates their responses and selects among the individual and combined answers. The process follows seven subtasks inspired by the Nominal Group Technique, a structured approach to group decision-making that collects ideas, identifies agreement and disagreement, and evaluates options. Unlike systems that debate repeatedly, Multi-expert Prompting completes its aggregation in a single chain of thought.

The authors test truthfulness, factuality, toxicity, hurtfulness, informativeness and usefulness using tasks including TruthfulQA, BOLD and ExpertQA. With ChatGPT, the method improves TruthfulQA truthfulness by 8.69% over the best baseline, according to the paper (Table 1, PDF p. 5). On open-ended ExpertQA questions, the model evaluator rated responses as more informative in 75% of comparisons on average and more useful in 76.5% (Figure 3 and section 4.2, PDF p. 5). A human evaluation comparing ChatGPT's ratings with annotators reports 93% agreement on average across two criteria (discussion, PDF p. 6).

A central feature is aggregation, not simply asking the model to adopt one role. The authors report that individual perspectives can counterbalance one another and produce a fuller response. That benefit is most relevant to open-ended questions with several defensible viewpoints.

The tradeoffs behind a richer answer

The experiments use model-based judgments for several outcomes, with a human evaluation to check agreement for selected comparisons. Evaluation choices shape what “truthful,” “useful” or “safe” means in the reported tables. The paper's limitations section (PDF pp. 9–10) says that the seven subtasks may be unnecessary for short-form tasks, that the method relies on instruction-following ability, and that equally weighted expert opinions may not reflect real settings. Models can also invent roles or perform them poorly in domains where they have limited training.

This prompting method adds generation and evaluation steps. Appendix C, Table 11 reports ChatGPT token use and API-cost estimates: the authors describe roughly twice as many tokens on TruthfulQA, about 1.5 times as many on BOLD, and the total as relatively affordable at around US$4 for both datasets. These are dated API estimates, not a measure of full business-deployment costs or latency.

Possible implications for practice

The study suggests that using multiple viewpoints could improve some open-ended answers, especially when relevant considerations differ. It also highlights that a generated “expert” is a simulated role, not a qualified human reviewer. The paper offers evidence about benchmark response quality, not a substitute for domain expertise or an assurance of factual correctness.

Bibliography & sources

1. Do Xuan Long; Duong Ngoc Yen; Luu Anh Tuan; Kenji Kawaguchi; Min-Yen Kan; Nancy F. Chen (2024). “Multi-expert Prompting Improves Reliability, Safety and Usefulness of Large Language Models.” Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20370–20401. DOI: 10.18653/v1/2024.emnlp-main.1135. Source paper