Fairer preferences, better AI judgments?

When a large language model (LLM) compares two AI-written summaries, a small change in the question can change which one it prefers. That makes automated evaluation difficult to trust: a score may reflect the wording of the prompt as much as the quality of the texts. Zhou and colleagues examine whether balancing a model’s preferences can improve the human alignment of its judgments.

A prompt optimizer for pairwise judgments

The authors introduce ZEPO, short for Zero-shot Evaluation-oriented Prompt Optimization. A pairwise evaluator receives two candidate texts and chooses a preferred one. The team first tests how such preferences shift under semantically equivalent instructions, then uses a zero-shot fairness objective to search for an evaluation prompt with a less skewed preference distribution. “Zero-shot” here means the optimizer does not require human-labeled examples for its tuning objective; it still uses unlabeled input samples to estimate preferences.

The study evaluates ZEPO on representative meta-evaluation benchmarks, including tasks that compare generated summaries with human judgments. It measures the association between the evaluator’s decisions and human ratings, and compares the optimized prompts with established LLM-evaluation methods. The workflow is designed to tune the prompt used by an evaluator, rather than retrain its underlying model.

Balancing preference improves alignment

The researchers find that LLM evaluators can show highly skewed preferences even when prompts carry nearly the same meaning. Their central result is that fairer predictive preferences consistently correspond to judgments more aligned with human assessments. ZEPO improves the reported performance of pairwise evaluators on the meta-evaluation benchmarks without labeled data (abstract, proceedings page 1241; experiments and results, proceedings pages 1246–1250).

The term “fairness” in this paper refers to the balance of an evaluator’s preference distribution, not a general measure of social fairness. A more balanced pattern matters because an evaluator that routinely favors one option position, style or class may mis-rank candidates. Yet the measured link to human alignment remains bounded by the evaluation benchmarks and human labels used in those comparisons.

Why this matters for teams using AI scores

Automated evaluators are increasingly used to compare generated content, including in model development and quality review. If a prompt can make a judge more consistent with people’s preferences, it may improve the signal that teams use to compare system versions. The finding also highlights a governance question: evaluation prompts themselves are part of the measurement instrument and deserve testing.

The measured gains are substantial but benchmark-bound: ZEPO improved average meta-evaluation performance by 17% with Mistral-7B and 10% with Llama-3 8B against state-of-the-art pairwise evaluators. Table 1 reports a 14% gain on NewsRoom COH for Mistral-7B and 8% on COH for Llama-3 8B; Table 2 shows the fairness objective adds to permutation debiasing. The optimizer uses GPT-3.5-turbo with greedy prompt search and needs enough random unlabeled examples to estimate preferences (Tables 1–2; §5).

Study limits

ZEPO needs enough random, unlabeled examples to estimate an evaluator’s preference distribution. The authors focus mainly on pairwise judgments, so the results do not show how the method performs for ranking many candidates or selecting from multiple classes. Their optimizer also explores prompt candidates through a greedy search (limitations, proceedings pages 1250–1251). The method’s performance may depend on the evaluator and benchmark.

Possible implications for practice

Organizations using AI to grade, rank or compare generated material could treat evaluator prompts as a source of measurement bias. ZEPO offers a method for testing and adjusting that part of the workflow, while the paper leaves open whether a balanced preference distribution transfers to new domains or reflects every human constituency. Broader evaluation could examine both questions.

Bibliography & sources

1. Han Zhou, Xingchen Wan, Yinhong Liu, Nigel Collier, Ivan Vulić, and Anna Korhonen. “Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 1241–1252 (2024). https://doi.org/10.18653/v1/2024.emnlp-main.72.