AI governance and society
What legal knowledge can language models show?
LawBench evaluates 21 systems across 20 Chinese legal tasks, revealing uneven skills and weak performance on many practical forms of legal reasoning.
A model that can define a legal term may still struggle to apply a statute to a case. Fei and colleagues build LawBench to assess a broader set of legal skills, with tasks designed for the Chinese civil-law system. Their results show why legal ability cannot be summarized by a single exam score.
Twenty tasks across three skill levels
LawBench contains 20 tasks for Chinese legal work and organizes them using three levels inspired by Bloom’s taxonomy: memorization, understanding and application. Memorization covers recalling legal knowledge. Understanding includes work such as interpreting or extracting information from legal texts. Application asks a model to use rules and facts to solve a legal task. The benchmark includes classification, extraction and generation tasks, drawing on existing legal datasets.
The authors evaluate 21 popular language models and compare performance across tasks. Some responses are scored with accuracy measures; generated text is assessed with ROUGE-L, an overlap metric that compares words and sequences with a reference answer. The benchmark and model predictions are publicly available (design, proceedings pages 7937–7943; evaluation and results, pages 7947–7955).
A fragmented picture of legal capability
The researchers find that current models often perform poorly on many tasks and do not yet provide meaningful judicial assistance across the tested areas. Fine-tuning open models on legal data improves some results, but those systems remain behind GPT-4 in the paper’s comparisons. The benchmark exposes uneven strengths and weaknesses rather than a single general level of “legal intelligence” (abstract, proceedings page 7933; conclusion, pages 7944–7945).
An important boundary is jurisdiction. LawBench focuses on Chinese law and its civil-law tradition. Results cannot be assumed to describe performance under another country’s statutes, court practices or common-law reasoning.
Why task-level evaluation matters
For law firms, public agencies and companies handling compliance, the study illustrates why a model’s performance on one legal task may not predict its performance elsewhere. A tool that retrieves a provision or summarizes a document still needs distinct evaluation before it is trusted to classify disputes or apply rules to facts. The benchmark gives researchers a shared way to compare those abilities.
The benchmark covers 20 tasks and 21 language models, split across memorization, understanding and application. Qwen-1.5-72B-Chat and GPT-4 led overall; Chinese-oriented systems generally outperformed multilingual models, while legal-specialist models often trailed general-purpose Chinese models. GPT-4 was strong at understanding and application but less reliable at exact statute citation. The authors flag possible training-data leakage and caution that answer extraction rules can fail; ROUGE-L may not match human judgments, so results are a lower-bound estimate (Tables 6 and 29–36; Appendices A–M).
Study limits
The authors warn that benchmark contamination is possible because many datasets are publicly available online. Models may have seen test material during training. They also note that rule-based answer extraction can fail and that ROUGE-L may not match human judgments of answer quality for generative tasks (limitations, proceedings pages 7955–7956). Results therefore reflect both model performance and measurement choices.
Possible implications for practice
LawBench suggests a useful direction for governance: evaluate models by the kinds of legal work they are expected to support, within the right jurisdiction. The findings could inform more targeted studies of specific tasks, but they do not establish readiness for legal advice or independent judicial decisions. Human review and domain-specific evidence remain central questions.
Bibliography & sources
- Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. “LawBench: Benchmarking Legal Knowledge of Large Language Models.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 7933–7962 (2024). https://doi.org/10.18653/v1/2024.emnlp-main.452.