AI and society
Culture Needs More Than One Language-model Score
A survey of 90 papers finds that culture is usually measured through proxies, leaving important contexts and lived experience outside the test.
What does it mean for a large language model (LLM) to understand culture? A survey of 90 studies by Muhammad Farid Adilazuarda and colleagues finds that most papers did not explicitly define culture. Instead, they tested selected features—such as values, language, region or food—as proxies for parts of cultural life.
The measurement is the definition in practice
The authors searched the ACL Anthology, Google Scholar, NeurIPS and The Web Conference for work published from 2020 to 2024. They manually coded how studies represented culture, which probing methods they used and which populations or languages they examined. None of the 90 papers in their final set explicitly defined culture, so the authors built a taxonomy around proxies present in the datasets (Adilazuarda et al., 2024, pp. 15765–15766).
They divide those proxies into two broad groups. Demographic proxies include attributes such as region, language, gender or ethnicity. Semantic proxies capture subject areas and meanings, including values, norms, food or other culturally shaped topics. The categories help compare studies, but a model’s score on one proxy cannot stand in for cultural understanding as a whole (pp. 15766–15769).
The paper also groups probing methods. Most studies use black-box tests: they present questions or prompts and inspect the output. These tests can reveal differences, but results depend on examples, wording and definitions chosen by researchers. The survey calls for stronger robustness and more interpretable work that can explain why a model responded as it did (pp. 15769–15771).
What the current evidence leaves out
The review finds a concentration on some aspects, such as values and objectives, while many semantic domains and “aboutness”—what topics a community treats as relevant—receive less attention. It points to English-centric datasets and a shortage of situated studies examining how cultural misrepresentation affects people in actual applications (pp. 15763, 15770–15771).
A central challenge is that culture is contextual and layered. People belong to multiple communities, and which identity matters can depend on a particular interaction. Text corpora can capture visible patterns while missing the history and lived experience that give them meaning. The authors draw on “thick description”: understanding behavior also requires understanding how people explain the setting and choices around it (pp. 15764, 15771).
This survey focuses on LLM probing. It excludes much work in human-computer interaction, development, speech and multimodal systems, and it does not review cultural modeling or mitigation techniques (p. 15771). Its taxonomy clarifies existing research but cannot establish how every community experiences a deployed system.
Possible implications for practice
For organizations deploying assistants across regions, a single “cultural competence” score could hide what was tested. A product might perform well on translated questions while missing local assumptions, conversational norms or topics that matter to users. Evaluation results are more informative when teams say which cultural dimensions their data represent and whose perspectives informed the examples.
The paper suggests a broader question for product research: how do model outputs work in a real social setting, and how do people interpret them? That inquiry may require collaboration with local researchers and users, not just a larger prompt set. The survey does not provide a universal definition of culture; it shows why making each study’s scope explicit is a necessary starting point.
That distinction matters for organizations because product choices can convert a narrow test into a broad marketing claim. A model evaluated on a few cultural proxies may pass that test while remaining unfamiliar with local issues that affect support, education or public services. Clear descriptions of what was measured would help teams and users interpret the limits of a benchmark without treating one score as a substitute for local experience.
Bibliography & sources
- Muhammad Farid Adilazuarda; Sagnik Mukherjee; Pradhyumna Lavania; Siddhant Singh; Alham Fikri Aji; Jacki O’Neill; Ashutosh Modi; Monojit Choudhury. 2024. “Towards Measuring and Modeling “Culture” in LLMs: A Survey.” In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, 15763–15784. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.882.