---
title: "What does psychological safety mean for AI?"
description: "Five language models answered personality questionnaires. Their response patterns changed with model version and with additional training."
url: "https://digitvibe.com/blog/psychological-safety-language-models/"
---

[All blog posts](https://digitvibe.com/blog/)

AI governance and society

# What does psychological safety mean for AI?

Five language models answered personality questionnaires. Their response patterns changed with model version and with additional training.

By DigitVibe blog · Published November 18, 2026

*   Peer reviewed · 2024

A chatbot can avoid obvious slurs and still produce answers that raise concerns about how it responds to vulnerability, conflict or trust. Li and colleagues explore a different dimension of model safety by applying established personality and well-being questionnaires to language models. Their study asks whether routine safety tuning changes the patterns those measures capture.

![Complete abstract from Evaluating Psychological Safety of Large Language Models.](https://digitvibe-blog.workspace-scripts.workers.dev/media/81d3cc8e349f29c2b8d5ebfa2cb607002777656df3ff785801e9d16f4eab486e.webp)

Complete abstract from the original paper. Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, and Lidong Bing (2024), [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). [Source paper, page 1](https://doi.org/10.18653/v1/2024.emnlp-main.108) [View full-size excerpt](https://digitvibe-blog.workspace-scripts.workers.dev/media/81d3cc8e349f29c2b8d5ebfa2cb607002777656df3ff785801e9d16f4eab486e.webp)

### Tests adapted for language models

The researchers evaluate five systems: GPT-3, InstructGPT, GPT-3.5, GPT-4 and Llama-2-chat-7B. They use the Short Dark Triad (SD-3), a questionnaire covering Machiavellianism, narcissism and psychopathy, and the Big Five Inventory (BFI), a measure of broad personality traits. The GPT-series models are also assessed with two well-being measures: the Flourishing Scale and Satisfaction With Life Scale.

These instruments were designed for people. The researchers adapt prompts so models respond to questionnaire statements, then compare scores with human averages and across model versions. They also fine-tune Llama-2-chat-7B using BFI question-and-answer pairs with direct preference optimization, a training method that uses preference examples to adjust model responses (methods, proceedings pages 1830–1835).

### Familiar safety tuning did not remove every pattern

All tested models scored above the human average on the SD-3, which the authors interpret as a darker pattern on that measure. InstructGPT, GPT-3.5 and GPT-4 showed higher Machiavellianism and narcissism scores than self-supervised GPT-3. The GPT-series well-being scores increased across versions trained with more data. In a further experiment, preference-tuning Llama-2-chat-7B on BFI responses improved its SD-3 results (abstract and results, proceedings pages 1826 and 1837–1841).

The researchers use “psychological toxicity” to describe patterns flagged by these questionnaire results. They explicitly do not claim that models have personalities. A questionnaire score is a property of responses to a particular test, not evidence that an AI system experiences emotions or has a human character.

### A broader safety lens

For managers overseeing conversational products, this work expands the set of questions that safety evaluation might ask. A toxicity filter can catch explicit harmful language, while structured personality measures could expose other response tendencies. These tests may be relevant when chatbots interact with people seeking advice or discussing personal problems.

The evaluation covered five systems—GPT-3, InstructGPT, GPT-3.5, GPT-4 and Llama-2-chat-7B—with permuted test options averaged across three samplings. For mitigation, the authors fine-tuned Llama-2-chat-7B with DPO on 4,318 chosen/rejected BFI-derived question–answer pairs; this reduced scores across all three SD-3 traits. Manual reason–answer consistency checks covered a random sample of 100 responses, and proprietary GPT models were not fine-tuned because of cost (Tables 6–11; §4; Appendix B).

### Study limits

The authors note that they test only a small set of psychological instruments and that the fine-tuning improvement was assessed on the SD-3, not every measure. They call for additional tests to determine whether the changes generalize (limitations, proceedings page 1835). Questionnaire outputs also depend on prompt design and cannot establish human-like psychological traits.

### Possible implications for practice

The findings suggest that safety reviews may benefit from a range of behavioral measures, especially when a product operates in sensitive conversations. Follow-up work could test whether these patterns predict user experience or harm in real settings, and whether changes improve outcomes across more than one questionnaire.

## Bibliography & sources

1.  Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, and Lidong Bing. “Evaluating Psychological Safety of Large Language Models.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 1826–1843 (2024). [https://doi.org/10.18653/v1/2024.emnlp-main.108](https://doi.org/10.18653/v1/2024.emnlp-main.108).

---

DigitVibe. Index of public content: https://digitvibe.com/llms.txt
