Generative AI in organizations and society
LLMs suggest news angles, but lack human creativity
A study of journalism planning finds that language models align more closely with reporters on story angles than on sources, while producing less creative suggestions.
A press release is written to frame a story. Journalists may challenge it, add context and seek voices beyond the organization that issued it. This study examines whether language models can help plan those decisions by comparing model suggestions with the angles and sources human reporters chose.
A large corpus, then a closer human review
The researchers assemble a dataset of roughly 250,000 press releases and 650,000 articles covering them. They identify “effective” coverage as reporting that both entails and contradicts elements of a press release: full agreement may be a simple summary, while full contradiction could be off-topic. A mix of the two signals that an article challenges or contextualizes the original account. Manual evaluation of this contrastive-summarization method reaches an F1 score of .81 (dataset and method, PDF pp. 1–3; proceedings pp. 21814–21816).
On 6,000 article–release pairs, the team studies two planning choices. An angle is the specific perspective a reporter takes; sources are the people or documents that contribute information. More critical coverage is correlated with more creative angles (r = .29) and a greater number of sources (r = .5). These are associations in the dataset, not evidence that creativity alone causes critical reporting (analysis, PDF pp. 4–6; proceedings pp. 21817–21819).
Better at angles than source choices
The language models’ angle recommendations align with human choices at an F1 score of 63.6. Their recommendations for types of sources score 27.9 F1. Human evaluation also finds that the suggested angles and sources are significantly less creative than journalists’ decisions (model evaluation, PDF pp. 7–10; proceedings pp. 21820–21823).
The distinction matters for newsrooms. A model may suggest a familiar perspective that resembles a human reporter’s choice without identifying the independent evidence or people needed to verify a claim. Source recommendations are a separate skill, and the paper finds that current systems are weaker there. The authors call for deeper alignment with human decision-making before creative-planning tools are widely used.
Boundaries of the comparison
The work studies press releases and news articles available in a large corpus, then uses a smaller manually reviewed set for selected analyses. Its measures of creativity and coverage depend on the study’s annotation and modeling choices. The authors discuss limitations and risks around the assumption that selected planning choices represent effective journalism (limitations and risks, PDF pp. 12–13; proceedings pp. 21825–21826).
Possible implications for newsroom tools
For organizations experimenting with AI-assisted reporting, the findings suggest that brainstorming a possible angle and recommending verifiable sources should be evaluated separately. The research offers a method for asking whether a suggestion adds context or merely repeats an announcement, while leaving open how human judgment and newsroom standards should shape any use.
Bibliography & sources
- Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze. “Do LLMs Plan Like Human Writers? Comparing Journalist Coverage of Press Releases with LLMs.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 21814–21828 (2024). https://doi.org/10.18653/v1/2024.emnlp-main.1216.