LitSearch: Benchmarking AI retrieval of scientific literature

A researcher rarely searches for a paper by exact title. More often, the question describes an idea: Which studies test whether generated summaries stay consistent? LitSearch evaluates how well retrieval systems answer that kind of literature-search question. The benchmark measures a practical skill that standard keyword searches and generic question-answering tests do not fully capture: finding relevant work from a natural-language description.

A benchmark with two kinds of questions

LitSearch contains 597 questions about recent machine-learning and natural-language-processing research. Some questions were generated by GPT-4 from paragraphs with inline citations in papers. Others were written by authors about their own recently published papers. Experts manually examined or edited the questions to improve their quality. The search corpus contains 64,183 papers from the ACL Anthology and ICLR.

The authors tested BM25, a keyword-based method, alongside dense retrieval models that encode questions and papers into vectors. They also evaluated GPT-4o reranking, where a language model reorders candidate papers after retrieval. A “one-hop” variant follows citations from initially retrieved papers to build another candidate list. Results are reported separately for broad questions, which can have many relevant papers, and specific questions with fewer relevant targets.

Dense retrieval leads, with reranking gains

GritLM-7B achieved the strongest base retrieval performance among the tested systems, with 70.8 recall@20 for broad questions and 74.8 recall@5 for specific questions. Recall measures the share of relevant papers recovered near the top of a ranked list. The abstract reports a 24.8 percentage-point advantage in recall@5 over BM25 for the comparison highlighted by the authors. GPT-4o reranking improved the best dense retriever by 4.4% on the reported measure.

The paper also found that the two question sources behaved differently. Author-written questions were easier for retrieval systems than inline-citation questions, which the authors relate to stronger overlap between the authors’ phrasing and paper titles or abstracts. More descriptive questions were not always easier: the authors’ manually rated harder questions generally produced lower retrieval performance.

A surprising result concerns how much text a system sees. Adding full papers to titles and abstracts did not consistently improve retrieval. In some conditions, the added material reduced performance. More context is therefore not automatically useful when models have limited input capacity or were trained on shorter documents.

Boundaries of the evidence

The dataset is English-only and focuses on ML and NLP papers. Some questions may be easier than intended because their wording overlaps with the target paper; the authors also note that author-written questions were easier than expected. Their benchmark comparison was not exhaustive and omitted some newer or more complex retrieval and reranking approaches. Performance against the corpus cannot establish how researchers would use the results in an end-to-end literature review.

The difference between inline-citation and author-written questions is especially important. A system may perform well when questions reuse language from abstracts, yet struggle when a researcher asks about a subtle concept in unfamiliar terms. The benchmark includes both sources, offering a more informative picture than a single blended score, but it cannot represent every search need.

Possible implications for practice

For research organizations, LitSearch gives a concrete way to test whether search tools retrieve relevant papers from conceptual questions, not just keywords. The results suggest dense retrieval and reranking can improve candidate discovery, while also showing why output should remain a ranked list for a person to inspect. Better retrieval can save reading time; it does not determine which sources are methodologically sound or whether they answer the researcher’s question.

The study “LitSearch: A Retrieval Benchmark for Scientific Literature Search” was published at EMNLP 2024. Its benchmark and evaluation make the limits of scientific search visible as well as the gains from stronger retrieval models.

Bibliography & sources

1. Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, Tianyu Gao. 2024. LitSearch: A Retrieval Benchmark for Scientific Literature Search. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15068–15083. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.840.