Science and research
SciDQA: Testing how deeply language models read scientific papers
SciDQA tests language models with 2,937 questions from paper-review exchanges. Providing full papers helped some models answer, while others gained little.

Reading a research paper can require more than finding a sentence that contains an answer. Some questions depend on figures, equations, appendices or a related paper. SciDQA was built to test that deeper form of comprehension with questions drawn from the scientific review process, where domain experts ask questions and paper authors respond.
Questions from real scholarly exchanges
The authors collected reviewer–author discussions associated with papers from ICLR and NeurIPS. A multi-stage process extracted candidate questions and answers, filtered them with expert annotation, rewrote them to make sense outside the original discussion and tracked the version of the paper that supports each answer. The resulting SCIDQA dataset contains 2,937 question–answer pairs. It also includes bibliography information so that questions referring to earlier work can be identified.
The researchers evaluated language models in four settings. In closed-book tests, a model saw only the question. A title-and-abstract condition added limited paper context. Retrieval-augmented generation supplied the top-ranked text chunks, while a full-text condition gave the model the paper or processed segments of it. The study compared open and proprietary models using overlap-based measures and language-model judging of relevance, accuracy, completeness and conciseness.
Context helps, but results need interpretation
GPT-4o and GPT-4o-mini achieved the highest average scores across the tested settings. For one open model, Vicuna 7B, the full-text condition improved the average score by 25 points over the closed-book condition. But the pattern was not uniform: several models did not gain much when the paper was provided. The authors note that strong performance without context could reflect memorized information or shallow answers rather than close reading.
Human comparison was limited to 28 questions. Evaluators judged GPT-4 and author-written answers for comprehensiveness, factuality and clarity. The answers tied in 32% of comparisons; people preferred the human answer in 29%, while GPT-4 was preferred in 21%. The remaining cases were not accepted as satisfactory. This small exercise underscores how difficult it is to score long, specialized answers and why no single automatic metric can establish that a model understood a paper.
Where the benchmark remains incomplete
SCIDQA focuses on machine-learning research, not science as a whole. Some questions require evidence from cited papers, but the experiments did not retrieve those documents. About seven percent of questions depend on figures; the text-processing pipeline extracted captions rather than images, and the evaluation did not include visual or multimodal language models. Potential training-data contamination is also hard to rule out because model training sources are not fully disclosed. The authors call for further study of the evaluation metrics themselves.
These limits matter because the dataset aims to represent deep reading. A benchmark cannot test visual reasoning when the system never sees the figure, or multi-document reasoning when references are absent. The results therefore describe the evaluated text and context configurations, not every capability needed to read a scientific article.
Possible implications for practice
For research teams considering AI support for literature review, SciDQA shows how valuable questions from real scholarly exchanges can be for testing comprehension. Its careful version tracking and expert review offer a model for constructing challenging evaluation data. The results also support a cautious role for assistants: help locate relevant passages and formulate responses, while keeping the underlying paper, figures and cited sources available for human verification.
The paper “SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers” was published in the 2024 EMNLP proceedings. Its benchmark raises a practical standard for scientific reading systems: an answer should be evaluated against the evidence and context it was expected to use.
Bibliography & sources
- Shruti Singh, Nandan Sarkar, Arman Cohan. 2024. SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20908–20923. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.1163.
