Science and research
SciAgent: Tool-augmented Language Models for Scientific Reasoning
Tool-use training improved language models’ answers to scientific questions. Many baseline models became less accurate when given external tools.

Scientific questions often require more than recalling facts. A model may need to calculate, retrieve a relevant function or combine knowledge from a field with a mathematical method. “SciAgent: Tool-augmented Language Models for Scientific Reasoning” proposes a way to train models to use general-purpose tools instead of expecting them to solve every task from memory.
Train the model to work with tools
The researchers built MATHFUNC, a training corpus with more than 31,000 tool-augmented examples and nearly 6,000 functions. They used this material to train SCIAGENT, an agent that plans a question, retrieves candidate functions, produces an action and executes code when needed. The aim is to make tool use a learned capability that can transfer across scientific subjects.
To test the idea, the team created SCITOOLBENCH, a benchmark with 4,250 questions and 2,285 functions across five areas: mathematics, physics, chemistry, finance and electrical engineering/computer science. Questions combine human-annotated examples with automatically synthesized ones. The benchmark includes positive functions that can answer a question and negative functions that look related but do not directly solve it. This design tests whether a model can select a useful tool rather than match a question to a convenient function by surface similarity.
Tool use depends on training
The authors found that many baseline models lost accuracy when supplied with an external toolset. Simply attaching a list of functions did not make a model proficient at using them. Models fine-tuned on tool-augmented examples were more able to benefit from retrieved tools. SCIAGENT-LLAMA3-8B exceeded comparable-size open models by more than eight percentage points in absolute accuracy on the benchmark. SCIAGENT-DEEPMATH-7B reached 40.0% overall accuracy, compared with 31.0% for ChatGPT in the reported comparison.
Ablation tests help explain the result. Removing planning weakened function retrieval, and replacing cross-retrieval with direct tool selection reduced performance. The training mix also mattered: the paper reports drops when function-augmented or function-free examples were removed. When the team artificially increased the chance that a useful function appeared among the top retrieved tools, accuracy rose. Even with the correct function made available, however, the system reached only about 40% accuracy in one analysis. Correct tool access does not guarantee correct scientific reasoning.
Limits and interpretation
The authors recognize possible information leakage because some benchmark functions were derived directly from the questions. They used human review and generalized functions to reduce the risk, but it remains a design concern. The quality of the 3,394 synthesized questions also needs further study. Model performance stayed well below GPT-4 in the reported comparison, by more than 20 percentage points in some settings.
SCITOOLBENCH spans several domains, but the results are tied to its question mix, tools and evaluation setup. A tool can be correct yet not solve the question, and a model can make mistakes in the code or in interpreting its output. A benchmark score should not be taken as evidence that a deployed agent can safely perform scientific work without oversight.
Possible implications for practice
For scientific organizations, the paper suggests that tool use may be more reliable when it is part of training and evaluation, rather than an interface added after the fact. Its results encourage a more precise question: can a model retrieve a relevant tool, use it correctly and interpret the result? Those capabilities need separate evaluation. The remaining accuracy gap also points to the value of designing systems that make their sources, functions and calculations inspectable by human collaborators.
“SCIAGENT: Tool-augmented Language Models for Scientific Reasoning” appeared in the 2024 EMNLP proceedings. It offers evidence for tool-augmented training, alongside clear caveats about benchmark construction and remaining reasoning errors.
Bibliography & sources
- Yubo Ma, Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang, Liangming Pan, Yujiu Yang, Yixin Cao, Aixin Sun. 2024. SciAgent: Tool-augmented Language Models for Scientific Reasoning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15701–15736. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.880.
