Generative AI in organizations and society
More compute can make simple LLM reasoning competitive
A budget-aware study compares reasoning strategies at matched compute and finds that extra tokens can explain gains credited to complex methods.
When a language model uses more tokens, it may solve more problems simply because it had more chances to think or sample answers. A comparison that ignores this cost can make a complex reasoning method look more efficient than it is. This paper proposes testing strategies at comparable inference budgets.

Measuring what a strategy spends
The researchers describe a budget-aware evaluation that tracks queries, generated tokens and monetary cost. They favor token count as a broad measure because it captures both how much computation a method uses and some of its practical implications. The study compares seven reasoning strategies across five datasets and five language models, including GPT-4 (framework and study design, PDF pp. 1–3; proceedings pp. 19916–19918).
One baseline is chain-of-thought self-consistency. A model generates several independent reasoning-and-answer samples, then selects the answer that appears most often. The paper compares this method with approaches such as multi-agent debate, Reflexion, Plan-and-Solve, Tree of Thoughts and hint-based prompting. Matching the compute budget helps distinguish the value of a strategy from the benefit of spending more on it (methods, PDF pp. 4–8; proceedings pp. 19919–19923).
Extra computation changes the ranking
Across several comparisons, self-consistency performs strongly when given a budget comparable to more elaborate strategies. The authors report that it frequently matches or outperforms these methods at equal compute. In scale-aware tests, some approaches—such as multi-agent debate or Reflexion—can perform worse as additional compute is allocated, while self-consistency benefits from independent samples (results and analysis, figures and tables, PDF pp. 9–15; proceedings pp. 19924–19930).
The paper also separates answer-generation budget from evaluation budget. Self-evaluation sometimes helps, but its success depends on the model, task and calibration: a model’s confidence needs to relate meaningfully to correctness. The results therefore do not imply that every simple method is best, or that more computation always makes results worse. They show that rankings can change when the accounting changes.
One measure cannot capture every deployment cost
Token count is a useful shared yardstick, but real systems also face latency, price, reliability and task-specific constraints. The authors discuss limitations in the models, datasets and budget assumptions behind the experiments (limitations, PDF pp. 19–20; proceedings pp. 19934–19935). Results on these benchmarks do not automatically transfer to every product workflow.
Possible implications for model evaluation
For teams comparing AI reasoning features, reporting performance alongside token use can make trade-offs easier to see. A budget-matched baseline may reveal whether a proposed method adds algorithmic value or mainly buys more attempts. Further work could test those comparisons under the timing and reliability requirements of deployed systems.
Bibliography & sources
- Junlin Wang, Siddhartha Jain, Ben Athiwaratkun, Dejiao Zhang, Baishakhi Ray, and Varun Kumar. “Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 19916–19939 (2024). https://doi.org/10.18653/v1/2024.emnlp-main.1112.
