MENU
Skip to main content
All blog posts

Generative AI in organizations and society

More compute can make simple LLM reasoning competitive

A budget-aware study compares reasoning strategies at matched compute and finds that extra tokens can explain gains credited to complex methods.

  • Peer reviewed · 2024

When a language model uses more tokens, it may solve more problems simply because it had more chances to think or sample answers. A comparison that ignores this cost can make a complex reasoning method look more efficient than it is. This paper proposes testing strategies at comparable inference budgets.

Complete abstract from Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies.
Complete abstract from the original paper. Junlin Wang, Siddhartha Jain, Ben Athiwaratkun, Dejiao Zhang, Baishakhi Ray, Varun Kumar (2024), CC BY 4.0. Source paper, page 1 View full-size excerpt

Measuring what a strategy spends

The researchers describe a budget-aware evaluation that tracks queries, generated tokens and monetary cost. They favor token count as a broad measure because it captures both how much computation a method uses and some of its practical implications. The study compares seven reasoning strategies across five datasets and five language models, including GPT-4 (framework and study design, PDF pp. 1–3; proceedings pp. 19916–19918).

One baseline is chain-of-thought self-consistency. A model generates several independent reasoning-and-answer samples, then selects the answer that appears most often. The paper compares this method with approaches such as multi-agent debate, Reflexion, Plan-and-Solve, Tree of Thoughts and hint-based prompting. Matching the compute budget helps distinguish the value of a strategy from the benefit of spending more on it (methods, PDF pp. 4–8; proceedings pp. 19919–19923).

Extra computation changes the ranking

Across several comparisons, self-consistency performs strongly when given a budget comparable to more elaborate strategies. The authors report that it frequently matches or outperforms these methods at equal compute. In scale-aware tests, some approaches—such as multi-agent debate or Reflexion—can perform worse as additional compute is allocated, while self-consistency benefits from independent samples (results and analysis, figures and tables, PDF pp. 9–15; proceedings pp. 19924–19930).

The paper also separates answer-generation budget from evaluation budget. Self-evaluation sometimes helps, but its success depends on the model, task and calibration: a model’s confidence needs to relate meaningfully to correctness. The results therefore do not imply that every simple method is best, or that more computation always makes results worse. They show that rankings can change when the accounting changes.

One measure cannot capture every deployment cost

Token count is a useful shared yardstick, but real systems also face latency, price, reliability and task-specific constraints. The authors discuss limitations in the models, datasets and budget assumptions behind the experiments (limitations, PDF pp. 19–20; proceedings pp. 19934–19935). Results on these benchmarks do not automatically transfer to every product workflow.

Possible implications for model evaluation

For teams comparing AI reasoning features, reporting performance alongside token use can make trade-offs easier to see. A budget-matched baseline may reveal whether a proposed method adds algorithmic value or mainly buys more attempts. Further work could test those comparisons under the timing and reliability requirements of deployed systems.

Bibliography & sources

  1. Junlin Wang, Siddhartha Jain, Ben Athiwaratkun, Dejiao Zhang, Baishakhi Ray, and Varun Kumar. “Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 19916–19939 (2024). https://doi.org/10.18653/v1/2024.emnlp-main.1112.