Fractured Chain-of-Thought Reasoning
Fractured Sampling method reduces token usage while improving Pass@k accuracy.
Key Findings
Methodology
Fractured Sampling interpolates between full CoT and solution-only sampling along three axes: number of reasoning trajectories, number of final solutions per trajectory, depth of reasoning trace truncation. It excels across five reasoning benchmarks.
Key Results
- On the MATH500 L5 benchmark, truncated CoT sampling achieved the same Pass@1 accuracy as full CoT sampling with 30% fewer tokens.
- On the AIME24 benchmark, Fractured Sampling improved Pass@k accuracy by approximately 15% under the same token budget.
- On the GPQA benchmark, Fractured Sampling significantly enhanced diversity, reducing error mode overlap.
Significance
This research optimizes computational resource allocation during inference, significantly enhancing LLM reasoning efficiency and addressing the cost bottleneck of long CoT methods.
Technical Contribution
Introduced a novel inference-time sampling framework that reduces token usage without sacrificing performance, offering a new computational allocation strategy.
Novelty
First to propose Fractured Sampling during inference, improving efficiency by truncating reasoning trajectories, significantly reducing computational cost compared to traditional full CoT methods.
Limitations
- In some tasks, truncated reasoning trajectories may lead to information loss, affecting final answer accuracy.
- Fractured Sampling's performance under extremely low token budgets needs further validation.
Future Work
Future research could explore optimizing Fractured Sampling parameters across different model scales and task types to further enhance efficiency.
AI Executive Summary
Chain-of-Thought reasoning techniques have significantly advanced the reasoning capabilities of large language models, yet their high token cost limits application in latency-sensitive environments. The Fractured Sampling method proposed in this paper drastically reduces token usage by truncating reasoning trajectories while maintaining high accuracy.
Through extensive experiments across five diverse reasoning benchmarks, the Fractured Sampling method demonstrates superior accuracy-cost trade-offs, particularly achieving significant improvements in Pass@k metrics. This method optimizes computational resource allocation by interpolating along three axes during inference: number of reasoning trajectories, number of final solutions per trajectory, and depth of reasoning trace truncation.
Fractured Sampling paves the way for more efficient and scalable LLM reasoning with broad application potential. However, the method's information loss in certain tasks requires further study to ensure stability and reliability across various application scenarios.
Deep Analysis
Background
Recent advances in large language models have enabled impressive capabilities in complex reasoning and problem-solving. Chain-of-Thought (CoT) prompting techniques improve accuracy by generating rich intermediate reasoning trajectories, but incur substantial token costs.
Core Problem
Full CoT methods require a large number of tokens, leading to high computational costs, making them unsuitable for latency-sensitive or resource-constrained environments.
Innovation
Fractured Sampling reduces token usage by truncating reasoning trajectories while maintaining high accuracy. It interpolates along three axes during inference: number of reasoning trajectories, number of final solutions per trajectory, depth of reasoning trace truncation.
Methodology
- �� Number of reasoning trajectories: sample multiple independent reasoning paths.
- �� Solution diversity: generate multiple final solutions per trajectory.
- �� Reasoning prefix length: truncate each reasoning trajectory at different depths.
Experiments
Experiments conducted across five diverse reasoning benchmarks: MATH500 L5, AIME24, AIME25, AIMO2, and GPQA. Set n=16, H=16, m=4 to evaluate Fractured Sampling's accuracy and token usage.
Results
Fractured Sampling consistently excels across all benchmarks, particularly achieving significant improvements in Pass@k metrics. Truncated reasoning trajectories significantly reduce token usage while maintaining high accuracy.
Applications
Fractured Sampling is suitable for efficient reasoning in large language models, particularly in latency-sensitive or resource-constrained environments.
Limitations & Outlook
In some tasks, truncated reasoning trajectories may lead to information loss, affecting final answer accuracy. Future research needs to further optimize parameter settings.
Plain Language Accessible to non-experts
Imagine cooking in a kitchen, where full Chain-of-Thought is like following a recipe step by step. Fractured Sampling is like preparing all ingredients in advance and choosing the right steps as needed. This saves time and resources while ensuring the dish is delicious.
ELI14 Explained like you're 14
Hey there, kiddo! Imagine you're playing a game where you usually need to complete tasks step by step to win. But sometimes, you can skip some steps and get rewards directly! That's the magic of Fractured Sampling, making large language models faster and more efficient while keeping high scores!
Glossary
Fractured Sampling
An inference-time sampling method that reduces token usage by truncating reasoning trajectories.
Used in this paper to optimize computational resource allocation during inference.
Chain-of-Thought
A prompting technique that improves model accuracy by generating intermediate reasoning steps.
Used to generate rich intermediate reasoning trajectories.
Pass@k
Estimates the probability that at least one sample is correct.
Used to measure the accuracy of Fractured Sampling.
Token Budget
The maximum number of tokens used for inference.
Limits computational cost during inference.
Inference-time Scaling
Enhances performance by increasing computational effort during inference.
Improves model performance without altering parameters.
Open Questions Unanswered questions from this research
- 1 How to optimize Fractured Sampling performance under extremely low token budgets remains to be explored.
- 2 Truncated reasoning trajectories may lead to information loss, requiring exploration of mitigation strategies.
Applications
Immediate Applications
Real-time reasoning
Applicable in scenarios requiring quick responses, such as instant translation and voice assistants.
Long-term Vision
Large-scale reasoning
Efficient reasoning on large datasets, advancing AI development further.
Abstract
Inference-time scaling techniques have significantly bolstered the reasoning capabilities of large language models (LLMs) by harnessing additional computational effort at inference without retraining. Similarly, Chain-of-Thought (CoT) prompting and its extension, Long CoT, improve accuracy by generating rich intermediate reasoning trajectories, but these approaches incur substantial token costs that impede their deployment in latency-sensitive settings. In this work, we first show that truncated CoT, which stops reasoning before completion and directly generates the final answer, often matches the full CoT sampling while using dramatically fewer tokens. Building on this insight, we introduce Fractured Sampling, a unified inference-time strategy that interpolates between full CoT and solution-only sampling along three orthogonal axes: (1) the number of reasoning trajectories, (2) the number of final solutions per trajectory, and (3) the depth at which reasoning traces are truncated. Through extensive experiments on five diverse reasoning benchmarks and several model scales, we demonstrate that Fractured Sampling consistently achieves superior accuracy-cost trade-offs, yielding steep log-linear scaling gains in Pass@k versus token budget. Our analysis reveals how to allocate computation across these dimensions to maximize performance, paving the way for more efficient and scalable LLM reasoning. Code is available at https://github.com/BaohaoLiao/frac-cot.