Funnel of Thoughts: Efficient Test-Time Scaling via Early Voting and Rollout Pruning

Over the past few years, large-scale reasoning models like GPT-4, PaLM, and others have achieved remarkable success in complex language understanding and reasoning tasks. Early efforts focused on improving accuracy through multi-sample inference techniques such as Self-Consistency (SC@k), which aggregates answers from multiple trajectories to boost correctness. However, these methods are computationally intensive, as each trajectory involves full-length generation, leading to quadratic growth in attention FLOPs. Researchers have identified that a significant portion of generated tokens—up to 50%—are redundant or unproductive, especially in long, spiraling, or repetitive paths. This inefficiency limits the scalability and deployment of large models in real-world applications, prompting the need for more resource-aware inference strategies.

Despite the success of multi-sample inference, the high computational cost remains a critical bottleneck. Long trajectories with unproductive behaviors—such as hesitation, repetition, or self-correction—consume disproportionate attention FLOPs without contributing to answer accuracy. Existing solutions like external confidence models or similarity-based pruning add complexity or require additional training, reducing practicality. The core challenge is to identify and eliminate these wasteful paths dynamically during inference, without retraining or external modules, while preserving the diversity and accuracy benefits of multi-sample approaches. Achieving this balance is vital for making large reasoning models more efficient and deployable at scale.

FoT introduces a novel, training-free lexical signal—hesitation markers embedded naturally in generated text—as an indicator of unproductive reasoning paths. It employs a multi-stage, checkpoint-based framework where at each checkpoint, it performs early voting on committed answers, and computes hesitation density for ongoing trajectories. Trajectories exhibiting high hesitation are pruned, significantly reducing attention FLOPs. This approach differs from prior methods by directly leveraging language cues without external models or similarity metrics, enabling real-time, in-flight pruning. Its simplicity, model-agnostic design, and strong transferability across architectures and tasks represent key innovations, providing a practical solution to the resource bottleneck in multi-sample inference.

• �� Initialize a pool of k parallel trajectories during inference.
• �� At multiple checkpoints (t1, t2, ..., tm), perform early voting on trajectories that have committed answers, storing results in a vote bank.
• �� For remaining trajectories, calculate hesitation marker density over generated tokens.
• �� Rank trajectories by hesitation density; retain the lowest-density subset based on a fixed keep ratio (e.g., 80%).
• �� Discard high-density trajectories, which are likely spiraling or unproductive.
• �� Continue generation only on retained trajectories, repeating the process at subsequent checkpoints.
• �� Final answer is determined by majority vote among committed trajectories.
• �� This process is fully parallel, requires no training, and is driven solely by text analysis, making it efficient and easy to implement.

The evaluation involved six diverse models (DeepSeek, Qwen, Skywork, Phi-4) across four math reasoning benchmarks (AIME24/25, AMC23, MATH500). Each problem generated 32 independent trajectories, totaling 115K paths. Baselines included SC@32, Adaptive Consistency, and Slim-SC. Hyperparameters such as checkpoint positions and pruning ratios were optimized via grid search. The experiments measured accuracy, attention FLOPs, and wall-clock time, assessing the tradeoff between resource savings and correctness. Cross-model and cross-task transfer tests validated the robustness and generality of FoT, with additional ablation studies confirming the importance of hesitation signals.

FoT achieves an average of 28.8% reduction in attention FLOPs and 37.6% in inference wall time across models and datasets, while maintaining SC@32 accuracy. It effectively filters out long, spiraling, no-answer trajectories characterized by high hesitation marker density, leading to resource savings without accuracy loss. On difficult problems, FoT outperforms similarity-based pruning methods by better preserving reasoning diversity. Transfer experiments show consistent performance across different architectures and tasks, including code generation, demonstrating broad applicability. Ablation results highlight hesitation density as a critical pruning signal, outperforming alternative metrics.

FoT can be integrated into any large reasoning system requiring multi-sample inference, such as automated question answering, reasoning assistants, and code synthesis. Its lightweight, inference-only design allows deployment in resource-constrained environments, reducing costs and latency. The method enables more scalable and energy-efficient AI services, facilitating real-time applications in industry and research. Future extensions could incorporate semantic or multimodal signals, broadening its scope to visual reasoning or multi-turn dialogues, further enhancing large model deployment.

FoT relies on the assumption that hesitation markers correlate with unproductive paths, which may vary across languages, tasks, or models. In scenarios with subtle or less explicit hesitation cues, pruning effectiveness diminishes. There is a risk of mistakenly pruning correct but hesitant trajectories, slightly impacting accuracy in ambiguous cases. Additionally, the method introduces some computational overhead for text analysis and checkpoint management, which could be optimized further. Future work should address these limitations by integrating semantic signals and adaptive thresholds to improve robustness across diverse settings.