Exploration-Driven Optimization for Test-Time Large Language Model Reasoning

Recent advancements in large language models (LLMs) such as GPT-4 and PaLM have demonstrated impressive reasoning and language understanding capabilities. Post-training techniques like reinforcement learning from human feedback (RLHF) and preference optimization (DPO, SPO) have further enhanced alignment and reasoning. However, these methods tend to produce overly sharp output distributions, limiting diversity and exploration. Test-time scaling strategies like self-consistency and majority voting improve performance but are constrained by the training distribution's sharpness. The core challenge remains: how to balance exploration and exploitation during training to enable models to generate diverse, high-quality solutions, especially for complex reasoning tasks and out-of-distribution scenarios.

The main issue is the distribution sharpness induced by RL-based post-training, which reduces the model’s ability to explore alternative solutions. This leads to overfitting to high-reward modes, decreased solution diversity, and poor generalization to novel tasks. Existing methods either regularize the distribution or rely on external inference techniques, but they do not fundamentally address the exploration-exploitation trade-off during training. Consequently, models struggle with complex reasoning, and their performance on distributional shifts remains limited. Developing a training paradigm that inherently encourages exploration without sacrificing alignment is critical for advancing LLM capabilities.

The key innovations include: 1) Introducing reward biasing as an exploration regularizer that encourages policies to diverge from previous solutions, effectively flattening the output distribution. 2) Deriving a closed-form solution for the optimal policy under this bias, grounded in RL theory, ensuring theoretical soundness. 3) Integrating the mechanism into iterative RL algorithms like iDPO and GRPO, creating ED-iDPO and ED-GRPO variants that promote diversity while maintaining stability. 4) Demonstrating that the approach stabilizes training, prevents collapse, and enhances reasoning performance, especially when combined with inference-time scaling techniques. This represents a significant step beyond traditional regularization and preference optimization methods.

• �� Model test-time inference as an implicit Q-function, where candidate responses are sampled and evaluated. • Introduce a reward biasing term into the RL objective, encouraging policies to explore responses beyond current modes. • Impose a normalization constraint on the reward function to remove ambiguity, leading to a closed-form solution involving inverse KL divergence. • Derive the optimal policy update that maximizes the biased RL objective, promoting divergence from previous policies. • Embed this mechanism into iterative algorithms (iDPO, GRPO), adjusting the policy with exploration incentives. • During training, generate candidate solutions, compute rewards based on correctness, and update policies using the biased objective. • Combine with inference techniques like self-consistency to further improve performance.

The authors evaluate on reasoning benchmarks including GSM8K, ARC, and CommonsenseQA, comparing ED-iDPO and ED-GRPO against standard RLHF, DPO, and SPO baselines. Metrics include accuracy, entropy, and out-of-distribution performance. Hyperparameters such as bias coefficient and sampling size are tuned via ablation studies. The experiments analyze the impact of reward biasing on diversity, stability, and reasoning accuracy, demonstrating consistent improvements. Additional tests combine EDO with self-consistency, showing synergistic gains. The training stability is assessed through convergence curves and entropy measures, confirming that EDO prevents collapse and maintains exploration over iterations.

EDO variants outperform baselines with 1.0-1.3% higher accuracy on in-distribution benchmarks and 1.5% gains on out-of-distribution tasks. Diversity metrics like entropy increase significantly, indicating broader solution spaces. Combining EDO with self-consistency leads to further accuracy improvements, especially on complex reasoning problems. Ablation results show that reward biasing and inverse KL regularization jointly contribute to exploration and stability. The models also demonstrate robustness across different model sizes and datasets, confirming the method’s general applicability.

This approach is suitable for applications requiring robust reasoning, such as automated tutoring, scientific research assistants, and decision support systems. It enables models to generate diverse, high-quality solutions, especially in uncertain or novel scenarios. The framework can be integrated into existing training pipelines and combined with inference-time scaling, making it practical for deployment in real-world AI systems. Long-term, it can facilitate autonomous AI systems capable of continuous self-improvement and creative problem-solving across domains.

The method relies on accurate reward models; errors can limit exploration quality. Additional hyperparameters increase tuning complexity. Computational costs rise due to iterative sampling and training. In extremely high-dimensional tasks, exploration may still be insufficient. Future work should focus on adaptive biasing, more efficient sampling, and better reward estimation to address these limitations and scale the approach to larger models and more complex tasks.