Boosting LLM Exploration via Weak-Model Guidance in RLVR

The evolution of large-scale pretraining has led to the widespread adoption of reinforcement learning techniques like RLHF and RLVR to refine reasoning abilities. Notable works such as DeepSeek, GPT-5, and Gemini have demonstrated significant performance gains in mathematical and logical reasoning tasks. However, these methods often suffer from early policy overconfidence, resulting in reduced exploration and diversity. Existing regularization strategies, including entropy regularization and reward calibration, have only partially alleviated these issues. As models scale, the exploration-exploitation trade-off becomes more critical, necessitating novel mechanisms to sustain diversity. Recent research indicates that leveraging multi-model generation diversity could serve as a promising direction, but systematic integration into RLVR remains underexplored.

Despite advances, RLVR training often leads to rapid policy entropy collapse, causing models to overfit to a narrow set of reasoning paths. This limits the coverage of alternative solutions, especially in high-k pass metrics, undermining the robustness and generalization of large models. The core challenge is how to maintain sufficient exploration during training, preventing premature convergence while ensuring high-quality reasoning outputs. Traditional regularization methods struggle to balance exploration and solution fidelity, highlighting the need for more effective, scalable strategies that can dynamically perturb the policy distribution without complex reward engineering.

The key innovation is the introduction of cross-model generated prefixes as external, non-parametric perturbations to guide RLVR training. This approach exploits the distributional differences among models—smaller, weaker models produce diverse, less confident reasoning trajectories that, when used as prefixes, perturb the target model’s policy, promoting exploration. An entropy-based automatic truncation mechanism dynamically identifies the optimal prefix length, maximizing exploration signals while preserving reasoning quality. This framework diverges from prior work by leveraging model heterogeneity rather than internal regularization, offering a scalable, effective way to combat early policy collapse and expand reasoning coverage.

• �� Base RLVR framework using GRPO, sampling multiple responses per prompt and computing advantage estimates via group normalization.
• �� Auxiliary weak models generate partial reasoning trajectories, which are truncated into prefixes based on entropy dynamics.
• �� These prefixes are concatenated with the original prompt, and the target model is trained to complete the remaining reasoning steps, encouraging exploration of unfamiliar states.
• �� An entropy-based truncation point is identified by measuring the largest drop in policy entropy between consecutive reasoning steps, ensuring the prefix maximizes exploration potential.
• �� A mixed training strategy combines standard question-only RLVR with prefix completion, with a probability p (e.g., 0.2), balancing exploration and solution accuracy.
• �� During training, multiple auxiliary models generate diverse prefixes, which are integrated into the RLVR pipeline, and the model updates are guided by group advantage estimates and KL regularization.

• �� The experiments utilize the MATH dataset (7500 problems) and multiple reasoning benchmarks like AIME 2024/2025, Olympiad, and Minerva, evaluating pass@k metrics.
• �� Baselines include vanilla RLVR (GRPO) and variants with external prefixes from models like Gemma-2-2B, LLaMA-3.2-1B, and Qwen models.
• �� Hyperparameters include a prefix injection probability p=0.2, learning rate 10^-6, batch size 1024, and 8 responses per prompt.
• �� Ablation studies examine the impact of entropy truncation, prefix quality, and auxiliary model choice.
• �� Results show consistent improvements in pass@k, especially at large k, confirming enhanced exploration and coverage.

• �� Incorporating external prefixes from auxiliary models consistently improves pass@k across all benchmarks, with the most notable gains at higher k values. For example, on Qwen2.5-7B, pass@128 increased from 67.76% to 70.71%. Similar trends are observed on other models and datasets.
• �� Training dynamics reveal that prefix-guided models maintain higher policy entropy early in training, delaying premature convergence and fostering broader exploration.
• �� Even low-quality prefixes, often generated by weaker models, effectively stimulate exploration, validating the non-parametric perturbation hypothesis.

• �� The approach can be applied to enhance reasoning diversity in educational AI tutors, automated theorem proving, and scientific research assistants.
• �� It is suitable for scenarios requiring broad solution coverage, especially where multiple reasoning paths exist.
• �� The method can be integrated into existing RLVR pipelines to improve exploration without complex reward engineering, making it practical for industrial deployment.

• �� The reliance on auxiliary model quality means that poor weak models may introduce misleading trajectories, potentially degrading performance.
• �� Additional computational overhead from generating multiple prefixes and maintaining multiple models may limit scalability.
• �� Effectiveness in highly specialized or out-of-distribution tasks remains uncertain, requiring further validation and adaptation.