Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction

The evolution of large language models (LLMs) such as GPT-3 and GPT-4 has revolutionized AI reasoning tasks, enabling complex language understanding and problem-solving. Early methods focused on chain-of-thought prompting and structured reasoning, which improved interpretability but limited flexibility. Reinforcement learning (RL) approaches like RLHF and RLVMR introduced environment interaction to enhance adaptability, yet often relied on predefined cognitive paths. Multi-turn interaction techniques emerged, allowing models to refine responses through feedback, but challenges in exploration efficiency and environment internalization persisted. Recent efforts aim to develop models capable of autonomous environment understanding without rigid structures, addressing the bottleneck of inefficient exploration and superficial learning. This paper builds upon these foundations, proposing a flexible, active learning framework that internalizes environment dynamics through reward mechanisms and adaptive interaction strategies.

Despite progress, current models struggle with efficiently internalizing environment knowledge, especially in complex, high-dimensional spaces. Internal simulation without external feedback leads to over-reliance on erroneous internal representations, causing failures in generalization. Excessive interaction can cause redundant exploration, wasting resources and impeding learning. Moreover, models tend to exploit environmental cues rather than internalize dynamics, limiting long-term reasoning. Addressing these issues requires mechanisms that promote purposeful exploration, reduce redundancy, and facilitate internal knowledge compression, enabling models to solve complex tasks efficiently and robustly.

The paper introduces two key innovations: first, reward rescaling based on action effectiveness, which incentivizes models to prioritize meaningful actions and reduce redundancy; second, interaction frequency annealing, which dynamically reduces the maximum interaction turns during training, encouraging internalization of environment dynamics. These mechanisms work synergistically, guiding models from reactive, multi-turn exploration to proactive, single-turn reasoning. Unlike prior structured reasoning methods, WMAct promotes autonomous, flexible learning, allowing models to adaptively internalize environment behaviors. The approach is compatible with PPO, ensuring stable training and broad applicability across diverse environments, validated through extensive experiments.

• �� The framework employs PPO with modifications to incorporate reward rescaling and frequency annealing.
• �� Reward rescaling: compute effective actions (state changes), scale reward proportionally to reduce ineffective actions, formula: R_scaled = R_outcome × (Neff/N).
• �� Frequency annealing: periodically compute average interaction length, adjust maximum turns L_max = (L̄ + L'_max)/2, gradually decreasing interaction opportunities.
• �� Multi-turn interaction: model generates thinking steps T_t and action sets A_t, observes environment o_t, and updates policy accordingly.
• �� The training objective maximizes expected reward J(θ), optimized via policy gradients, ensuring stable convergence.
• �� The mechanisms collectively promote purposeful exploration, environment internalization, and efficient reasoning, validated through experiments showing rapid task mastery and transferability.

Experiments involved training models on Sokoban, Maze, and Taxi environments with varying difficulty levels, comparing WMAct against PPO-EntirePlan and multi-turn PPO baselines. The training used a 8B-parameter model with a batch size of 256, response sampling at temperature 1.0, and maximum 30 interaction turns. Evaluation metrics included success rate, single-turn accuracy, and generalization to harder variants. Results demonstrated that WMAct significantly outperformed baselines, achieving over 78% success in Sokoban and 88% in Maze, with rapid convergence of single-turn accuracy. Ablation studies confirmed the contributions of reward rescaling and frequency annealing, showing their critical roles in efficiency and robustness.

WMAct achieved 78.57% success in Sokoban, surpassing PPO-EntirePlan’s 49.12%. In challenging variants, success rates remained above 50%, indicating strong generalization. Single-turn accuracy approached multi-turn performance across environments, validating effective internalization. The model’s ability to solve tasks with fewer interactions demonstrated the success of the proposed mechanisms. Performance on reasoning benchmarks like MMLU-Pro and HMMT25 also improved, confirming transferability. Overall, the approach significantly enhances reasoning efficiency and robustness in complex environments.

This method can be applied in autonomous robotics, intelligent planning, and complex decision-making systems, where environment understanding and long-term reasoning are critical. It enables models to internalize environment dynamics, reducing reliance on external feedback, thus lowering operational costs. In industry, WMAct can improve autonomous agents' adaptability, efficiency, and safety in real-world scenarios such as warehouse automation, self-driving cars, and intelligent assistants. Its ability to learn from interaction and internalize environment models makes it suitable for deployment in dynamic, uncertain settings.

Current limitations include challenges in scaling to extremely high-dimensional or real-world multimodal environments, where exploration becomes computationally intensive. The reliance on environmental feedback still poses issues in sparse or noisy signals. Additionally, training costs are high due to multi-turn interactions and reward adjustments. Future work should focus on enhancing intrinsic reasoning, reducing resource demands, and extending applicability to more complex, real-world scenarios with multimodal data.