RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
StarPO and StarPO-S expose the Echo Trap in multi-turn agent RL; uncertainty filtering extends FrozenLake-PPO stability from 100 to 140 steps.
Key Findings
Methodology
StarPO optimizes complete state–thought–action–reward trajectories with J(θ)=E[R(τ)], while supporting PPO and GRPO token updates. RAGEN supplies modular environments, rewards, and rollout infrastructure. StarPO-S adds reward-variance trajectory filtering, critic baselines, KL-term removal, and Clip-Higher asymmetric gradient shaping to stabilize multi-turn learning.
Key Results
- Across Bandit, Sokoban, and Frozen Lake, vanilla PPO/GRPO initially improve but later collapse. In FrozenLake-PPO, reward standard deviation drops around step 40 before mean reward collapses around step 90; gradient spikes occur near steps 170, 110, and 90 for Bandit, Sokoban, and Frozen Lake respectively.
- StarPO-S retains high-uncertainty samples: retaining 75% of FrozenLake-PPO trajectories extends stability from 100 to 140 steps, while retaining 50% largely prevents collapse; 25% is the default. Online-1, with fresh rollouts every update, outperforms Online-5/10 in convergence and generalization.
- Rollout design matters: four responses per prompt yields 25.39% on SokobanNewVocab and 21.48% on FrozenLake; six actions per turn yields 33.59% on Sokoban and 6.64% on LargeSokoban, versus 12.11% and 1.17% with one action.
Significance
The paper reframes multi-turn RL failure as a measurable Echo Trap: entropy and reward variability contract before gradient spikes and performance collapse. It shows why single-turn PPO or GRPO cannot simply be transplanted to stochastic, long-horizon agents. RAGEN provides a reproducible testbed for web agents, robotics-oriented planning, and assistants, while the reasoning experiments warn that visible chain-of-thought formatting is not evidence of genuine reasoning.
Technical Contribution
StarPO connects autoregressive token likelihoods to trajectory-level rewards using a_t^T=<think>...</think><answer>a_t</answer>. It supports PPO advantages or GRPO’s normalized group rewards over the complete trajectory. StarPO-S introduces active-learning-style uncertainty U=Std[R(τ)] for prompt selection and combines critics with KL removal and Clip-Higher, offering a unified sampling-and-gradient stabilization recipe for multi-turn agent RL.
Novelty
The main novelty is a systematic account of self-evolution dynamics rather than another single-turn objective. Echo Trap is identified as a recurring failure mode, and task diversity, interaction granularity, rollout freshness, and reward semantics are evaluated together. Compared with ordinary PPO/GRPO adaptations, the work treats trajectory construction and training stability as first-class design variables.
Limitations
- The main experiments use Qwen-2.5-Instruct 0.5B, with 3B for WebShop, and mostly controlled environments. Although WebShop adds realism, evidence for open-world browsing, robotics, and very long horizons remains limited.
- Rewards primarily reflect rules or task success. The study itself shows that merely requiring <think> output can produce shallow or hallucinated thoughts; a general fine-grained, reasoning-aware reward is still missing.
Future Work
Future work should test StarPO-S on real toolchains and long-horizon tasks, adapt filtering ratios automatically, and combine hierarchical GAE or response masking with process supervision. Larger models, replay-based training, and cross-environment transfer should be evaluated alongside rewards that verify state consistency, causal planning, and intermediate progress.
AI Executive Summary
Training a language model as an agent requires more than producing a correct answer. The model must remember changing states, make sequential decisions, and cope with stochastic feedback. RAGEN shows that directly adapting single-turn PPO or GRPO often gives only temporary gains: the model amplifies locally successful verbal templates, loses exploration, and eventually collapses.
The authors propose StarPO—State-Thinking-Actions-Reward Policy Optimization—and the modular RAGEN system. StarPO treats reasoning, executable actions, observations, and feedback as one trajectory, optimizing J(θ)=E[R(τ)]. RAGEN implements this loop across Bandit, Sokoban, Frozen Lake, and WebShop. To counter the Echo Trap, StarPO-S filters prompts whose repeated rollouts have low reward variability and adds critic-based stabilization, KL-term removal, and Clip-Higher gradient shaping.
The evidence is concrete. FrozenLake-PPO’s reward variability drops near step 40, while mean reward collapses around step 90; StarPO-S extends stable training from 100 to 140 steps. Four responses per prompt reach 25.39% on SokobanNewVocab, and six actions per turn reach 33.59% on Sokoban. Online-1 beats Online-5/10 by keeping data aligned with the current policy. The broader lesson is sobering: without process-level, reasoning-aware rewards, a model may write plausible thoughts without actually using them.
Deep Analysis
Background
Most LLM reinforcement learning targets single-turn mathematics or code, where PPO and GRPO optimize one prompt–response reward. Agents instead maintain history, act repeatedly, and receive stochastic transitions. RAGEN combines three controllable symbolic environments with WebShop to study self-improvement under these genuinely interactive conditions.
Core Problem
Long trajectories create credit-assignment, exploration, policy–data mismatch, and value-estimation problems. A terminal success reward cannot identify which thought or action mattered. Repeatedly exploiting a successful template can reduce diversity, producing low reward variance, abnormal entropy, gradient spikes, and eventual failure.
Innovation
- ��StarPO optimizes J=E[R(τ)] over complete thought–action trajectories. •StarPO-S defines uncertainty U=Std[R(τ)] and trains preferentially on informative prompts, adding critic baselines, KL removal, and Clip-Higher. •The study establishes practical rollout principles: diverse initial states, 5–6 actions per turn, multiple responses per prompt, and Online-1 freshness.
Methodology
- ��Given state s_t and history, the model emits a_t^T=<think>...</think><answer>a_t</answer>. •The environment returns r_t and s_{t+1}; episodes run up to five turns and ten actions. •Each iteration samples P=8 prompts and N=16 trajectories, then applies PPO with GAE (γ=λ=1) or GRPO. •StarPO-S ranks prompts by repeated-rollout reward standard deviation, keeps a configurable high-uncertainty fraction—25% by default—and uses Adam, entropy coefficient 0.001, and a −0.1 format penalty.
Experiments
Qwen-2.5-Instruct 0.5B is trained on the three symbolic tasks and the 3B model on WebShop, using H100 GPUs for 100–200 rollout-update iterations. Evaluation uses 256 fixed prompts, temperature 0.5, and five-turn truncation. Metrics are success rate, rollout entropy, in-group reward standard deviation, response length, and gradient norm. Comparisons include PPO, GRPO, StarPO-S, filtering ratios, action budgets, and Online-1/5/10.
Results
Vanilla training collapses after early gains on Bandit and Sokoban; PPO is usually steadier, whereas GRPO is stronger on stochastic Frozen Lake and WebShop. FrozenLake-PPO’s variance falls at step 40 and mean reward at step 90. StarPO-S extends stability from 100 to 140 steps. Six actions yield 33.59% on Sokoban, while four responses per prompt yield 25.39% on SokobanNewVocab.
Applications
The framework can inform web-shopping agents, tool-use assistants, tutoring systems, and simulated robot planners. Practitioners should monitor reward variance, entropy, and gradient norms; use diverse starts, moderate action budgets, and fresh rollouts. Process checks are essential because final success alone cannot validate an agent’s reasoning.
Limitations & Outlook
The models and tasks are small relative to deployed agents, and WebShop is not an open-world benchmark. Process rewards remain underdeveloped, so plausible post-hoc reasoning can be mistaken for planning. Online sampling and repeated responses increase cost, and the 25% filter is not universally optimal. Future work should test 72B/frontier models, real tools, longer horizons, and verifiable process supervision.
Plain Language Accessible to non-experts
Imagine a delivery company sending drivers through an unfamiliar city. Each driver observes the streets, chooses a route, and receives feedback about whether the package arrived. Ordinary training may make everyone copy one route that worked once. Performance rises briefly, but a road closure causes the whole company to fail. That is the Echo Trap.
StarPO keeps the whole journey: what the driver saw, what route was considered, what turn was taken, and what happened next. StarPO-S gives extra training attention to neighborhoods where drivers produce different outcomes, because disagreement signals useful information. It also prevents the company from changing its routing rules too violently after one lucky delivery.
The experiments show that varied starting points, a moderate number of turns, and frequent updates produce better learning. But a successful delivery alone is not enough: a driver might write a convincing explanation that has nothing to do with the actual streets. Reliable training must check whether each step understood the situation and genuinely helped the delivery.
ELI14 Explained like you're 14
Think of an AI playing a changing maze game. It sees the board, decides what to do, presses buttons, and gets points. A normal training system mostly cares about whether the game ends in a win. If the AI wins by luck, it may repeat the same moves forever. It looks smarter for a while, then crashes on a new maze. That is the Echo Trap!
StarPO saves the whole match: what the AI saw, what it seemed to think, which move it made, and what happened afterward. StarPO-S looks for levels where repeated attempts give different scores, because those levels teach more. It also stops the AI from changing its behavior too wildly after one lucky result.
The researchers tested Bandit, Sokoban, Frozen Lake, and WebShop. In Frozen Lake, reward variety dropped around step 40, while average performance fell around step 90. The stabilized method extended steady training from 100 to 140 steps. In Sokoban, allowing six moves per turn produced 33.59% success.
Here is the surprising part: writing a “thinking” paragraph does not prove the AI actually thought. If the only reward is winning, it can invent a cool-sounding explanation afterward. Future game bots, tutors, and robots need rewards for understanding each step—not just reaching the finish line!
Glossary
StarPO (State-Thinking-Actions-Reward Policy Optimization)
A trajectory-level policy-optimization framework for multi-turn agents. It maximizes cumulative reward over complete state, thought, action, and feedback sequences.
The paper’s central framework, implemented with PPO and GRPO.
StarPO-S
A stabilized StarPO variant using uncertainty-based filtering, critic information, KL-term removal, and Clip-Higher. Its default filtering keeps about 25% of high-variance prompts.
Introduced to mitigate Echo Trap collapse.
Echo Trap
A failure mode in which an agent repeatedly copies locally rewarded patterns, reducing entropy and behavioral diversity before gradients spike and performance collapses. It resembles self-training feedback loops that amplify memorized paths.
The paper’s main diagnosis of multi-turn RL instability.
GRPO (Group Relative Policy Optimization)
A critic-free policy method that normalizes each trajectory’s reward relative to a sampled group: Â=(R−mean(R))/std(R). It avoids explicit value estimation but can be sensitive to reward variability.
Compared with PPO across all environments.
Trajectory uncertainty
The standard deviation U=Std[R(τ)] of rewards from repeated rollouts of one initial state. High U indicates uncertain outcomes and potentially informative training data.
Used by StarPO-S to select prompts.
Online-k rollout
A schedule that reuses one rollout set for k consecutive policy updates. Online-1 collects fresh trajectories after every update and therefore minimizes policy–data mismatch.
Used to study rollout freshness.
Open Questions Unanswered questions from this research
- 1 How can process rewards verify intermediate state understanding and causal planning without rewarding persuasive but false language? The paper identifies this mismatch but does not provide a general solution.
- 2 Whether StarPO-S remains stable and affordable in real browsers, robots, and very long tasks is unknown; larger-scale online studies and cross-environment transfer are needed.
Applications
Immediate Applications
Web-shopping agents
Developers can train search-and-purchase agents in WebShop-like environments using diverse starts, Online-1 collection, and reward-variance monitoring. Tool feedback must be logged and task outcomes must be objectively verifiable to avoid reinforcing accidental shortcuts.
Tool-use and tutoring assistants
Store complete call traces and score state interpretation, tool selection, and final completion separately. StarPO-S can focus learning on uncertain cases, but process rewards are required before reasoning text is treated as a trustworthy explanation.
Long-term Vision
Self-improving robots and general agents
With reliable process supervision, trajectory-level RL could let robots and assistants learn planning policies under stochastic conditions. Major obstacles include safety constraints, expensive real-world exploration, long-horizon credit assignment, and simulation-to-reality transfer.
Abstract
Training large language models (LLMs) as interactive agents presents unique challenges including long-horizon decision making and interacting with stochastic environment feedback. While reinforcement learning (RL) has enabled progress in static tasks, multi-turn agent RL training remains underexplored. We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents. Our study on four stylized environments reveals three core findings. First, our agent RL training shows a recurring mode of Echo Trap where reward variance cliffs and gradient spikes; we address this with StarPO-S, a stabilized variant with trajectory filtering, critic incorporation, and gradient stabilization. Second, we find the shaping of RL rollouts would benefit from diverse initial states, medium interaction granularity and more frequent sampling. Third, we show that without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge through multi-turn RL and they may show shallow strategies or hallucinated thoughts. Code and environments are available at https://github.com/RAGEN-AI/RAGEN.