ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
ArCHer employs hierarchical RL with high-level value functions guiding token-level policies, achieving 100x sample efficiency in multi-turn tasks.
Key Findings
Methodology
ArCHer integrates off-policy value functions with token-level policy gradients within a hierarchical Markov decision process (MDP). The high-level Q-function, trained via Bellman backups, summarizes long-term rewards over utterances, while the low-level policy optimizes token generation using advantage estimates derived from this high-level critic. The framework supports offline and online training, employing double Q-learning and Polyak averaging for stability. Experiments with GPT-2 and RoBERTa models on web navigation and customer support tasks demonstrate a 100-fold increase in sample efficiency and scalability up to 7 billion parameters.
Key Results
- ArCHer outperforms PPO and single-turn RL by approximately 100x in sample efficiency, converging faster and achieving higher success rates. In web navigation, success rate improved by 15%, with reward convergence 2-3 times faster. The model maintained performance when scaled to 7 billion parameters, showing robustness.
- Ablation studies confirmed the importance of hierarchical structure and off-policy training, with token-level advantages reducing variance and enhancing learning speed.
- The approach effectively handles long-horizon rewards and information gathering, enabling better credit assignment and historical reasoning in multi-turn settings.
Significance
This work advances the application of reinforcement learning in large language models, especially for multi-turn decision-making tasks. By addressing sample inefficiency and long-horizon credit assignment, ArCHer opens new possibilities for deploying LLMs in complex interactive environments like web automation, customer support, and intelligent assistants. Its flexible, scalable design facilitates integration with existing models and algorithms, promising broader industrial impact.
Technical Contribution
The core innovation is the hierarchical RL framework combining high-level off-policy value functions with token-level policy gradients, enabling efficient multi-turn training. The hierarchical MDP formulation allows decoupling long-term reward estimation from token-level action optimization. The use of dual Q-networks, Bellman backups, and advantage-based policy updates ensures stability and scalability. This approach generalizes existing RLHF methods, providing a new paradigm for multi-turn LLM fine-tuning.
Novelty
This is the first work to embed hierarchical MDPs into multi-turn RL for LLMs, leveraging high-level value functions to guide token policies. Unlike prior single-step or flat RL methods, ArCHer effectively manages long horizons and large action spaces, representing a significant leap in RL methodology for language models.
Limitations
- Dependence on pre-trained models limits adaptability to tasks with extremely sparse rewards or very long horizons. The training process remains computationally intensive, requiring substantial hardware resources.
- High-quality high-level value estimation can be biased, especially in out-of-distribution scenarios. Further research is needed to improve robustness and generalization.
Future Work
Future directions include integrating self-supervised signals to reduce reliance on external rewards, extending the framework to multi-modal tasks, and scaling to larger models. Additionally, exploring more efficient hierarchical structures and bias correction techniques will further enhance performance and applicability.
AI Executive Summary
In recent years, large language models (LLMs) like GPT-3 and RoBERTa have revolutionized natural language processing, demonstrating remarkable capabilities in understanding and generating human-like text. However, their application in goal-directed, multi-turn decision-making tasks remains limited by training inefficiencies and the inability to handle long-term rewards effectively. Traditional reinforcement learning methods such as PPO excel in single-turn settings but struggle with multi-turn interactions due to long horizons, sparse rewards, and large action spaces.
To address these challenges, this paper introduces ArCHer, a hierarchical reinforcement learning framework designed specifically for fine-tuning LLMs in multi-turn agent tasks. The key idea is to decompose the decision process into two levels: a high-level off-policy value function that aggregates rewards over entire utterances, and a low-level token policy that generates responses conditioned on this high-level estimate. This structure allows the model to leverage the stability and sample efficiency of off-policy learning while maintaining the flexibility of token-level policy optimization.
The technical core of ArCHer involves training a pair of neural networks—Q and V functions—at the utterance level using Bellman backups, and optimizing token generation policies via advantage-weighted policy gradients. This dual-level approach effectively manages the long-horizon credit assignment problem, enabling the model to learn from both online interactions and static datasets. Experiments with GPT-2 and RoBERTa on web navigation and customer support tasks show that ArCHer achieves approximately 100 times higher sample efficiency than PPO, converges faster, and maintains performance when scaled to models with up to 7 billion parameters.
This work significantly pushes the boundary of multi-turn RL in language models, providing a scalable, flexible framework that can be integrated into existing systems. Its ability to handle delayed rewards, long-term planning, and information-seeking behaviors opens new avenues for deploying LLMs in complex, real-world scenarios. Future research will focus on further reducing training costs, enhancing robustness, and extending the framework to multi-modal and larger-scale models, promising a new era of intelligent, interactive AI agents.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家大型工厂工作，工厂里有两个部门：一个负责制定整体生产计划（高层决策），另一个负责具体操作（低层执行）。高层部门根据市场需求和目标，制定出大方向，比如“今天要生产多少产品”，然后把这个目标传达给操作部门。操作部门会根据这个目标，逐步完成具体任务，比如“先准备原料”、“组装零件”、“检查质量”。在这个过程中，工厂会不断调整策略，比如发现某个环节效率低，就会改进流程。ArCHer就像这个工厂，把复杂的任务拆成两个层次：高层负责整体策略，低层负责具体行动。这样，工厂能更快适应变化，效率更高。它用一种聪明的方法，让每个部门都能合作得更好，完成复杂的任务，比如网页导航或客户咨询，效果比以前高出很多。这种层级设计让整个系统变得更智能、更高效，就像一个有两个指挥官的团队，合作无间，完成各种挑战。
Abstract
A broad use case of large language models (LLMs) is in goal-directed decision-making tasks (or "agent" tasks), where an LLM needs to not just generate completions for a given prompt, but rather make intelligent decisions over a multi-turn interaction to accomplish a task (e.g., when interacting with the web, using tools, or providing customer support). Reinforcement learning (RL) provides a general paradigm to address such agent tasks, but current RL methods for LLMs largely focus on optimizing single-turn rewards. By construction, most single-turn RL methods cannot endow LLMs with the ability to intelligently seek information over multiple turns, perform credit assignment, or reason about their past actions -- all of which are critical in agent tasks. This raises the question: how can we design effective and efficient multi-turn RL algorithms for LLMs? In this paper, we develop a framework for building multi-turn RL algorithms for fine-tuning LLMs, that preserves the flexibility of existing single-turn RL methods for LLMs (e.g., proximal policy optimization), while accommodating multiple turns, long horizons, and delayed rewards effectively. To do this, our framework adopts a hierarchical RL approach and runs two RL algorithms in parallel: a high-level off-policy value-based RL algorithm to aggregate reward over utterances, and a low-level RL algorithm that utilizes this high-level value function to train a token policy within each utterance or turn. Our hierarchical framework, Actor-Critic Framework with a Hierarchical Structure (ArCHer), can also give rise to other RL methods. Empirically, we find that ArCHer significantly improves efficiency and performance on agent tasks, attaining a sample efficiency of about 100x over existing methods, while also improving with larger model capacity (upto the 7 billion scale that we tested on).