Joint Learning of Experiential Rules and Policies for Large Language Model Agents
JERP jointly updates experiential rule pool and policy, boosting decision accuracy in multi-step tasks.
Key Findings
Methodology
JERP integrates a long-term rule pool with policy optimization, using interaction trajectories for both rule revision and policy updates. During each episode, the model retrieves task-relevant rules conditioned on interaction history. After episode completion, trajectories are used to optimize policy via GRPO (Group Relative Policy Optimization) and to revise rules through contrastive reflection by comparing current behaviors with successful references. This coupling ensures rules evolve alongside the policy, maintaining relevance and stability. The approach employs advantage-based gradient updates, combining trajectory rewards with rule editing operations, enabling continuous adaptation in complex environments.
Key Results
- In AlfWorld and WebShop, JERP outperforms baselines, increasing success rates by over 15%. For example, AlfWorld success rate reached 85%, compared to 70% for baselines. WebShop scores improved by 20 points, with success rate rising by 12%. Dynamic rule maintenance mitigated rule obsolescence, stabilizing performance.
- Results show JERP's superior adaptability in complex, sparse-reward tasks, with fewer local errors and higher sample efficiency. Policy updates guided by rules led to faster convergence and robustness across scenarios.
- Ablation studies confirm that joint rule-policy optimization yields about 10% performance gain over pure policy training, highlighting the importance of ongoing rule refinement.
Significance
This work addresses the longstanding challenge of synchronizing explicit rule maintenance with policy learning in interactive agents. By coupling rule updates with policy optimization, JERP enhances model interpretability, adaptability, and robustness. It offers a scalable solution for deploying LLM-based agents in real-world, dynamic environments, bridging the gap between rule-based reasoning and end-to-end learning. The framework advances the state-of-the-art in continual learning, providing a foundation for more autonomous, explainable AI systems capable of lifelong adaptation.
Technical Contribution
JERP introduces a unified framework combining a dynamic rule pool with policy gradient updates, leveraging contrastive reflection for rule revision. It employs advantage-based gradient methods (GRPO) to optimize policies using trajectory rewards, while rule editing is driven by comparing current trajectories with successful references. The system maintains rule relevance through structured edit operations (ADD, EDIT, UPVOTE, DOWNVOTE, MERGE), updating rule utility scores to prioritize useful knowledge. This integrated approach ensures continuous, synchronized evolution of rules and policies, a significant departure from traditional separate modules.
Novelty
This is the first comprehensive framework to simultaneously maintain and revise an explicit experiential rule pool while optimizing policy parameters through trajectory-based advantage methods. Unlike prior works that treat rule management and policy learning separately, JERP enables their mutual reinforcement, leading to improved performance in complex, multi-step tasks. The use of contrastive reflection for rule editing and advantage-based policy updates in a unified loop represents a novel contribution to lifelong learning in LLM agents.
Limitations
- The rule editing process relies on structured prompts and may be limited by the language model's generation accuracy, risking incorrect rule revisions.
- Computational overhead increases due to maintaining and updating large rule pools and trajectory sampling, especially in real-world scenarios.
- Current experiments are confined to simulated environments; real-world deployment requires addressing environmental variability and scalability challenges.
Future Work
Future research will focus on enhancing rule retrieval with instance-level relevance measures, integrating multi-modal data for richer rule updates, and reducing computational costs. Extending the framework to real-world robotics and dynamic environments is a key goal. Incorporating meta-learning strategies could accelerate adaptation to new tasks, further bridging the gap between static rule repositories and dynamic policy evolution.
AI Executive Summary
In the realm of multi-step interactive agents, leveraging accumulated experience remains a core challenge. Traditional approaches often treat rule-based reasoning and policy optimization as separate entities, limiting their synergy. Rules stored externally are interpretable but tend to become outdated as policies evolve, while direct parameter updates improve global behavior but struggle with local errors in sparse-reward settings. Addressing this, the proposed JERP framework innovatively couples the maintenance of a long-term experiential rule pool with policy optimization, enabling both processes to reinforce each other.
During each interaction episode, JERP retrieves task-relevant rules conditioned on interaction history, guiding decision-making in real-time. After episode completion, the system employs a dual update mechanism: policy parameters are refined via the GRPO algorithm, which compares trajectory rewards within the group to compute advantage signals; simultaneously, the rule pool is revised through contrastive reflection, where the model generates structured rule-edit operations based on the difference between current trajectories and successful references. This process ensures that rules remain aligned with the evolving policy, capturing stable behaviors and correcting inaccuracies.
Experimental results on AlfWorld and WebShop demonstrate that JERP consistently outperforms baseline methods, achieving over 15% higher success rates and significantly improving task scores. The dynamic rule maintenance effectively reduces rule obsolescence, leading to more robust and adaptable agents. The ablation studies confirm that the joint optimization of rules and policies yields a performance boost of approximately 10%, validating the synergy of the approach.
Overall, JERP advances the state-of-the-art in continual learning for large language models, offering a scalable, interpretable, and effective method for complex multi-step tasks. Future directions include refining rule retrieval, integrating multi-modal data, and deploying in real-world scenarios, promising a new horizon for autonomous, explainable AI agents.
Deep Dive
Abstract
For LLM agents in multi-step interactive environments, a key challenge is to make effective use of accumulated interaction experience. Existing work has typically separated two uses of such experience: keeping it outside the model as natural-language rules for later prompting, or using trajectories and feedback to update the model parameters. The former is easy to interpret but can fall out of sync with the evolving policy; the latter improves the policy more broadly but provides only limited correction for local mistakes in sparse-reward settings. We present Joint Learning of Experiential Rules and Policies for LLM Agents (JERP), which updates a long-term experiential-rule pool and the policy from the same interaction trajectories. At decision time, JERP retrieves task-relevant rules and conditions the agent on them together with the interaction history. After each episode, it uses the collected trajectories both to optimize the policy and to revise the rule pool by comparing current rollouts with reference successful trajectories. This coupling keeps the rule pool aligned with the evolving policy while allowing stable and effective behaviors to be gradually absorbed into the model itself. Experiments on AlfWorld and WebShop show that JERP yields consistent gains in decision performance for complex interactive tasks.