Learning on the Job: Continual Learning from Deployment Feedback for Frozen-Weights Agents
Proposes deployment-feedback-driven continual learning using external memory, boosting τ-bench success rate by 1.6× single-trial, 2.6× with corrections.
Key Findings
Methodology
This work introduces a gradient-free continual learning framework that pairs frozen models with an external memory system. The memory distills each episode into natural language rules, stored in Spark, and retrieved via queries. Feedback signals—binary outcome verdicts and after-the-fact corrections—are used to update the memory without modifying model weights. Experiments on τ-bench banking tasks show that learning from outcome verdicts alone raises success rates from 6.4% to 17.0%, and with corrections to 39.7%. Cross-model transfer experiments demonstrate that models reading each other's frozen memory improve performance, confirming memory portability. The approach leverages retrieval-augmented generation (RAG) as a baseline, enhancing it with memory-based continual learning.
Key Results
- On τ-bench, success rate with instruction learning reaches 39.7%, solving 62/97 tasks, a 2.6× increase over baseline. Experience-only learning achieves 10.3%, and memory transfer between models boosts success by ~30%. The success rate improves across multiple trials, with success retention rates of 0.88 for experience and 0.65 for instruction, confirming knowledge persistence.
- The experiments validate that deployment feedback—success/failure verdicts and corrections—serves as a sufficient signal for continual knowledge accumulation. The memory system, based on natural language rules, effectively captures and transfers knowledge, demonstrating practical viability.
- Cross-model transfer experiments show that models can benefit from each other's memory, with success rates exceeding no-memory baselines, indicating the robustness and portability of the learned knowledge.
Significance
This research addresses a fundamental challenge: enabling AI agents to learn continuously in deployment without costly retraining. By harnessing existing feedback signals, it offers a scalable, low-cost solution for real-world applications. The approach reduces reliance on frequent weight updates, mitigates catastrophic forgetting, and enhances model adaptability across tasks and models. Its potential impact spans customer service, finance, healthcare, and beyond, fostering more autonomous, knowledge-aware AI systems capable of lifelong learning in dynamic environments.
Technical Contribution
The paper introduces a novel feedback-based knowledge distillation mechanism that encodes insights as natural language rules in an external memory. It departs from traditional gradient-based methods, enabling gradient-free continual learning. The system supports cross-model memory transfer, demonstrating the portability of knowledge. Protocols for episodic memory update, retrieval, and validation are formalized, providing a practical framework for deploying continual learning in real-world AI systems. The experimental validation on τ-bench benchmarks underscores the effectiveness of the approach.
Novelty
This work is the first to leverage deployment feedback—binary outcome verdicts and post-episode corrections—as a primary signal for continual learning without model weight updates. It innovatively encodes knowledge as natural language rules stored externally, enabling scalable, transfer-friendly knowledge accumulation. Unlike prior methods focused on internal model updates or static retrieval, this approach emphasizes real-world deployment signals and cross-model knowledge transfer, marking a significant step forward in practical continual learning.
Limitations
- The method assumes perfect, noise-free feedback signals; in real deployments, noisy or delayed feedback could diminish effectiveness.
- Natural language rule expression may require manual curation or refinement to maintain accuracy and consistency.
- Current experiments focus on deterministic tasks; scalability to more complex, stochastic environments remains to be tested.
Future Work
Future research will explore robustness to noisy feedback, integrating micro-adjustments with parameter fine-tuning, and extending to multi-modal data. Additionally, developing automated rule generation and validation mechanisms will enhance scalability. Cross-domain and multi-task transfer experiments are planned to evaluate generalization, aiming to realize truly autonomous, lifelong learning AI agents.
AI Executive Summary
In the realm of AI deployment, models traditionally remain static after training, unable to adapt from ongoing interactions. This limitation hampers their ability to improve in real-world settings, where continuous feedback is abundant but seldom utilized effectively. Addressing this challenge, the paper introduces a novel framework that leverages deployment feedback—success/failure verdicts and post-episode corrections—to enable continual learning without modifying model weights. The core idea is to distill episode-specific insights into natural language rules stored in an external memory system, Spark, which the models can retrieve during subsequent tasks.
Experimental validation on τ-bench banking tasks demonstrates the approach’s effectiveness. The results show that models using deployment feedback significantly outperform static retrieval-augmented generation (RAG) baselines. Specifically, success rates from a baseline of 6.4% rise to 17.0% with outcome verdicts alone, and further to 39.7% when corrections are incorporated. These improvements translate into solving 22 additional tasks out of 84 that the baseline never managed. The method’s robustness is confirmed through cross-model transfer experiments, where models read each other’s frozen memory stores, achieving performance gains of approximately 30%. This indicates that knowledge stored externally can be portable and beneficial across different models.
This research offers a scalable, low-cost pathway for real-world AI systems to learn continuously. By avoiding costly weight updates and mitigating catastrophic forgetting, it opens new avenues for deploying adaptable AI agents in customer service, finance, healthcare, and other knowledge-intensive domains. The approach’s reliance on existing deployment signals makes it practical and immediately applicable, while its potential for cross-model knowledge transfer suggests a future where AI systems collaboratively build and share organizational knowledge. Despite current limitations like feedback noise sensitivity, the framework sets a foundation for future innovations in autonomous, lifelong learning AI.
Deep Dive
Abstract
AI agents encounter learning opportunities in every episode they run, and discard nearly all of them: the underlying models are frozen at deployment, so an agent that resolves a difficult request today starts from zero when it recurs tomorrow. Yet ordinary operation already produces feedback, in the form of outcome verdicts and after-the-fact corrections. We show that this feedback is a sufficient signal for continual learning when the frozen model is paired with an external memory that distils each episode into retrievable natural-language rules. On the banking domain of $τ$-bench, against a static-RAG control retrieving over the complete policy corpus, learning from the one-bit outcome verdict lifts single-trial success to 1.6$\times$ the baseline, and learning from corrections to 2.6$\times$, converting 22 of the 84 tasks the baseline never solves. The result spans the deployment spectrum, measured on Mistral Large, an open-weights model that organisations with data sovereignty requirements can self-host, and replicated on a frontier model, Claude Sonnet 5. The accumulated memory also transfers: each model, reading the store built by the other, rises above its own no-memory baseline. The harness, protocol, and data are released.