Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning
HILA framework with Dual-Loop Policy Optimization enables adaptive human–agent collaboration, outperforming state-of-the-art multi-agent systems by 10%+ on reasoning benchmarks.
Key Findings
Methodology
This paper introduces HILA, a framework integrating metacognitive policies with dual-loop optimization (DLPO). The inner loop employs Group Relative Policy Optimization (GRPO) with cost-aware rewards to refine deferral decisions online. The outer loop leverages expert feedback, transforming it into supervised signals for continual learning, thus expanding the model's knowledge boundary. The system encodes high-level cognitive states, including task context, self-assessment, peer responses, and structured cognitive cues, enabling strategic regulation of autonomous and human-involved actions. This design allows agents to dynamically balance exploration, exploitation, and deferral, leading to significant performance gains across reasoning benchmarks.
Key Results
- On GSM8K, accuracy improved from 80.79% to 89.86%, and on AIME from 12.45% to 35.83%, surpassing all baselines. The approach effectively avoids knowledge bottlenecks by learning when to defer, especially under high uncertainty, reducing errors by over 15%. Ablation studies confirm the importance of policy learning and continual updates, with performance gains consistent across different model sizes (Qwen2.5-7B, LLaMA3-8B).
- The framework's robustness was validated across multiple backbones, showing 4-12 point improvements in reasoning tasks. The combination of strategic deferral and continual learning was key, with ablation revealing that removing either component diminishes performance. Results demonstrate that the approach generalizes well, especially for smaller or weaker models, by compensating limited reasoning capacity.
- In complex reasoning scenarios, HILA exhibits stronger resilience to errors and better knowledge expansion, especially in high-uncertainty tasks. Its ability to strategically defer reduces mistake rates, leading to more reliable and scalable AI systems for real-world applications.
Significance
This work advances AI towards open-ended, continually improving multi-agent systems capable of strategic collaboration with humans. By integrating metacognitive decision-making with continual learning, it addresses longstanding limitations of static knowledge horizons. The approach enhances AI's adaptability, robustness, and capacity for knowledge growth, with broad implications for autonomous reasoning, scientific discovery, and industrial automation. It paves the way for systems that can learn from experience, dynamically incorporate external expertise, and operate reliably in complex, real-world environments.
Technical Contribution
The paper's main contributions include the design of a hierarchical metacognitive policy framework, the development of the Dual-Loop Policy Optimization algorithm combining reinforcement learning with supervised continual learning, and the integration of structured cognitive states for strategic decision-making. This approach introduces a novel way to decouple short-term deferral actions from long-term capability development, enabling scalable, adaptive multi-agent systems. The algorithm guarantees improved decision quality and knowledge expansion, setting new standards for AI collaboration and lifelong learning.
Novelty
This is the first systematic integration of metacognitive strategies into multi-agent LLM systems, combining reinforcement learning with continual learning to optimize deferral decisions and knowledge growth. Unlike prior work relying on heuristics or static protocols, this approach dynamically learns when and how to involve external expertise, effectively overcoming the knowledge ceiling of pre-trained models. It establishes a new paradigm of open, self-improving AI agents.
Limitations
- Dependence on high-quality expert feedback, which can be costly and limit scalability in real-world deployment. The feedback collection process remains resource-intensive.
- Training complexity and computational costs are high, requiring extensive hyperparameter tuning and large-scale resources.
- The current framework's effectiveness on multi-modal data, real-time interaction, and highly dynamic environments remains to be validated. Future work should focus on reducing feedback dependency and improving efficiency.
Future Work
Future directions include developing automated feedback mechanisms, integrating multi-modal inputs, and optimizing for real-time applications. Enhancing model interpretability and robustness in open-world settings is also critical. Further research will explore reducing feedback costs, scaling to larger models, and deploying in diverse real-world scenarios such as autonomous agents, scientific research, and industrial automation.
AI Executive Summary
The evolution of large language models (LLMs) has revolutionized natural language understanding and generation, yet their capabilities remain constrained by static knowledge and limited adaptability. Multi-agent systems (MAS) have emerged as a promising approach to address these limitations by enabling collaborative problem solving among multiple models. However, most existing MAS rely on fixed interaction protocols, which restrict their ability to learn and adapt beyond pre-defined behaviors. They are essentially 'closed-world' systems, unable to incorporate new knowledge or respond flexibly to novel challenges.
This paper introduces the Human-In-the-Loop Multi-Agent Collaboration (HILA) framework, which fundamentally shifts this paradigm by embedding a metacognitive policy into each agent. This policy allows agents to reason about their own competence and that of their peers, enabling strategic decisions on when to act autonomously and when to defer to human experts. The core innovation is the Dual-Loop Policy Optimization (DLPO), a training methodology that separates short-term decision-making from long-term capability growth. The inner loop employs Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm that optimizes deferral actions based on cost-aware rewards, encouraging agents to defer only when necessary. The outer loop leverages expert feedback, transforming it into supervised signals that continually expand the model's knowledge base.
Experimental results across diverse benchmarks, including GSM8K, AIME, and HumanEval, demonstrate that HILA with DLPO consistently outperforms existing state-of-the-art multi-agent systems by 10% or more. Notably, the approach reduces errors in high-uncertainty scenarios and enhances model robustness, especially for smaller or weaker models. The framework's ability to dynamically balance exploration, exploitation, and external guidance marks a significant step towards open-ended, self-improving AI systems.
This work has profound implications for AI research and applications, enabling systems that are not only more accurate but also capable of continual learning and adaptation. Future research will focus on automating feedback collection, expanding to multi-modal data, and deploying in real-time, dynamic environments. Despite current limitations related to feedback costs and computational demands, the proposed approach opens new avenues for scalable, intelligent, and collaborative AI systems that can learn and grow with experience.
Deep Analysis
Background
Recent advances in large language models (LLMs) like GPT-4 and LLaMA have significantly improved natural language understanding and generation. Early efforts focused on single-model scaling, but limitations in reasoning, knowledge scope, and adaptability became apparent. Multi-agent systems (MAS) emerged to address these issues by enabling models to collaborate, share information, and perform complex tasks collectively. Techniques such as structured debates, dynamic routing, and graph-based workflows improved internal knowledge recombination but remained confined within static knowledge boundaries. Human-in-the-loop approaches introduced external supervision, but often relied on heuristics like confidence thresholds for deferral, lacking learned decision policies. These methods improved performance but failed to address the core challenge of open-ended, continual learning, which is essential for real-world deployment. The need for systems that can dynamically assess their own capabilities, defer intelligently, and learn from ongoing interactions remains unmet, motivating the development of more adaptive frameworks.
Core Problem
The main challenge is enabling multi-agent LLM systems to make strategic deferral decisions that balance autonomous problem-solving with external guidance. Existing heuristics are insufficient for complex, uncertain tasks, leading to suboptimal performance and knowledge stagnation. The core bottleneck is the inability to model uncertainty and to learn when to seek human assistance effectively. Additionally, integrating expert feedback into the model for sustained capability growth is non-trivial, often limited to one-time fine-tuning rather than ongoing learning. This results in systems that cannot adapt to new information or improve over time, restricting their usefulness in dynamic environments. Addressing these issues requires a principled decision-making framework that can evaluate confidence, task difficulty, and knowledge gaps, and a training paradigm capable of translating feedback into long-term skill enhancement.
Innovation
The paper introduces several key innovations: 1) a metacognitive policy framework that enables agents to reason about their own and peers' competence; 2) the Dual-Loop Policy Optimization (DLPO) algorithm that separates short-term deferral decisions from long-term knowledge expansion; 3) the integration of reinforcement learning (GRPO) with supervised continual learning, allowing agents to learn when to defer and how to incorporate expert feedback effectively. Unlike prior heuristics-based methods, this approach learns strategies dynamically, improving decision quality and knowledge growth simultaneously. The structured cognitive state space, including task context, self-assessment, peer responses, and cognitive cues, provides a rich basis for strategic regulation. Overall, the framework advances the field by enabling open-ended, self-improving multi-agent systems capable of continual adaptation.
Methodology
- �� Formalize human–agent collaboration as a Metacognitive Markov Decision Process (Meta-MDP), defining states with task, self, peer, and cognitive cues.
- �� Design a high-level action space including evaluate, create, and defer, for strategic decision-making.
- �� Implement multi-round interaction protocol, where agents sample actions based on policy πθ, and generate responses accordingly.
- �� Develop the Dual-Loop Policy Optimization (DLPO):
- �� Inner loop employs GRPO with cost-sensitive rewards to optimize deferral and exploration actions.
- �� Outer loop uses expert feedback as supervised data, converting deferrals into training samples for continual learning.
- �� During training, combine reinforcement learning objectives with supervised fine-tuning, balancing immediate decision quality with knowledge expansion.
- �� Conduct experiments on GSM8K, AIME, and HumanEval, comparing performance with baselines, and performing ablation studies to verify contributions.
Experiments
The experimental setup involves benchmarking HILA against state-of-the-art multi-agent systems on datasets like GSM8K, AIME, and HumanEval. Models include LLaMA3-8B and Qwen2.5-7B. Metrics include accuracy, error rate, and robustness under uncertainty. The training process involves hyperparameter tuning for the RL inner loop, expert feedback sampling, and continual learning iterations. Ablation studies assess the impact of policy optimization and knowledge expansion modules. Results show consistent improvements, with accuracy gains of 10-15% over baselines, especially in high-uncertainty tasks. The experiments validate the effectiveness of strategic deferral and continual learning in enhancing reasoning capabilities and knowledge growth.
Results
HILA achieves 89.86% accuracy on GSM8K, surpassing previous best of 80.79%. On AIME, accuracy rises from 12.45% to 35.83%. Across multiple backbones, performance gains range from 4 to 12 percentage points. Ablation confirms that removing either the RL-based policy or continual learning reduces performance significantly. The results demonstrate that strategic deferral, guided by learned policies, effectively mitigates errors and expands knowledge, especially in complex, uncertain scenarios. The approach generalizes well across model sizes and tasks, confirming its robustness and scalability.
Applications
This framework can be applied to automated reasoning, scientific research assistants, and industrial automation where continual learning and external expertise are vital. It enables AI systems to adaptively balance autonomous operation with human oversight, improving reliability and scalability. Future deployment may include real-time decision-making in dynamic environments, multi-modal data integration, and scalable feedback mechanisms, broadening AI's practical utility in diverse sectors.
Limitations & Outlook
Dependence on high-quality expert feedback poses scalability challenges, especially in resource-constrained settings. The training process is computationally intensive, requiring significant hyperparameter tuning. The framework's effectiveness in multi-modal, real-time, or highly dynamic environments remains to be validated. Future work should focus on reducing feedback dependency, improving efficiency, and extending applicability to broader scenarios.
Plain Language Accessible to non-experts
想象一个学校里的学生（智能体），他们在解决难题。有时候，学生自己能想出答案，但遇到特别难的问题，就会请老师（人类专家）帮忙。学生会根据情况决定：自己尝试，还是请老师指导。每次老师的建议，学生都会记住，下次遇到类似问题，就能自己解决了。这个系统就像一个不断学习、会请教老师的学生团队，既能自己努力，也会学会新知识，变得越来越聪明。它的核心思想是：在需要时请教老师，但也要自己努力，逐步变得更强。这样，学生们就能不断进步，解决越来越难的问题。
Abstract
While scaling individual Large Language Models (LLMs) has delivered remarkable progress, the next frontier lies in scaling collaboration through multi-agent systems (MAS). However, purely autonomous MAS remain ''closed-world'' systems, constrained by the static knowledge horizon of pre-trained models. This limitation makes them brittle on tasks requiring knowledge beyond training data, often leading to collective failure under novel challenges. To address this, we propose the Human-In-the-Loop Multi-Agent Collaboration (HILA) framework, a principled paradigm for human--agent collaboration. HILA trains agents to learn a metacognitive policy that governs when to solve problems autonomously and when to defer to a human expert. To operationalize this policy, we introduce Dual-Loop Policy Optimization, which disentangles immediate decision-making from long-term capability growth. The inner loop applies Group Relative Policy Optimization (GRPO) with a cost-aware reward to optimize deferral decisions, while the outer loop implements continual learning, transforming expert feedback into high-quality supervised signals that strengthen the agent's reasoning ability. Experiments on challenging mathematical and problem-solving benchmarks show that HILA, equipped with Dual-Loop Policy Optimization, consistently outperforms advanced MAS, establishing a principled foundation for collaborative and continually improving agentic systems.