AgentCPM-Explore: Realizing Long-Horizon Deep Exploration for Edge-Scale Agents
AgentCPM-Explore achieves 97.09% GAIA accuracy via parameter merging, reward denoising, and context refinement in a 4B model.
Key Findings
Methodology
Introduces a three-stage framework: parameter merging using DELLA algorithm, reward signal denoising to filter noise/errors, and context refinement for long-text reasoning.
Key Results
- Achieved 97.09% accuracy on GAIA benchmark, surpassing Claude-4.5-Sonnet and larger models.
- Reached 82.7% on FRAMES, outperforming IterResearch-30B by 11.7 percentage points.
- Improved BrowseComp performance by 12.4 percentage points over WebDancer-32B.
Significance
Demonstrates edge-scale models can overcome inference stability bottlenecks, challenging large-model dominance and enabling intelligent applications on low-resource devices.
Technical Contribution
First systematic study of 4B agent models, introducing reward denoising and context refinement methods to enhance small-model performance.
Novelty
First to achieve cross-tier performance exceeding 8B and some 30B models, with innovative parameter merging and reward filtering techniques.
Limitations
- Reward denoising relies on accurate noise detection, which may fail in complex environments.
- Context refinement depends on high-quality teacher models, limiting low-resource applications.
- Hyperparameter sensitivity in model merging requires further optimization.
Future Work
Future research could explore more efficient reward filtering algorithms, automated context optimization, and broader task applications.
AI Executive Summary
Current large language model (LLM) agents rely heavily on large-scale models (e.g., 8B+ parameters), limiting their applicability on edge devices. AgentCPM-Explore systematically studies 4B-scale agent models and proposes a three-stage training framework: parameter merging, reward signal denoising, and context refinement. Experiments show the model surpasses 8B and some 30B models on multiple benchmarks, achieving 97.09% accuracy on GAIA tasks.
The core innovation lies in using the DELLA algorithm for parameter merging, combined with reward signal filtering to stabilize reinforcement learning. Additionally, teacher model distillation enhances long-text context processing, significantly improving inference quality.
Despite its success, AgentCPM-Explore faces challenges in noise detection and hyperparameter optimization. Future work could refine reward filtering efficiency and expand applications, paving the way for intelligent edge-device ecosystems.
Deep Analysis
Background
The rise of intelligent devices and privacy demands has made edge intelligence crucial for LLM agents. However, existing systems rely on large models unsuitable for edge devices due to resource constraints. The challenge lies in enabling complex reasoning on small models.
Core Problem
Edge-scale models (e.g., 4B parameters) face three bottlenecks: catastrophic forgetting during fine-tuning, sensitivity to noisy reward signals in RL, and reasoning degradation in long-context scenarios. These issues hinder performance improvements.
Innovation
Proposes a three-stage framework: 1) Parameter merging using DELLA algorithm to combine general and task-specific capabilities; 2) Reward signal denoising to filter environmental noise and formatting errors; 3) Context refinement via teacher model distillation for long-text information extraction.
Methodology
- �� Parameter merging: Uses DELLA algorithm to fuse base and fine-tuned models, retaining general skills while injecting task-specific expertise.
- �� Reward denoising: Filters environmental noise, format errors, and extreme trajectories to stabilize gradients.
- �� Context refinement: Optimizes instruction generation via RL and enhances summarization through teacher model distillation.
Experiments
Experiments on GAIA, FRAMES, and six other benchmarks compare AgentCPM-Explore against 8B and 30B models. Ablation studies validate reward denoising and parameter merging contributions.
Results
AgentCPM-Explore achieves 97.09% GAIA accuracy, surpassing Claude-4.5-Sonnet; outperforms IterResearch-30B on FRAMES by 11.7 points; improves BrowseComp by 12.4 points over WebDancer-32B.
Applications
Applicable to edge-device scenarios like virtual assistants, mobile productivity tools, and privacy-preserving applications, especially in low-resource environments.
Limitations & Outlook
Reward denoising may fail in noisy environments; context refinement depends on teacher models; hyperparameter sensitivity in merging requires optimization.
Plain Language Accessible to non-experts
Imagine AgentCPM-Explore as a team of chefs: the base model is a versatile head chef, while the task model specializes in a specific cuisine. Parameter merging combines their skills, creating a balanced team. Reward denoising acts like a quality inspector, filtering out bad ingredients. Context refinement is the menu designer, ensuring dishes meet customer needs.
ELI14 Explained like you're 14
Think of AgentCPM-Explore as a gaming sidekick! It's a small robot that learns from a big robot and a specialist robot, combining their skills to be both smart and focused. It avoids traps (bad rewards) and picks out treasure (key info) from long quests. It's like leveling up your character to beat bosses with fewer resources!
Glossary
Parameter Merging
Combines weights from two models to balance general and task-specific skills.
Used to address catastrophic forgetting.
Reward Signal Denoising
Filters noisy signals to stabilize reinforcement learning.
Key mechanism during RL training.
Context Refinement
Enhances long-text reasoning via teacher model distillation.
Improves inference quality.
Catastrophic Forgetting
Loss of general skills during fine-tuning.
Major issue in supervised fine-tuning.
DELLA Algorithm
A parameter merging algorithm for capability complementarity.
Used in model fusion.
Open Questions Unanswered questions from this research
- 1 How to improve reward filtering robustness in noisy environments.
- 2 Reducing dependency on high-quality teacher models for context refinement.
- 3 Exploring more efficient parameter merging algorithms to reduce sensitivity.
Applications
Immediate Applications
Virtual Assistants
Enable efficient task handling on edge devices like voice assistants and smart homes.
Privacy Tools
Support localized intelligent applications, reducing data upload needs.
Long-term Vision
Edge Intelligence Ecosystem
Advance edge-device capabilities for complex tasks in low-resource settings.
Abstract
While Large Language Model (LLM)-based agents have shown remarkable potential for solving complex tasks, existing systems remain heavily reliant on large-scale models, leaving the capabilities of edge-scale models largely underexplored. In this paper, we present the first systematic study on training agentic models at the 4B-parameter scale. We identify three primary bottlenecks hindering the performance of edge-scale models: catastrophic forgetting during Supervised Fine-Tuning (SFT), sensitivity to reward signal noise during Reinforcement Learning (RL), and reasoning degradation caused by redundant information in long-context scenarios. To address the issues, we propose AgentCPM-Explore, a compact 4B agent model with high knowledge density and strong exploration capability. We introduce a holistic training framework featuring parameter-space model fusion, reward signal denoising, and contextual information refinement. Through deep exploration, AgentCPM-Explore achieves state-of-the-art (SOTA) performance among 4B-class models, matches or surpasses 8B-class SOTA models on four benchmarks, and even outperforms larger-scale models such as Claude-4.5-Sonnet or DeepSeek-v3.2 in five benchmarks. Notably, AgentCPM-Explore achieves 97.09% accuracy on GAIA text-based tasks under pass@64. These results provide compelling evidence that the bottleneck for edge-scale models is not their inherent capability ceiling, but rather their inference stability. Based on our well-established training framework, AgentCPM-Explore effectively unlocks the significant, yet previously underestimated, potential of edge-scale models.