RSMeM: Knowledge-Enhanced Memory Evolution for Remote Sensing Agents with Systematic Evaluation
RSMeM enhances remote sensing agents' tool usage accuracy by 6% through knowledge-enhanced memory evolution.
Key Findings
Methodology
RSMeM consists of Hierarchical Knowledge Grounding and Failure-Aware Experience Refinement. Hierarchical Knowledge Grounding performs taxonomy-aware retrieval to guide planning and tool selection. Failure-Aware Experience Refinement distills failure-annotated tool-use traces into reusable constraints. By iteratively applying these processes, RS agents can absorb task-level domain knowledge and effectively translate it into instance-level execution experience.
Key Results
- RSMeM achieves a 6% accuracy improvement on DeepSeek-V3.2 with less than 1% additional experience tokens, demonstrating high knowledge density.
- Extensive experiments on EarthBench show that RSMeM consistently improves tool-use performance and end-to-end answer quality across diverse LLM backbones.
- RSMeM outperforms baseline models in tool calling accuracy and result accuracy, especially in complex tasks.
Significance
RSMeM addresses the brittleness of existing remote sensing agents in complex tasks by integrating pre-distilled domain knowledge and online experience. It not only improves tool usage accuracy but also enhances execution capability, having significant academic and industrial impact.
Technical Contribution
RSMeM introduces a novel memory evolution framework by incorporating Hierarchical Knowledge Grounding and Failure-Aware Experience Refinement. This approach fundamentally differs from existing state-of-the-art methods, providing new theoretical guarantees and engineering possibilities.
Novelty
RSMeM is the first to apply knowledge-enhanced memory evolution to remote sensing agents, innovatively combining hierarchical knowledge retrieval with failure experience refinement, significantly improving execution capability in complex tasks.
Limitations
- RSMeM requires substantial manual intervention in knowledge base construction, which may affect scalability.
- The effectiveness of experience refinement may be limited by the quality of the initial knowledge base in some complex tasks.
- Further optimization may be needed for extremely complex tasks to enhance efficiency.
Future Work
Future work could include automating the construction of the knowledge base, enhancing the intelligence of experience refinement, and exploring applications in other domains.
AI Executive Summary
Geoscience research requires complex analysis and domain expertise, with remote sensing observations as a key foundation. However, existing remote sensing agents built on general-purpose LLMs remain largely domain-agnostic, resulting in brittle and error-prone workflows. To address this issue, RSMeM introduces a knowledge-enhanced memory evolution mechanism that bootstraps remote sensing agents with pre-distilled domain knowledge and iteratively integrates online experience for robust multi-step tool execution. RSMeM is composed of two components: Hierarchical Knowledge Grounding, which performs taxonomy-aware retrieval over a hierarchical domain corpus to guide planning and tool selection; and Failure-Aware Experience Refinement, which distills failure-annotated tool-use traces into reusable constraints for next-round tool execution. By iteratively employing these two processes, RS agents can evolve to absorb task-level domain knowledge and effectively translate it into instance-level execution experience. Extensive experiments on EarthBench demonstrate that RSMeM consistently improves tool-use performance and end-to-end answer quality across a diverse set of LLM backbones. Notably, RSMeM achieves a 6% accuracy improvement on DeepSeek-V3.2 with less than 1% additional experience tokens, demonstrating the strong knowledge density of our distilled experience. While RSMeM requires manual intervention in knowledge base construction, its performance in complex tasks is outstanding, offering significant potential for academic and industrial applications.
Deep Analysis
Background
Remote sensing technology plays a crucial role in geoscience research, supporting complex analysis and decision-making. Traditional remote sensing agents mostly rely on general-purpose LLMs, lacking domain specificity, resulting in poor performance in complex tasks. Recently, with the development of large language models and tool-augmented agents, remote sensing research has gradually moved towards automation and intelligence.
Core Problem
Existing remote sensing agents are prone to errors in complex tasks due to a lack of domain-specific knowledge and experience accumulation. This leads to brittle tool-use trajectories, prone to errors such as incorrect invocation or premature termination, affecting overall task execution.
Innovation
RSMeM innovatively addresses execution issues in remote sensing agents for complex tasks by introducing Hierarchical Knowledge Grounding and Failure-Aware Experience Refinement. Hierarchical Knowledge Grounding guides planning and tool selection through taxonomy-aware retrieval, while Failure-Aware Experience Refinement distills failure-annotated tool-use traces into reusable constraints.
Methodology
- �� Hierarchical Knowledge Grounding: Guides planning and tool selection through taxonomy-aware retrieval.
- �� Failure-Aware Experience Refinement: Distills failure-annotated tool-use traces into reusable constraints.
- �� Iterative Application: Continuously applies these processes to enhance execution capability.
Experiments
Experiments were conducted on EarthBench, using various LLM backbones including DeepSeek-V3.2, Kimi-K2, and Qwen3. The experimental setup includes evaluations across multiple dimensions such as tool calling accuracy, result accuracy, and efficiency.
Results
Experimental results show that RSMeM outperforms baseline models in tool calling accuracy and result accuracy, especially in complex tasks. RSMeM achieves a 6% accuracy improvement on DeepSeek-V3.2 with less than 1% additional experience tokens.
Applications
RSMeM can be applied to complex tasks in the geoscience field, such as disaster management and environmental monitoring. Its knowledge enhancement and experience refinement mechanisms make it perform excellently in these tasks, with significant application potential.
Limitations & Outlook
RSMeM requires manual intervention in knowledge base construction, which may affect scalability. In some complex tasks, the effectiveness of experience refinement may be limited by the quality of the initial knowledge base. Future work could include automating the construction of the knowledge base and enhancing the intelligence of experience refinement.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, needing various tools and ingredients to complete a meal. RSMeM is like a smart kitchen assistant that not only knows how to use each tool but also learns from past experiences to avoid repeating mistakes. For example, if you added too much salt to the soup before, the assistant would remind you to add less this time. This way, your cooking process becomes smoother, and you end up with a delicious dish. RSMeM helps remote sensing agents execute complex tasks better by combining knowledge and experience.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super complex game that requires a lot of tools to complete tasks. RSMeM is like an awesome game assistant that not only knows how to use each tool but also remembers the mistakes you made before, helping you avoid them. For instance, if you always forget to pick up the key in the game, the assistant will remind you next time. This way, you can finish the game faster! RSMeM helps remote sensing agents do their job better in complex tasks.
Glossary
RSMeM (Knowledge-Enhanced Memory Evolution)
A mechanism that enhances remote sensing agents' tool usage capability by combining pre-distilled domain knowledge and online experience.
RSMeM is used to improve remote sensing agents' execution capability in complex tasks.
Hierarchical Knowledge Grounding
A knowledge base that guides planning and tool selection through taxonomy-aware retrieval.
Used in RSMeM's knowledge enhancement part.
Failure-Aware Experience Refinement
The process of distilling failure-annotated tool-use traces into reusable constraints.
Used in RSMeM's experience refinement part.
EarthBench
A benchmark for evaluating tool-augmented Earth Observation agents.
RSMeM is evaluated on this benchmark.
DeepSeek-V3.2
An LLM backbone model used to evaluate RSMeM.
RSMeM achieves significant accuracy improvement on this model.
Open Questions Unanswered questions from this research
- 1 How to automate the construction of high-quality domain knowledge bases to reduce manual intervention and improve scalability.
- 2 How to further optimize RSMeM for extremely complex tasks to enhance efficiency and accuracy.
- 3 How to apply RSMeM's mechanisms to other domains and explore its cross-domain application potential.
Applications
Immediate Applications
Disaster Management
RSMeM can be used for real-time analysis and decision support, improving the efficiency and accuracy of disaster response.
Environmental Monitoring
By enhancing the analysis capability of remote sensing data, RSMeM can be used for monitoring and assessing long-term environmental changes.
Long-term Vision
Cross-Domain Applications
RSMeM's mechanisms can be extended to other fields, such as medical image analysis, to enhance intelligence levels.
Abstract
Geoscience research requires complex analysis and domain expertise, with remote sensing (RS) observations as a key foundation. However, existing RS agents built on general-purpose LLMs remain largely domain-agnostic, resulting in brittle and error-prone workflows. Moreover, these failures are seldom consolidated into a reusable experience for subsequent analyses. To address this issue, we introduce RSMeM, a knowledge-enhanced memory evolution mechanism that bootstraps RS agents with pre-distilled domain knowledge and iteratively integrates online experience for robust multi-step tool execution. RSMeM is composed of two components: (i) Hierarchical Knowledge Grounding, which performs taxonomy-aware retrieval over a hierarchical domain corpus to guide planning and tool selection; and (ii) Failure-Aware Experience Refinement, which distills failure-annotated tool-use traces into reusable constraints for next-round tool execution. By iteratively employing these two processes, RS agents can evolve to absorb task-level domain knowledge and effectively translate it into instance-level execution experience. Extensive experiments on EarthBench demonstrate that RSMeM consistently improves tool-use performance and end-to-end answer across a diverse set of LLM backbones. Notably, RSMeM achieves a 6% accuracy improvement on DeepSeek-V3.2 with less than 1% additional experience tokens, demonstrating the strong knowledge density of our distilled experience.