DolphinBench: Mapping the Pareto Frontier of Agent Memory
DolphinBench evaluates agent memory via task completion, achieving up to 70.67% accuracy.
Key Findings
Methodology
DolphinBench evaluates agent memory through simulated real-world tasks. It includes three personas: a startup CEO, an infrastructure engineer, and a product manager. Each persona has approximately 500k tokens of user message history, testing agents on tasks dependent on that history.
Key Results
- Achieved highest accuracy of 70.67% using Hermes and Mem0 system.
- Significant differences in cost and latency across configurations; Hermes and GPT-5.6-Luna combination performed best.
- Experiments show lower-cost configurations don't necessarily have lower accuracy.
Significance
This study evaluates memory through direct task completion, providing comprehensive reports on cost, latency, and accuracy. It fills a gap in existing memory benchmarks, offering a new method for holistic agent memory system evaluation.
Technical Contribution
DolphinBench combines task completion, cost, and latency reporting for comprehensive memory system evaluation. It verifies the solvability of each test, ensuring result reliability.
Novelty
First to evaluate memory through task completion rather than simple Q&A, offering more realistic agent usage scenarios. Adds cost and latency evaluation compared to existing benchmarks.
Limitations
- Dataset is synthetic rather than real user data, may not fully reflect real writing styles.
- Only covers three personas, excluding domains like law and medicine.
- Agents interact with simulated apps, not testing real system failures.
Future Work
Future expansion to more personas, longer tasks, and testing agent memory handling in continuous dialogue.
AI Executive Summary
DolphinBench is a novel benchmark that evaluates agent memory through simulated real-world tasks. Existing memory benchmarks often rely on Q&A formats, failing to comprehensively assess agent performance in real tasks. DolphinBench defines three personas, simulates years of conversation history, and tests agents on tasks dependent on that history, providing a more realistic evaluation.
Experimental results show that agents using Hermes and Mem0 systems performed best in 70.67% of tasks. The benchmark not only evaluates accuracy but also reports cost and latency, making the evaluation more comprehensive. Significant differences in cost and latency across configurations highlight the impact of configuration choices on agent performance.
DolphinBench's innovation lies in combining task completion, cost, and latency reporting for comprehensive memory system evaluation. Future work can expand to more personas, longer tasks, and test agent memory handling in continuous dialogue to better simulate real-world usage scenarios.
Deep Analysis
Background
As agents are increasingly used in software engineering, customer service, and other fields, evaluating their memory capabilities becomes crucial. Existing memory benchmarks mostly rely on Q&A formats, failing to comprehensively assess agent performance in real tasks. DolphinBench fills this gap by evaluating agent memory through simulated real-world tasks.
Core Problem
Existing memory benchmarks rely on Q&A formats, failing to comprehensively assess agent performance in real tasks. A method that evaluates agent memory through direct task completion is needed.
Innovation
DolphinBench defines three personas, simulates years of conversation history, and tests agents on tasks dependent on that history. It combines task completion, cost, and latency reporting for comprehensive memory system evaluation.
Methodology
- �� Define three personas: startup CEO, infrastructure engineer, and product manager.
- �� Simulate years of conversation history, generating approximately 500k tokens of user messages.
- �� Test agents on tasks dependent on that history.
- �� Report cost, latency, and accuracy.
Experiments
Experiments used multiple agent configurations, including Hermes, Mem0, Honcho, etc. Tested each configuration's accuracy, cost, and latency. Evaluated agent memory system performance through configuration comparisons.
Results
Achieved highest accuracy of 70.67% using Hermes and Mem0 system. Significant differences in cost and latency across configurations highlight the impact of configuration choices on agent performance. Lower-cost configurations don't necessarily have lower accuracy.
Applications
Can be used to evaluate agent memory capabilities in real-world tasks. Applicable to software engineering, customer service, and other fields, helping choose the best agent configuration.
Limitations & Outlook
Dataset is synthetic rather than real user data, may not fully reflect real writing styles. Only covers three personas, excluding domains like law and medicine. Agents interact with simulated apps, not testing real system failures.
Plain Language Accessible to non-experts
Imagine you're in a kitchen, DolphinBench is like a chef assistant. It not only remembers your favorite ingredients but also helps you make delicious dishes based on past conversations. Existing assistants only ask what you want to eat, but DolphinBench automatically prepares everything based on your habits and past choices. It considers not just the taste but also tells you how much time and cost it takes to make the dish.
ELI14 Explained like you're 14
Hey, imagine you're playing a game where your character needs to remember past tasks to complete new challenges. DolphinBench is like a super memory assistant, helping you remember all the important information. It not only tells you how to complete tasks but also how much time and resources it takes. With it, you can complete all the challenges in the game faster and better!
Glossary
DolphinBench
A benchmark for evaluating agent memory capabilities through task completion.
Used to test agent memory performance in simulated tasks.
Mem0
A memory system designed to enhance agent performance in tasks.
One of the best-performing memory systems in experiments.
Hermes
An agent framework used to test agent memory capabilities.
Performed best when combined with Mem0.
GPT-5.6-Luna
A language model used for agent task completion.
One of the language models used in experiments.
Honcho
A memory infrastructure supporting agent state management.
Performed well in certain configurations.
Open Questions Unanswered questions from this research
- 1 How to validate DolphinBench's effectiveness on real user data?
- 2 How to expand to more domains like law and medicine?
- 3 How to test agent memory handling in continuous dialogue?
Applications
Immediate Applications
Software Engineering
Helps developers choose the best agent configuration, enhancing project management efficiency.
Customer Service
Improves response speed and accuracy of customer service agents through memory systems.
Long-term Vision
Cross-domain Applications
Expand to domains like law and medicine, providing broader memory evaluation.
Abstract
Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are built for a conversational question-answer format, where the question itself signals that some fact must be retrieved, and often which one. Moreover, benchmarks rarely require anything beyond accuracy from submissions, allowing memory systems to make unreasonable cost/time tradeoffs to achieve higher scores. We present DolphinBench, a benchmark that evaluates memory directly through an agent's task completion. DolphinBench includes three knowledge-work personas with roughly 500k tokens of user messages per persona and evaluates agents on tasks that depend on information from that history. We verify all 200 tasks per persona by running an agent with and without the relevant history, requiring success with it and failure without it. Finally, we require all evaluations to report total cost and latency alongside accuracy, which enables us to evaluate agent memory systems holistically. No existing memory benchmark combines all three. The dataset and evaluation code are available at https://dolphinbench.ai.