EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models
EgoMem enables personalized dialogues with full-duplex models, achieving over 95% accuracy in user recognition.
Key Findings
Methodology
EgoMem operates with three asynchronous processes: user identification, dialogue generation, and memory management. User identification is done via face and voice, retrieving relevant context from long-term memory. Dialogue generation creates personalized audio responses based on retrieved context. Memory management detects dialogue boundaries from multimodal streams and extracts information to update long-term memory.
Key Results
- EgoMem's retrieval and memory management modules achieve over 95% accuracy on the test set.
- Integrated with the RoboEgo chatbot, the system achieves fact-consistency scores above 87% in real-time personalized dialogues.
- EgoMem effectively manages user social relation graphs in multi-user scenarios.
Significance
EgoMem holds significant implications for academia and industry. It addresses long-standing challenges in multi-user recognition and personalized service, particularly suitable for robots in homes and public spaces. By leveraging raw audiovisual streams, EgoMem offers a novel solution for real-time personalized dialogues.
Technical Contribution
EgoMem fundamentally differs from existing methods by relying entirely on raw audiovisual streams rather than text inputs, making it suitable for real-time and embodied scenarios. It also introduces management of multi-user social relation graphs, which is overlooked in existing systems.
Novelty
EgoMem is the first lifelong memory system tailored for full-duplex multimodal scenarios. Unlike existing methods, it does not rely on explicit user identities and dialogue boundaries, extracting information directly from audiovisual streams.
Limitations
- In noisy environments, user recognition accuracy may decline.
- Dialogue generation depends on the quality of pre-trained models.
Future Work
Future research directions include extending EgoMem's memory system to encompass procedural memory and multimodal contents, and exploring larger model scales and more complex functionalities.
AI Executive Summary
EgoMem is the first lifelong memory agent designed for full-duplex multimodal models, addressing challenges in real-time personalized dialogues. Existing solutions often rely on text inputs and cannot handle the ever-growing audiovisual streams. EgoMem achieves user identification, dialogue generation, and memory management through three asynchronous processes, significantly improving the accuracy of personalized dialogues.
Experimental results show that EgoMem's retrieval and memory management modules achieve over 95% accuracy on the test set. Integrated with the RoboEgo chatbot, the system achieves fact-consistency scores above 87% in real-time personalized dialogues. These results demonstrate EgoMem's effectiveness in managing user social relation graphs in multi-user scenarios, providing a strong baseline.
While EgoMem excels in personalized dialogues, user recognition accuracy may be affected in noisy environments. Future research will focus on extending the memory system to encompass more types of memory and exploring larger model scales and more complex functionalities.
Deep Analysis
Background
With the advancement of AI technology, lifelong multimodal streams are increasingly applied in homes and public spaces. These applications require models not only to respond to instructions swiftly but also to recognize users, remember their histories, understand social relationships, and deliver personalized services. Existing solutions often rely on text inputs and cannot handle the ever-growing audiovisual streams.
Core Problem
Existing memory agents typically assume explicit user identities, clear dialogue session boundaries, and all inputs being textual. In full-duplex multimodal applications, these assumptions do not hold as user identities are implicitly encoded in audiovisual streams, and there are no well-defined boundaries for dialogue turns or user sessions.
Innovation
EgoMem achieves user identification, dialogue generation, and memory management through three asynchronous processes. User identification is done via face and voice, retrieving relevant context from long-term memory. Dialogue generation creates personalized audio responses based on retrieved context. Memory management detects dialogue boundaries from multimodal streams and extracts information to update long-term memory.
Methodology
- �� User Identification: Done via face and voice, retrieving relevant context from long-term memory.
- �� Dialogue Generation: Creates personalized audio responses based on retrieved context.
- �� Memory Management: Detects dialogue boundaries from multimodal streams and extracts information to update long-term memory.
Experiments
The experimental design includes testing EgoMem's retrieval and memory management modules' accuracy on multiple datasets. The test set includes different user scenarios to evaluate the system's performance in multi-user environments. Experiments also include integration tests with the RoboEgo chatbot to assess the system's performance in real-time personalized dialogues.
Results
Experimental results show that EgoMem's retrieval and memory management modules achieve over 95% accuracy on the test set. Integrated with the RoboEgo chatbot, the system achieves fact-consistency scores above 87% in real-time personalized dialogues. These results demonstrate EgoMem's effectiveness in managing user social relation graphs in multi-user scenarios.
Applications
EgoMem is applicable in robots for homes and public spaces, capable of real-time user recognition and personalized services. The system's multi-user management feature is particularly suited for scenarios requiring complex social relationship handling.
Limitations & Outlook
While EgoMem excels in personalized dialogues, user recognition accuracy may be affected in noisy environments. Dialogue generation depends on the quality of pre-trained models and may perform poorly in certain cases. Future research will focus on extending the memory system to encompass more types of memory and exploring larger model scales and more complex functionalities.
Plain Language Accessible to non-experts
Imagine you have a super smart robot assistant that remembers everything you say and your preferences. This assistant can not only understand what you say but also recognize who you are by observing your facial expressions and listening to your voice. It's like having a friend who knows what you like and dislike and can give you advice when you need it. EgoMem is such a system, capable of recognizing different users in various scenarios and providing personalized services.
ELI14 Explained like you're 14
Imagine having a super cool robot friend that remembers all your likes and habits. Every time you talk to it, it recognizes you by your voice and face. It's like when playing games, it knows which character you like to use and can give you the best advice. EgoMem is such a system, recognizing different users in various scenarios and providing personalized services. Isn't that cool?
Glossary
EgoMem
EgoMem is a lifelong memory agent designed for full-duplex multimodal models.
Used for user recognition and personalized services.
Full-duplex
Full-duplex refers to the ability to send and receive data simultaneously.
Describes EgoMem's capability to handle real-time audiovisual streams.
Omnimodal
Omnimodal refers to the ability to process multiple data types, such as audio and video.
EgoMem uses omnimodal streams for user recognition.
RoboEgo
RoboEgo is a multimodal chatbot integrated with EgoMem for personalized dialogues.
Used to evaluate EgoMem's personalized dialogue capabilities.
Memory Management
Memory management is responsible for updating and maintaining the system's long-term memory.
EgoMem uses memory management to achieve long-term storage of user information.
Open Questions Unanswered questions from this research
- 1 How to improve user recognition accuracy in noisy environments?
- 2 How to extend EgoMem to support more types of memory?
Applications
Immediate Applications
Home Assistant
EgoMem can be used in home robots to provide personalized household management and entertainment suggestions.
Long-term Vision
Public Service Robots
EgoMem can be used in public service robots to provide personalized consultation and guidance services.
Abstract
We introduce EgoMem, the first lifelong memory agent tailored for full-duplex models that process real-time omnimodal streams. EgoMem enables real-time models to recognize multiple users directly from raw audiovisual streams, to provide personalized response, and to maintain long-term knowledge of users' facts, preferences, and social relationships extracted from audiovisual history. EgoMem operates with three asynchronous processes: (i) a retrieval process that dynamically identifies user via face and voice, and gathers relevant context from a long-term memory; (ii) an omnimodal dialog process that generates personalized audio responses based on the retrieved context; and (iii) a memory management process that automatically detects dialog boundaries from omnimodal streams, and extracts necessary information to update the long-term memory. Unlike existing memory agents for LLMs, EgoMem relies entirely on raw audiovisual streams, making it especially suitable for lifelong, real-time, and embodied scenarios. Experimental results demonstrate that EgoMem's retrieval and memory management modules achieve over 95% accuracy on the test set. When integrated with a fine-tuned RoboEgo omnimodal chatbot, the system achieves fact-consistency scores above 87% in real-time personalized dialogs, establishing a strong baseline for future research.