EmbodiedLGR: Integrating Lightweight Graph Representation and Retrieval for Semantic-Spatial Memory in Robotic Agents

As artificial intelligence technology continues to evolve, the ability of robots to autonomously operate in complex environments has become a research hotspot. Traditional robotic systems primarily rely on simple reactive command execution, making it difficult to achieve a deep understanding and memory of the environment. In recent years, with the rise of large-scale language models (LLMs) and visual-language models (VLMs), researchers have begun exploring how to apply these models to robotic systems to enhance their semantic-spatial memory capabilities. Many studies have made progress in translating visual observations into structured semantic maps, such as CLIP-fields and Visual Language Maps. However, these methods perform poorly in real-time robotic scenarios because they are not optimized for data representation, resulting in high computational overhead and response latency.

In complex robotic operating environments, efficiently constructing and retrieving semantic-spatial memory is a core problem. Existing methods perform poorly in handling redundancies and repetitions among semantic concepts, leading to high computational overhead and response latency. Additionally, many methods rely on computationally expensive models, making it difficult to achieve fast querying and reasoning in real-time scenarios. Solving this problem is crucial for achieving natural human-robot interaction, as people expect robots to provide precise answers within human-like inference times.

EmbodiedLGR-Agent offers an innovative solution by integrating lightweight graph representation and retrieval. Its core innovations include:

• �� Utilizing a visual-language model (VLM)-driven agent architecture to construct a dense and efficient representation of robot operating environments.

• �� Leveraging parameter-efficient VLMs to store low-level information about objects and their positions in a semantic graph, while retaining high-level descriptions of observed scenes with a traditional retrieval-augmented architecture.

• �� Employing a hybrid building-retrieval method that can store low-level information in a semantic graph while maintaining high-level descriptions in a vector database.

These innovations not only significantly reduce computational overhead but also enhance system responsiveness.

The methodology of EmbodiedLGR-Agent includes the following key steps:

• �� Memory Building: Begins with the collection of image frames, positions, and timestamps from the robot, processed by a VLM to extract objects and their visual descriptions from the scene.

• �� Memory Graph Population and Update: Object labels and frame descriptions are used to populate the memory graph and vector database. Multiple perceptions of the same object are updated in the memory graph, avoiding duplicate entries.

• �� Memory Retrieval: Defines three on-graph search tools: semantic, positional, and temporal search. Depending on the complexity of the user query, the LLM agent can also invoke querying tools on the vector database to provide detailed answers.

• �� Inference Process: Upon receiving a user query, the LLM enters a reasoning loop, invoking memory retrieval tools until it can provide an answer.

In the experimental design, EmbodiedLGR-Agent is evaluated on the NaVQA dataset to test its ability to build and retrieve memory for navigation-related tasks. The experiments use Florence-2-base and Florence-2-large models to test object labels and visual captions generation. In the experiments, the system excels in handling simple, atomic queries, while the integration of the vector database improves overall accuracy for semantically complex queries. The experiments also test the system's deployment on a physical robot, demonstrating its practical utility in real-world contexts.

The experimental results show that EmbodiedLGR-Agent achieved state-of-the-art performance in inference and querying times on the NaVQA dataset, while maintaining competitive accuracy on the global task relative to current state-of-the-art approaches. Specifically, the method significantly outperformed vector database query times based on ReMEmbR, with response latency reduced by half. Additionally, the graph memory component of EmbodiedLGR-Agent excelled in handling simple, atomic queries, while the integration of the vector database improved overall accuracy for semantically complex queries.

EmbodiedLGR-Agent has potential in multiple application scenarios. Its application in robot navigation and human-robot interaction is particularly prominent, enabling rapid and precise information retrieval in complex environments. Additionally, the method can be applied in smart homes, autonomous driving, and other fields, helping systems better understand and remember environmental information.

Despite the outstanding performance of EmbodiedLGR-Agent in multiple aspects, there are still some limitations. First, the system may experience update lags when handling dynamic entities. Second, the method relies on the vector database for semantically complex queries, which may increase response latency in certain cases. Additionally, the system's performance heavily depends on the accuracy and efficiency of the visual-language model. Future research directions include optimizing the memory graph update mechanism, exploring more efficient VLMs, and expanding the system's application scenarios.