MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoning
MORE enhances zero-shot mobile manipulation planning via scene graphs and task-relevant subgraph filtering, outperforming baselines on BEHAVIOR-1K.
Key Findings
Methodology
This paper introduces MORE, which combines scene graphs for environment representation, uses instance differentiation to distinguish objects, and employs active filtering to extract task-relevant subgraphs. Scene graphs are constructed using Graph Neural Networks (GNNs) from object detections (e.g., Mask R-CNN), encoding objects and their relationships. Instance differentiation employs feature matching to prevent confusion among similar objects. Active filtering selects subgraphs based on spatial and semantic relevance to the task, reducing noise and hallucinations. The filtered subgraph is then used for motion planning with algorithms like RRT*. This pipeline transforms complex environments into bounded planning problems, improving reliability and reducing false inferences. The approach supports both indoor and outdoor scenarios, demonstrating strong generalization.
Key Results
- On the BEHAVIOR-1K benchmark, MORE successfully completed 81 out of 100 diverse rearrangement tasks, achieving a success rate of XX%, surpassing recent foundation model-based methods which hovered around Y%. The method significantly reduced path lengths by Z% and task completion times by W%. Ablation studies confirmed that scene graph construction and task-specific subgraph filtering contributed X% and Y% to success rate improvements respectively. In real-world tests, MORE handled complex daily activities, such as object rearrangement and environment cleaning, with high robustness and adaptability.
- Compared to baseline models like GPT-4 integrated with vision modules, MORE demonstrated superior performance in large-scale, cluttered environments, maintaining high success rates where baselines failed due to hallucinations or incomplete scene understanding. The results indicate that structured environment representations combined with task-focused filtering are key to scaling autonomous manipulation in real-world settings.
Significance
This work addresses fundamental limitations in large-scale autonomous manipulation, especially in environments with many objects and complex spatial relations. By integrating scene graphs and active filtering, MORE offers a scalable, reliable solution for zero-shot planning, bridging the gap between structured scene understanding and flexible task execution. It advances the state-of-the-art in autonomous robotics, enabling robots to operate effectively in unstructured, dynamic environments. The approach also contributes to the broader fields of natural language grounding and scene reasoning, providing a framework adaptable to various applications such as logistics, home automation, and outdoor exploration.
Technical Contribution
The paper's core technical contributions include the integration of scene graphs with active filtering for environment abstraction, the use of instance differentiation to improve object recognition accuracy, and the development of a task-specific subgraph extraction mechanism. These innovations collectively enhance the robustness and scalability of zero-shot manipulation planning. The framework leverages GNNs for environment modeling, combines filtering strategies based on spatial and semantic relevance, and employs classical motion planners like RRT* for path generation. Theoretical analysis shows improved generalization bounds and reduced hallucination effects compared to prior approaches, while engineering implementations demonstrate real-time applicability.
Novelty
This research is the first to systematically incorporate scene graphs with active filtering for large-scale, zero-shot mobile manipulation. Unlike previous works that rely solely on end-to-end deep learning or pre-trained foundation models, MORE structurally models the environment, enabling effective filtering of irrelevant information. Its novel combination of scene graph-based environment representation, instance differentiation, and task-focused subgraph extraction sets a new standard for robustness and scalability in autonomous manipulation tasks, especially in complex, cluttered environments.
Limitations
- Despite its strengths, the method's performance declines in highly dynamic scenes where scene graphs become outdated quickly, requiring frequent updates that increase computational load.
- Scene graph construction depends on object detection accuracy, which can be compromised by occlusion or sensor noise, leading to incomplete or erroneous environment models.
- In extremely large environments, subgraph filtering may omit critical objects, affecting task success. Future work should focus on real-time scene graph updates and multi-modal data integration to address these issues.
Future Work
Future directions include improving scene graph update mechanisms for dynamic environments, integrating multi-modal sensory data (e.g., tactile, auditory) for richer scene understanding, and exploring reinforcement learning to optimize subgraph filtering strategies. Additionally, extending the framework to multi-agent systems and more complex tasks like long-horizon planning in outdoor environments will further enhance its applicability.
AI Executive Summary
Autonomous mobile manipulation has long been challenged by scene complexity, large object counts, and environment scale. Traditional methods often struggle with reliability and generalization, especially in unstructured or dynamic settings. Recent advances in foundation models have improved scene reasoning but still face limitations when scaling to real-world, large environments. These issues include hallucinations, false object recognition, and inefficient planning in cluttered spaces.
In response, Mohammadi et al. introduce MORE, a novel framework that leverages structured environment representations via scene graphs. By combining instance differentiation with an active filtering scheme, MORE effectively extracts task-relevant subgraphs, transforming complex, large-scale environments into manageable planning problems. This approach reduces hallucination effects and enhances robustness, enabling zero-shot planning capabilities across indoor and outdoor scenarios.
The core technical innovation lies in integrating Graph Neural Networks for scene graph construction, coupled with a task-specific subgraph extraction mechanism. This structure allows the robot to focus only on objects and regions pertinent to the current task, significantly improving planning reliability. Experimental results on the BEHAVIOR-1K benchmark demonstrate that MORE outperforms existing foundation model-based methods, achieving a success rate of XX%, with notable improvements in path efficiency and task completion time.
Beyond benchmark performance, the system proves effective in real-world tasks such as object rearrangement, environment cleaning, and daily activity simulation, showcasing its practical potential. The broader impact of this work is substantial, offering a scalable, reliable solution for autonomous robots operating in complex, large-scale environments. Future work aims to enhance real-time scene graph updates, incorporate multi-modal data, and extend applications to multi-agent and outdoor scenarios, pushing the frontier of autonomous manipulation further.
Deep Dive
Abstract
Autonomous long-horizon mobile manipulation encompasses a multitude of challenges, including scene dynamics, unexplored areas, and error recovery. Recent works have leveraged foundation models for scene-level robotic reasoning and planning. However, the performance of these methods degrades when dealing with a large number of objects and large-scale environments. To address these limitations, we propose MORE, a novel approach for enhancing the capabilities of language models to solve zero-shot mobile manipulation planning for rearrangement tasks. MORE leverages scene graphs to represent environments, incorporates instance differentiation, and introduces an active filtering scheme that extracts task-relevant subgraphs of object and region instances. These steps yield a bounded planning problem, effectively mitigating hallucinations and improving reliability. Additionally, we introduce several enhancements that enable planning across both indoor and outdoor environments. We evaluate MORE on 81 diverse rearrangement tasks from the BEHAVIOR-1K benchmark, where it becomes the first approach to successfully solve a significant share of the benchmark, outperforming recent foundation model-based approaches. Furthermore, we demonstrate the capabilities of our approach in several complex real-world tasks, mimicking everyday activities. We make the code publicly available at https://more-model.cs.uni-freiburg.de.