CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graph
CuriousBot uses an actionable 3D relational graph for mobile interaction, achieving 82% average task success and outperforming 2D VLM baselines.
Key Findings
Methodology
CuriousBot combines RTAB-Map v0.21.4, YOLO-World, Segment Anything (SAM), and GPT-4o in a closed loop. RGB-D observations and odometry yield camera poses; 3D object nodes are built and associated across frames using label consistency and IoU, with a 0.15 threshold. Directed edges encode behind, of, inside, on, and under relations. GPT-4o reads a depth-first serialized graph and selects open, flip, lift, push, sit, or collect skills.
Key Results
- Across five tasks—flipping boxes, opening drawers, checking underneath, pushing boxes, and lifting cloth—each repeated ten times, the system achieved 82% overall success, 81.6% Object Recovery, and 1.28 Graph Editing Distance.
- CuriousBot reached 82% average success, versus 22% for LLaVA, 32% for Gemini, 32% for GPT-4o, and 12% for heuristics. Its average GED of 1.28 was also lower than baseline values ranging from 2.86 to 3.62.
- Task success was 90% for checking underneath and lifting cloth, 80% for flipping boxes, and 70% for pushing boxes. Among 50 rollouts, perception succeeded 47 times, planning 44 times, and action execution 41 times.
Significance
The paper shifts exploration from merely moving a camera to actively changing the environment to obtain information. This addresses persistent blind spots inside cabinets, beneath furniture, and behind obstacles. For service robots, an explicit graph is more inspectable and reusable than asking a VLM to remember a stream of images. Academically, the work links active perception, mobile manipulation, semantic mapping, and long-horizon planning in one system.
Technical Contribution
The central representation is an actionable graph G=(V,E). Nodes store labels, point clouds, normals, and obstruction status; edges encode directed spatial relations. A voxel map labels space as unexplored, free, unknown, or outside using ray-depth tests. Actions update relational knowledge: open/flip implies inside, lift or bottom-check implies under, and push implies behind. Thus, the graph represents both the current scene and how interaction can reveal hidden states.
Novelty
Compared with RoboEXP, CuriousBot addresses larger mobile spaces, richer relations, and combined navigation-manipulation actions. Compared with ConceptGraph and SceneGPT, it explicitly models occlusion and action consequences rather than only semantic or geometric structure. The authors position it among the first systems to combine interactive, mobile, and exploratory behavior in a single household-robot framework.
Limitations
- The skill library and task assumptions are narrow: boxes are flipped only when open, cloth is lifted, and large objects are pushed. Generalization to unknown mechanisms, clutter, and complex deformable objects remains untested.
- Of 50 trials, failures included three perception, three decision, and three action failures, caused by SLAM error, open-vocabulary detection error, wrong skill selection, loose grasping, early release, and unexpected interference.
Future Work
Future work should expand the skill library, replace heuristics with learned controllers, and improve uncertainty-aware planning. Longer-term directions include persistent memory, dynamic relation updates, stronger LLMs than GPT-4o, and evaluation in unconstrained homes with unseen objects, changing layouts, and multi-room tasks.
AI Executive Summary
Robots can often see household objects without being able to discover what is hidden behind them. Conventional exploration optimizes camera placement, while RoboEXP introduced interaction mainly in tabletop settings. CuriousBot instead treats exploration as physical information gathering: it opens cabinets, pushes chairs, lifts cloth, flips boxes, and sits down to inspect concealed regions.
The system contains RTAB-Map SLAM, a Graph Constructor, a GPT-4o Task Planner, and low-level skills. YOLO-World and SAM produce object masks and 3D point clouds; temporal association uses label consistency and IoU. The graph records behind, of, inside, on, and under relations, while a voxel map distinguishes unexplored, free, unknown, and outside space. GPT-4o receives a depth-first text serialization and outputs a skill and object ID, after which the graph is updated.
In 3m×4m rooms with 12 object categories and six layouts, five tasks were each repeated ten times. CuriousBot achieved 82% average success, 81.6% Object Recovery, and 1.28 GED, compared with 22%, 32%, 32%, and 12% success for LLaVA, Gemini, GPT-4o, and heuristics. The system remains limited by perception, handcrafted skills, and real-world manipulation reliability, but the results show that actionable 3D structure can turn an invisible region into an experimentally accessible one.
Deep Analysis
Background
Robotic exploration supports search and rescue, object search, and mobile manipulation. Active-perception methods mainly optimize camera viewpoints and unknown-area reduction, but cannot reveal objects hidden inside cabinets or beneath furniture. RoboEXP added interaction but focused on tabletop scenes. ConceptGraph and SceneGPT provide semantic scene structures without modeling how opening, pushing, or lifting changes observability. CuriousBot combines these missing capabilities for mobile settings.
Core Problem
The objective is to construct G=(V,E) while minimizing unknown space, discovering objects, and establishing correct relations. The challenge is not only localization and recognition; it is choosing among navigation and manipulation actions under occlusion. The robot must infer whether a cabinet, chair, box, or cloth obstructs useful information and select an action that maximizes exploration progress.
Innovation
- ��Actionable graph: nodes combine semantic and geometric attributes, while directed edges encode five household relations.
- ��Interaction-conditioned relations: open/flip, lift, and push produce inside, under, and behind evidence.
- ��Voxel occlusion model: unexplored, free, unknown, and outside labels expose hidden-space structure.
- ��Closed-loop mobile execution: GPT-4o plans from a serialized graph, Spot executes a skill, and the graph is updated to prevent repeated exploration.
Methodology
- ��Perception: RTAB-Map v0.21.4 estimates camera poses from RGB-D observations and odometry.
- ��Graph construction: YOLO-World detects objects and SAM segments them; depth converts masks into 3D point clouds. Same-label detections are associated using IoU, with values below 0.15 creating new nodes.
- ��Relations: 3D bounding-box tests create geometric relations such as on; action outcomes create inside, under, and behind edges.
- ��Occlusion: sampled camera rays compare predicted and measured depth to label voxels.
- ��Planning and execution: depth-first traversal serializes the graph; GPT-4o outputs a skill and target ID; Spot executes open, flip, lift, push, sit, or collect; the graph then removes the explored obstruction label.
Experiments
The platform uses a Boston Dynamics Spot with Spot API v4.0.0 and a front-mounted RealSense 455. Computation uses an Nvidia RTX A6000, AMD CPU, and 128GB RAM; graph construction runs at approximately 3Hz. Tests use 3m×4m rooms, 12 object categories, and six layouts. Five tasks are repeated ten times under different initial conditions. Baselines are LLaVA, Gemini, GPT-4o used as direct VLM guides, and a heuristic that opens all handles. Metrics are Success, Object Recovery, and GED.
Results
CuriousBot obtains 82% success, 81.6% OR, and 1.28 GED on average. LLaVA obtains 22%, 26%, and 3.62; Gemini 32%, 34%, and 3.42; GPT-4o 32%, 36%, and 3.32; heuristics 12%, 14.4%, and 2.86. CuriousBot reaches 90% on checking underneath and lifting cloth, 80% on flipping boxes, and 70% on pushing boxes. The failure breakdown shows that perception, planning, and execution remain distinct bottlenecks.
Applications
The system could support household object retrieval, cabinet inspection, room tidying, and searches beneath furniture. It may also help warehouse inventory robots inspect occluded items and disaster-response robots reveal blocked regions. Deployment requires RGB-D sensing, reliable localization, safe manipulation, a compatible skill library, and workspace constraints. The graph additionally provides an interpretable state for task planning.
Limitations & Outlook
The evaluation is small and controlled: 3m×4m rooms, 12 categories, and six layouts do not establish robustness in long-term homes. Skills rely on handcrafted heuristics and assumptions about object type and accessibility; failures in grasping or pushing propagate into later planning. Graph quality depends on RTAB-Map and YOLO-World. Future systems need learned skills, uncertainty-aware graph inference, dynamic-scene handling, safer recovery, and larger cross-home evaluations.
Plain Language Accessible to non-experts
Imagine a clever warehouse assistant searching for a package. An ordinary assistant can look at shelves or move its head, but if a chair blocks a box, it may never learn what is behind the chair. CuriousBot first draws a detailed inventory map: it records each item, its position, and which items hide others.
Then it behaves like a practical worker. It looks for a handle when a cabinet blocks the view, pushes a chair aside, lifts a cloth, or turns over an open box. After every move, it checks the room again and marks what has been seen, so it does not waste time repeating the same search.
In the study, it completed 82% of five search jobs on average, while systems that mainly interpreted pictures achieved only 22–32%. The lesson is simple: when information is blocked, better looking is not always enough; a robot must sometimes change the world safely in order to see more of it.
ELI14 Explained like you're 14
Imagine a video game treasure hunt in your bedroom. The treasure might be inside a cabinet, under a table, behind a chair, or inside a box. Taking screenshots will not reveal everything. CuriousBot is like a teammate that remembers where every object is and guesses which places might be hiding clues.
It uses a camera that can estimate distance, builds a room map, and asks a planning “captain” what to do next. The captain might say: open the cabinet, push the chair, lift the cloth, flip the box, or sit down to inspect under the table. After each move, the map changes so the robot knows what has already been checked.
Researchers tested five kinds of tasks ten times each. CuriousBot averaged 82% success, while LLaVA, Gemini, and GPT-4o used more directly as visual advisers scored only 22%, 32%, and 32%. So the important trick is not just understanding a picture; it is knowing when to take a useful action.
It is not perfect. Bad localization, mistaken object detection, weak grasps, or collisions can still ruin a mission. Its action menu is also limited. But give it more skills, safer recovery strategies, and messier real homes, and it could become a genuinely helpful robot that finds lost things instead of merely staring at them!
Glossary
Actionable 3D Relational Object Graph
A 3D representation whose nodes describe objects and whose edges describe their relations. Unlike a passive map, it also stores information relevant to actions and occlusion.
CuriousBot uses it as the interface between perception, planning, and execution.
Active Interaction
Information gathering by physically changing the environment, such as opening, pushing, lifting, or flipping. It differs from active perception, which mainly changes viewpoint.
The paper uses it to reveal concealed spaces.
Voxel Map
A three-dimensional grid in which small cells receive spatial-state labels. CuriousBot uses unexplored, free, unknown, and outside labels.
The map supplies occlusion cues and obstruction attributes.
Graph Editing Distance
The number of graph insertions, deletions, or substitutions needed to transform a predicted graph into a reference graph. Lower values indicate more accurate graph construction.
CuriousBot achieves an average GED of 1.28.
YOLO-World
An open-vocabulary object detector that can detect categories specified through language. It provides object candidates for graph construction.
Its detections are refined by SAM.
Segment Anything
A general-purpose segmentation model that extracts object masks from image prompts. Combining its masks with depth yields 3D point clouds.
SAM is part of the Graph Constructor.
Open Questions Unanswered questions from this research
- 1 Relation inference still relies heavily on hand-designed rules. It remains unclear whether a robot can learn new action-conditioned relations from successes and failures in open-world environments.
- 2 The controlled evaluation does not establish stability with people, moving furniture, reflective surfaces, dense clutter, or long-horizon multi-room missions.
- 3 GPT-4o improves over direct VLM baselines, but decision calibration, latency, cost, and safe behavior under uncertainty require systematic study.
Applications
Immediate Applications
Household cabinet retrieval
A service robot can scan a room with RGB-D sensing, identify cabinets, handles, and obstructed regions, then open doors and retrieve objects. It requires safe grasping, collision detection, and compatible manipulation skills; the expected benefit is fewer blind searches and higher object-recovery rates.
Warehouse occlusion inspection
An inventory robot can move blocking boxes, inspect shelf undersides, and update a relational map of stock. Reliable localization, force-limited pushing, and structured storage are prerequisites. The result could be more complete inventory records with interpretable evidence for each discovery.
Long-term Vision
Persistent household exploration assistant
A future robot could maintain cross-room relational memory, actively find objects, tidy spaces, and adapt to furniture changes. Major obstacles include open-world skill learning, uncertainty-aware safety, dynamic people, and reliable recovery from failed manipulation.
Abstract
Mobile exploration is a longstanding challenge in robotics, yet current methods primarily focus on active perception instead of active interaction, limiting the robot's ability to interact with and fully explore its environment. Existing robotic exploration approaches via active interaction are often restricted to tabletop scenes, neglecting the unique challenges posed by mobile exploration, such as large exploration spaces, complex action spaces, and diverse object relations. In this work, we introduce a 3D relational object graph that encodes diverse object relations and enables exploration through active interaction. We develop a system based on this representation and evaluate it across diverse scenes. Our qualitative and quantitative results demonstrate the system's effectiveness and generalization across object instances, relations, and scenes, outperforming methods solely relying on vision-language models (VLMs).