Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments

Understanding complex environments is a fundamental challenge for autonomous robots, especially in open, unstructured spaces where traditional geometric heuristics and local perception methods often fall short. Existing approaches typically rely on fixed assumptions—such as wall-separated rooms or street topologies—and struggle to adapt to environments with irregular layouts or distant, partially observed regions. These limitations hinder robots' ability to perform high-level reasoning, environment segmentation, and task-driven spatial reconfiguration.

In response, this paper introduces Prior-SG, a novel framework that reconceptualizes scene graph generation as a probabilistic alignment problem. Inspired by human spatial cognition, the system leverages a dynamically generated environment prior—produced by a large language model (LLM)—to encode expectations about environment structure and semantics. This prior acts as a structured probabilistic template, guiding the fusion of sensory observations with high-level environment expectations.

The core technical components include a multi-scale feature pyramid that extracts open-vocabulary descriptors from RGB images, capturing semantic and geometric cues at various levels of detail. These features are used to incrementally build an instance graph (Instance Graph), representing the physical environment through detected objects and navigable places. Simultaneously, the prior graph encodes environment hypotheses such as typical room layouts, object co-occurrences, and topological relations, all conditioned on the environment type and task.

Global inference is performed via a Markov Random Field (MRF) framework, where graph cuts optimize the alignment between observed data and the prior expectations. This process effectively resolves perceptual ambiguities, especially in environments lacking explicit physical boundaries. The system’s flexibility allows it to adapt to different tasks, such as navigation or environment reconfiguration, by dynamically updating the prior graph and spatial partitions.

Experimental results demonstrate that Prior-SG achieves state-of-the-art segmentation accuracy on simulated residential datasets, with an average IoU exceeding 87%, outperforming baseline models like Clio by approximately 8%. In real-world large-scale environments, such as train stations, the system accurately delineates functional zones with errors below 5 meters, even in the absence of physical walls. The ability to perform zero-shot environment restructuring based on high-level natural language commands showcases its potential for versatile autonomous applications.

This approach significantly advances the field by integrating semantic priors with sensory data within a probabilistic framework, enabling robots to interpret complex environments more robustly and flexibly. Its capacity for environment reconfiguration based solely on high-level task descriptions opens new avenues for autonomous navigation, human-robot interaction, and spatial reasoning. Despite current limitations related to computational costs and environment dynamics, future work aims to optimize efficiency, extend to dynamic scenes, and incorporate multimodal sensing, promising a new generation of intelligent, adaptable autonomous systems.