ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis

Over the past decade, scene understanding has evolved from simple geometric maps to rich semantic scene graphs, supporting tasks like navigation, object manipulation, and scene reasoning. Early approaches such as sparse object-based maps relied on predefined labels, limiting their flexibility. Recent advances leverage foundation models like CLIP and SAM for open-set semantic understanding, enabling recognition of novel concepts. However, these models often lack the ability to perform high-level task decomposition and environment-specific binding, especially in complex, dynamic scenes. Traditional hierarchical task analysis (HTA) methods depend heavily on manual design, which hampers scalability. This paper situates itself at the intersection of semantic scene graph construction, natural language understanding, and task planning, aiming to automate the process of environment-aware task decomposition grounded in 3D spatial structures.

The core challenge is bridging high-level natural language instructions with environment-specific, executable task hierarchies in complex 3D scenes. Existing methods either rely on predefined scene annotations or manual task decomposition, which are labor-intensive and environment-dependent. Automating this process requires a system that can understand the scene context, extract relevant objects, and generate a task hierarchy aligned with environment constraints. Furthermore, ensuring accurate spatial grounding of tasks and objects, especially in cluttered or dynamic environments, remains a significant obstacle. Addressing these issues is crucial for deploying autonomous agents capable of flexible, context-aware operation across diverse scenarios.

The paper introduces several key innovations: 1) Hierarchical Information Bottleneck (H-IB) for multi-resolution scene feature compression, enabling environment-dependent task hierarchy generation; 2) a novel iterative framework that combines scene graph construction with LLM-based natural language task analysis, forming a closed-loop system; 3) multi-modal embedding integration (visual and textual) for robust semantic grounding; 4) a spatially-informed conditional probability model that improves object-task association accuracy; 5) an automatic pruning and refinement process that dynamically adjusts scene and task hierarchies based on environment context. These innovations collectively enable fully automated, environment-aware task analysis and scene understanding.

• �� Extract primitives from RGB-D data using EfficientViT for segmentation and MobileCLIP for feature embedding.
• �� Associate primitives across frames via Hungarian matching based on visual and semantic similarity.
• �� Apply Hierarchical Information Bottleneck (H-IB) to compress primitives into multi-layer scene representations, guided by task hierarchy levels.
• �� Use large language models (LLMs) to analyze natural language instructions, decompose high-level tasks into subtasks, and generate environment bindings.
• �� Construct scene graph nodes corresponding to tasks, subtasks, and items, using the scene hierarchy and spatial attributes.
• �� Perform iterative scene hierarchy update by bottom-up H-IB compression and top-down pruning, refining task and scene structure.
• �� Incorporate spatial attributes into probabilistic models to improve object-task association accuracy.
• �� Use LLMs to refine task hierarchies based on scene graph feedback, enabling environment-dependent task planning.

Experiments utilize the SG3D HM3DSem dataset, comparing ASHiTA against baseline models in scene binding and task decomposition. Metrics include scene graph grounding accuracy and task hierarchy correctness. The evaluation covers various scene complexities, with ablation studies on the impact of multi-modal embeddings, H-IB layers, and LLM refinement. Hyperparameters such as the relevance thresholds (rs, rt=0.8) and the number of H-IB iterations (10-1000) are tuned for optimal performance. The experiments demonstrate the robustness of the approach across different environments and task types, validating the effectiveness of the iterative scene-task refinement process.

ASHiTA achieves 28.71% scene binding accuracy on the dataset, outperforming traditional zero-shot models by approximately 4%. When enhanced with text embeddings, accuracy increases to 65.4%, significantly surpassing baseline methods. The automatic task decomposition aligns well with human-annotated hierarchies, reducing manual effort. Ablation studies reveal that multi-modal fusion and multi-layer H-IB compression are critical for performance gains. The system maintains robustness in cluttered and complex scenes, demonstrating its potential for real-world deployment.

This framework can be integrated into autonomous robots for household chores, industrial inspection, and assistive systems, enabling natural language commands to be directly mapped to environment-specific actions. It reduces reliance on manual scene annotation and task scripting, making deployment scalable. The approach also benefits AR applications requiring environment-aware task guidance. Long-term, it could facilitate fully autonomous agents capable of learning new tasks and environments with minimal human intervention, advancing the fields of robotics, AI, and spatial cognition.

The current system relies on static scene representations, limiting real-time adaptation to dynamic environments. Its semantic understanding depends on pretraining datasets, which may not cover all domain-specific concepts. Computational complexity increases with scene size and hierarchy depth, impacting real-time performance. Handling highly dynamic or cluttered scenes remains challenging, requiring further algorithmic optimization and sensor integration. Future work should focus on dynamic scene updating, scalability, and robustness in real-world, unpredictable scenarios.