TaskGround: Structured Executable Task Inference for Full-Scene Household Reasoning
Proposes TaskGround framework for full-scene household reasoning, boosting success rates by over 20% and reducing input tokens by 18×.
Key Findings
Methodology
This paper introduces a training-free, model-agnostic Ground–Infer–Execute framework, TaskGround, which decomposes complete household scenes into compact, task-relevant slices. Scene graph representations encode all objects, relations, and states. The scene grounder extracts key entities based on language model queries, reducing input size significantly. The inference module predicts goal sequences from the grounded scene, leveraging household priors to fill in implicit intermediate steps, ensuring task completeness. The executor then converts the structured goals into grounded skill-level actions, such as 'WalkTo', 'Close', 'Clean', and 'Place'. This modular approach allows compatibility with multiple large language models (e.g., GPT-4, Qwen-3.5-9B) and is suitable for privacy-preserving, on-device deployment, addressing input cost and reasoning limitations of large models.
Key Results
- On the FullHome benchmark, TaskGround improves goal success rate by 20-30%, enabling Qwen3.5-9B to compete with GPT-5 under complete-scene prompting, while reducing total input tokens by up to 18×. The structured grounding approach significantly outperforms naive prompting, especially in complex scenes and privacy-sensitive environments.
- Ablation studies show scene grounding, task inference, and completion modules each contribute substantially, with combined use yielding the best performance. The method generalizes across models and environments, demonstrating robustness and practical utility.
- Results indicate that structured reasoning effectively bridges the gap between small, local models and large, proprietary systems, making household reasoning more accessible and efficient in real-world deployments.
Significance
This research addresses a critical bottleneck in household AI: reasoning over full, cluttered scenes with implicit goals and constraints. By enabling models to extract relevant information and infer complete task structures without task-specific training, it paves the way for more autonomous, privacy-preserving household robots. The approach reduces reliance on large, expensive models, making intelligent home automation more scalable and adaptable. It also advances scene understanding by integrating structured scene slicing, goal inference, and task completion, contributing to the broader field of embodied AI and human-centric robotics. Ultimately, this work facilitates smarter, more reliable household assistants capable of handling real-world complexity.
Technical Contribution
The core technical innovation is the development of a modular, training-free framework that combines scene graph-based scene slicing, goal prediction, and task completion, all within a unified pipeline. The scene grounder efficiently extracts task-relevant scene slices, drastically reducing input size while preserving critical context. The inference module predicts goal sequences, leveraging household priors to fill in implicit steps, thus enabling robust reasoning over incomplete or ambiguous natural language requests. The completion module augments the goal sequence with essential intermediate steps, ensuring executable actions. The executor then translates these structured goals into specific, grounded skill commands. This design allows compatibility with various large language models, supports privacy-preserving deployment, and enhances reasoning accuracy without additional training, representing a significant step forward in embodied AI.
Novelty
This work is the first to propose a fully modular, training-free approach for full-scene household reasoning that integrates scene slicing, goal inference, and task completion. Unlike prior methods relying on task-specific training or explicit goal annotations, it leverages scene graph representations and household priors to infer implicit intermediate steps. The scene cropping technique reduces input size by up to 18×, addressing the input bottleneck of large models. Its model-agnostic design enables broad applicability across different LLMs, making it highly adaptable for practical deployment. This combination of scene understanding, goal inference, and task completion in a unified, training-free pipeline is a novel contribution to embodied AI.
Limitations
- The approach depends heavily on accurate scene graph construction; errors in scene parsing can impair reasoning quality. Handling highly dynamic or cluttered scenes remains challenging, and the current method assumes static scene states during reasoning.
- The goal completion relies on fixed household priors, which may not generalize well to atypical or novel environments, limiting adaptability in diverse real-world homes.
- While model-agnostic, the framework still requires significant computational resources for scene graph processing and inference, especially in large or complex scenes, which could hinder real-time deployment.
Future Work
Future research will focus on integrating multi-modal sensory data (vision, audio) to improve scene understanding robustness. Developing adaptive priors for diverse household environments and extending reasoning to dynamic, real-time scenes are key directions. Additionally, optimizing the pipeline for edge deployment and reducing computational overhead will be crucial for practical, scalable household robots.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里准备一顿饭，但厨房里东西很多，有的你知道放在哪，有的则需要你自己找。你会先找出所有需要用的材料和工具，把重要的东西挑出来，然后按照菜谱一步步操作。这个过程就像家庭机器人在复杂环境中工作一样，它需要先筛选出关键信息（裁剪场景），再推断出下一步要做的事情（任务结构），最后按照计划行动。这个方法帮助机器人在信息繁杂的家庭环境中，像人一样聪明地完成任务，不用提前告诉它所有细节，也不用训练它专门做每个家庭的任务。它就像一个聪明的助手，能自己判断下一步该做什么，确保每个步骤都合理、顺利完成。
ELI14 Explained like you're 14
你知道在家里做事其实挺复杂的吧？你得找材料、准备工具，还要按顺序做。想象一下，你的机器人朋友也在帮你做事，但它不知道厨房里所有东西都在哪，也不知道你要做什么。这个研究就像教它先看一眼厨房，把重要的材料和工具都找出来，然后推断出下一步怎么做，比如先洗菜，再切菜，最后炒菜。它不用提前学会所有菜谱，只要能看懂厨房的场景，自己判断下一步。这样，机器人就能在不同家庭环境中灵活工作，不用每次都教它详细步骤，也不用事先准备好所有任务。它就像一个聪明的帮手，能自己判断怎么做，帮你节省时间，还能保护你的隐私，因为它不用把所有信息都告诉别人。
Abstract
In real home deployments, household agents must often operate from a complete household scene and a situated household request, rather than from a clean task specification. Such requests require agents to identify task-relevant entities, recover intended task conditions, and resolve ordering constraints from the surrounding scene context. We formalize this capability as full-scene household reasoning: given a complete household scene and a situated household request, an agent must infer executable task structure before producing a grounded skill-level action sequence. This setting is challenging because complete household scenes contain substantial task-irrelevant information, making direct complete-scene prompting inefficient and error-prone. In practical deployment, this challenge is further amplified by privacy and local compute constraints, which favor compact open-weight models with limited long-context reasoning ability. We propose TaskGround, a training-free and model-agnostic Ground-Infer-Execute framework that grounds complete scenes into compact task-relevant scene slices, infers executable task structure, and compiles it into grounded skill-level action sequences. To evaluate this setting, we introduce FullHome, a human-validated evaluation suite of 400 household tasks spanning diverse home-scale environments and both goal-oriented and process-constrained requirements. On FullHome, TaskGround improves task success rates by large margins across both proprietary and open-weight models. Notably, it makes Qwen3.5-9B competitive with GPT-5 under direct complete-scene prompting while reducing total input-token cost by up to 18x. Our results identify executable task-structure inference as a central bottleneck in full-scene household reasoning and show that structured grounding can make compact local models substantially more effective for practical household deployment.