Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infilling
Multi-task SceneScript model with infilling enhances 3D scene layout local correction, achieving 98.6% F1 in local tasks.
Key Findings
Methodology
This work introduces a multi-task SceneScript framework that combines global scene prediction with local correction via infilling. The model employs a sparse 3D ResNet encoder and a Transformer decoder, trained jointly to perform both tasks. Local correction is formulated as an infilling problem, where selected scene entities are masked and regenerated conditioned on egocentric pose, point cloud, and surrounding layout. Egocentric anchoring and subsequence positional embeddings improve interaction efficiency. The system is integrated into a mixed reality environment, enabling users to identify errors via a single click and trigger automatic local fixes, iteratively refining scene layouts.
Key Results
- On the ASE synthetic dataset, the model achieved a global prediction F1 of 92.1%, with local correction F1 reaching 98.6%, a 7-point improvement over single-task models. Cross-domain tests on AEO showed a slight drop (~1 point) in global prediction but a 10-point increase in local correction performance, demonstrating strong generalization.
- Ablation studies confirmed that angle-based sorting and subsequence positional embeddings significantly reduced user interactions, enhancing correction efficiency. The multi-task training maintained global prediction performance while boosting local correction capabilities.
- The system effectively models complex real-world scenes, allowing iterative refinement with minimal user effort, and surpasses prior approaches lacking interactive local correction mechanisms.
Significance
This research addresses the critical challenge of refining 3D scene layouts in real-world applications, where single-shot predictions often fail in complex or out-of-distribution environments. By integrating human feedback with infilling-based local correction, the method enables more accurate, flexible, and user-friendly scene modeling. It bridges the gap between autonomous scene understanding and interactive editing, paving the way for advanced AR/VR applications, robotic navigation, and architectural design tools that require precise, adaptable scene representations.
Technical Contribution
The paper introduces a novel multi-task training paradigm for SceneScript, leveraging infilling techniques from NLP to enable local corrections in 3D scene layouts. It innovatively combines egocentric anchoring, subsequence positional embeddings, and structured language commands within a Transformer-based architecture. The system supports low-friction human-in-the-loop interaction, allowing users to iteratively refine complex scenes efficiently. This approach extends the capabilities of existing scene understanding models by enabling targeted, high-precision local edits without sacrificing global prediction accuracy.
Novelty
This work is the first to formulate 3D scene layout correction as an infilling task inspired by NLP, integrating it into a multi-task learning framework. It uniquely combines egocentric spatial anchoring and structured language commands to facilitate efficient human-in-the-loop scene editing. Unlike prior methods limited to global predictions, this approach allows for targeted, iterative local corrections, significantly improving modeling of complex, real-world scenes.
Limitations
- The model's performance degrades in highly cluttered or irregular environments due to limited training data coverage and scene complexity.
- Dependence on structured language commands and sparse point clouds may limit applicability in unstructured or low-quality sensing scenarios.
- User identification of errors still requires manual effort; future work should incorporate automatic error detection and correction mechanisms.
Future Work
Future directions include integrating automatic error detection, expanding to more scene primitives, and improving robustness in diverse environments. Enhancing real-time performance and reducing user effort through better interface design are also key goals. Further research may explore unsupervised or semi-supervised training to handle more complex, real-world scenes, advancing the practical deployment of interactive scene understanding systems.
AI Executive Summary
This paper presents a groundbreaking approach to 3D scene layout estimation that combines structured language modeling with human-in-the-loop interaction. Traditional methods rely on single-shot global predictions, which often struggle with complex, cluttered, or out-of-distribution environments. To overcome these limitations, the authors propose a multi-task SceneScript model trained to perform both global scene prediction and local correction via infilling, a technique borrowed from NLP. The core idea is to enable users to iteratively refine scene layouts by identifying errors with a simple click, prompting the model to automatically generate corrected entities in a targeted manner.
The system employs a sparse 3D ResNet encoder and a Transformer decoder, trained jointly to balance global accuracy and local correction capabilities. Local correction is formulated as an infilling task, where selected scene entities are masked and regenerated conditioned on egocentric pose, surrounding layout, and point cloud data. Egocentric anchoring and subsequence positional embeddings are key innovations that improve interaction efficiency, reducing user effort.
Experimental results on synthetic ASE and real-world AEO datasets demonstrate the effectiveness of the approach. The model achieves a global prediction F1 of 92.1% and a local correction F1 of 98.6%, outperforming baseline single-task models. Ablation studies confirm the importance of sorting strategies and positional embeddings. The integrated system allows for rapid, precise scene refinement, making it highly suitable for AR/VR applications, architectural design, and robotic navigation.
Overall, this work advances the state-of-the-art in interactive 3D scene understanding, enabling more accurate, flexible, and user-friendly scene modeling. Future work will focus on automatic error detection, broader scene primitives, and real-time deployment, further bridging the gap between autonomous perception and human-guided editing.
Deep Dive
Abstract
We present a novel human-in-the-loop approach to estimate 3D scene layout that uses human feedback from an egocentric standpoint. We study this approach through introduction of a novel local correction task, where users identify local errors and prompt a model to automatically correct them. Building on SceneScript, a state-of-the-art framework for 3D scene layout estimation that leverages structured language, we propose a solution that structures this problem as "infilling", a task studied in natural language processing. We train a multi-task version of SceneScript that maintains performance on global predictions while significantly improving its local correction ability. We integrate this into a human-in-the-loop system, enabling a user to iteratively refine scene layout estimates via a low-friction "one-click fix'' workflow. Our system enables the final refined layout to diverge from the training distribution, allowing for more accurate modelling of complex layouts.