PerceptTwin: Semantic Scene Reconstruction for Iterative LLM Planning and Verification
PerceptTwin reconstructs semantic scenes from perception data, enabling interactive simulation for robot plan verification, improving success rates by 39%.
Key Findings
Methodology
PerceptTwin integrates open-vocabulary object maps with 3D asset generation, affordance prediction, and commonsense checks. It extracts semantic scene graphs from robot perception, employs CLIP for object matching, SAM and TRELLIS for high-quality 3D asset creation, and localizes objects via point clouds. The system predicts applicable skills using LLMs, constructs interactive simulations, and utilizes an LLM judge for plan validation. Experiments show an average 39% increase in plan success and 18% in human verification accuracy, demonstrating robust scene understanding and plan refinement capabilities.
Key Results
- In tests with GPT5, GPT5Mini, and GPT5Nano planners, PerceptTwin improved plan success rates by approximately 39%, significantly outperforming baseline methods. The simulation feedback allowed planners to optimize paths, enhance safety, and resist black-box prompt attacks. Human plan verification accuracy for failed plans increased up to 18%, indicating improved interpretability and reliability. These results highlight the effectiveness of automatic scene reconstruction in complex, open-world scenarios.
- The LLM judge effectively identified unsafe or infeasible plans, reducing execution risks. Automated scene generation lowered customization costs and increased scene diversity. Cross-scenario robustness was validated in indoor and outdoor environments, confirming the method’s adaptability.
Significance
This work addresses the longstanding challenge of static scene representations in robotics, offering a dynamic, open-vocabulary simulation framework grounded in perception data. By combining large pre-trained models with automated scene reconstruction, it advances safe, reliable autonomous planning. The approach enhances interpretability, reduces trial-and-error, and paves the way for deploying robots in complex real-world environments, thus bridging the gap between perception and decision-making.
Technical Contribution
Key innovations include: 1) an end-to-end real2sim pipeline from perception to interactive simulation; 2) multi-modal asset generation using CLIP, SAM, TRELLIS; 3) an LLM-based plan feedback and validation system inspired by AI alignment research. These components enable multi-round plan refinement, improve safety, and provide a scalable framework for open-vocabulary scene understanding, surpassing prior static or semi-automated methods.
Novelty
This is the first system to fully automate the conversion of perception-derived semantic scene graphs into interactive, verifiable simulations. It uniquely combines multi-modal models for asset creation and localization, and introduces an LLM judge for plan correctness and alignment. Unlike prior works like ProcThor or Holodeck, PerceptTwin emphasizes real-world scene reconstruction and iterative plan validation, representing a significant leap forward in autonomous scene understanding.
Limitations
- Dependence on high-quality perception data; noisy or incomplete inputs can reduce scene fidelity and asset accuracy.
- Processing time (~1 hour) and hardware demands (NVIDIA GPU) limit scalability for real-time applications.
- LLM judge’s validation relies on predefined rules; in unseen or adversarial scenarios, misjudgments may occur, requiring further robustness improvements.
Future Work
Future efforts will focus on optimizing asset generation speed, integrating reinforcement learning for autonomous interaction, and expanding multi-robot scenario support. Enhancing the generalization of the LLM judge to handle novel environments and reducing computational costs will be prioritized, aiming for real-time deployment in complex, dynamic settings.
AI Executive Summary
Robotics research has long grappled with the challenge of verifying plans within complex, real-world environments. Traditional simulation tools often require manual scene design, limiting flexibility and scalability. This gap hampers autonomous robots' ability to safely and reliably operate in diverse settings. To address this, the authors introduce PerceptTwin, a novel framework that automatically reconstructs semantic scenes from robot perception data into interactive, verifiable simulations.
PerceptTwin leverages cutting-edge models such as CLIP for object matching, SAM for segmentation, and TRELLIS for high-fidelity 3D asset generation. The system begins by extracting a semantic scene graph from perception inputs, then automatically generates and localizes 3D assets aligned with real objects. These assets are assembled into a virtual environment where robot plans can be tested and refined. Crucially, an LLM-based judge evaluates the logical correctness and safety of plans, inspired by AI alignment principles.
Experimental results demonstrate that this approach significantly enhances planning success rates—by approximately 39% on average—and improves human validation accuracy up to 18%. The interactive simulation environment not only reduces the risk of unsafe actions but also provides interpretable feedback, making robot plans more transparent and trustworthy. This work marks a substantial step toward safer, more reliable autonomous systems capable of operating in unstructured, open-vocabulary environments.
The broader impact of PerceptTwin lies in its potential to transform robot deployment across industries. By automating scene understanding and plan verification, it reduces development costs and accelerates deployment cycles. Future directions include optimizing computational efficiency, expanding multi-robot capabilities, and integrating reinforcement learning to enable autonomous adaptation. Despite current limitations, such as processing time and reliance on perception quality, this framework paves the way for next-generation autonomous robots that can reason, verify, and operate safely in complex real-world settings.
Deep Analysis
Background
Robotics scene understanding has evolved from simple geometric mapping (SLAM) to semantic mapping with models like ConceptGraph, CLIP-enhanced SLAM, and large language models. These advances have improved scene comprehension but remain largely passive and static, limiting their use in dynamic plan validation. Recent efforts like ProcThor and Holodeck introduced scene generation from prompts, yet they rely on manual asset curation or limited asset pools. The gap persists in automating perception-driven scene reconstruction into interactive, verifiable environments, crucial for safe autonomous operation in unstructured environments.
Core Problem
The main challenge is transforming perception data into dynamic, interactive simulations that accurately reflect real-world scenes for plan testing. Existing static maps lack the fidelity and interactivity needed for reliable validation. Manual scene creation is labor-intensive, costly, and not scalable. Moreover, current plan verification methods lack open-vocabulary reasoning and safety checks, making it difficult to ensure plans are both feasible and aligned with human preferences. Addressing these issues requires an automated, scalable pipeline that integrates perception, scene reconstruction, and plan validation.
Innovation
The core innovations include: 1) a fully automated real2sim pipeline converting perception data into interactive scenes; 2) multi-modal asset generation combining CLIP, SAM, and TRELLIS for accurate, diverse 3D models; 3) an iterative LLM-based plan feedback mechanism that refines and verifies plans, inspired by AI alignment research; 4) a scene validation system that detects unsafe or infeasible plans before execution. These innovations enable scalable, open-vocabulary scene understanding and robust plan validation, surpassing prior static or semi-automated approaches.
Methodology
- �� Extract semantic scene graph from perception inputs, including object descriptions, point clouds, and images.
- �� Use CLIP for object matching, selecting or generating 3D assets via mesh association or TRELLIS-based mesh generation.
- �� Localize and orient assets in the scene using a constrained ICP algorithm, ensuring spatial accuracy.
- �� Predict object affordances with LLMs, considering robot capabilities and scene context.
- �� Assemble assets into an interactive simulation environment, enabling plan testing.
- �� Implement an LLM judge that compares scene states before and after plan execution, detecting logical inconsistencies or safety issues, and providing feedback for iterative plan refinement.
Experiments
The evaluation involved diverse indoor and outdoor scenes, comparing static SLAM maps with PerceptTwin's reconstructed environments. Metrics included plan success rate, human verification accuracy, and safety validation. Experiments tested multiple LLM planners (GPT5, Mini, Nano) with and without PerceptTwin feedback. Results showed a 39% average increase in success rate, with significant improvements in plan safety and interpretability. Human studies confirmed enhanced understanding and detection of plan flaws, validating the system’s practical utility across scenarios.
Results
PerceptTwin achieved an average 39% boost in plan success across tested scenarios, with a notable 18% increase in human verification accuracy for plans that previously failed due to unfulfilled preconditions. The system effectively identified unsafe plans, reducing execution risks. Asset generation quality was validated through visual fidelity and semantic correctness, with TRELLIS and SAM producing more accurate models than previous methods. The iterative feedback loop enabled continuous plan refinement, demonstrating robustness in complex environments.
Applications
该技术适用于工业自动化、服务机器人、救援行动等场景，提供安全验证和计划优化工具。利用机器人感知数据自动重建场景，降低场景定制成本，提升系统适应性。未来可扩展多机器人协作和复杂环境适应能力，推动自主系统在实际应用中的广泛部署。
Limitations & Outlook
目前方法对感知质量敏感，复杂环境中资产匹配和定位存在误差，影响仿真真实性。构建过程耗时较长（约1小时），对硬件资源要求较高，限制了实时应用。LLM judge在极端或未见场景下可能误判，需增强泛化能力。模型依赖大量预训练数据，存在偏差风险，未来需优化算法和数据集以提升鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房准备一顿大餐。你用相机和传感器观察厨房里的所有东西，比如锅、刀、食材。这些信息就像一份详细的清单。然后，你用特殊的工具（比如3D模型和智能软件）把这些东西变成虚拟的厨房场景，可以在电脑上操作。你可以试着“炒菜”或“切菜”，看看计划是否合理，是否会碰到问题。这样，机器人也可以用类似的方法，提前模拟环境，验证它的行动计划，确保安全和效率。就像在虚拟厨房里练习，避免真正做饭时出错。
Abstract
Simulation environments are useful for both robot policy learning and planning verification and validation. Traditionally, the process of creating a simulation was onerous. Creating a bespoke simulation environment for each individual environment that a robot would operate in was simply infeasible. In this work, we introduce PerceptTwin, a fully automatic pipeline that constructs interactive simulations directly from semantic scene representations produced by a robot's perception stack. PerceptTwin combines open-vocabulary object maps with 3D asset generation, affordance prediction, and commonsense condition checking. These interactive simulations can be used to validate and refine plans before they are executed on the robot hardware. Borrowing from the AI alignment literature, we also introduce an LLM judge that verifies plan correctness and alignment with human preferences. Experiments show that PerceptTwin feedback allows LLM planners to refine plans, enhance safety, and resist harmful black-box prompting attacks. In our suite of tasks, PerceptTwin improves plan success by an average of approximately 39% for GPT5, GPT5Mini, and GPT5Nano planners. Additionally, PerceptTwin also improves human plan verification by up to 18% on average for plans that fail due to unfilled skill preconditions. Our results demonstrate the potential of open-vocabulary scene simulation from robot perception as a foundation for safer, more reliable robot planning.