Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator
Image2Sim uses neural simulation to build scalable, high-fidelity interactive environments for embodied navigation training.
Key Findings
Methodology
Image2Sim employs a feed-forward feature Gaussian model to convert RGB-D observations into persistent 3D scene representations, avoiding costly optimization. A geometry-aware one-step pixel flow model synthesizes panoramic RGB-D images from sparse, noisy projections at 40 FPS. The system integrates collision-aware motion planning and automatic instruction generation, creating nearly 20,000 interactive scenes and over 10 million navigation samples. The approach enables models trained solely in neural environments to outperform traditional simulators and transfer zero-shot to real-world tasks.
Key Results
- On benchmarks like R2R and REVERIE, models trained in Image2Sim environments show over 10% improvement in SPL scores compared to baseline simulators. The rendering quality achieves PSNR of 23.88 and SSIM of 0.591 at 41 FPS, maintaining robustness under noisy conditions with PSNR of 17.43. The large-scale data enhances generalization and zero-shot transfer, demonstrating the system’s effectiveness.
- The neural simulator’s efficiency and fidelity surpass previous neural scene methods like DiT360 and SE3DS, especially under high noise. The system’s ability to generate diverse, realistic environments at scale addresses longstanding data bottlenecks in embodied navigation research.
Significance
This work bridges the gap between high-fidelity real-world data and scalable synthetic environments, providing a practical solution for large-scale embodied navigation training. It addresses core challenges of data scarcity, realism, and physical grounding, enabling more robust and generalizable autonomous agents. The approach opens new avenues for research in neural scene representation, simulation, and language-guided navigation, with broad implications for robotics, VR, and AI training pipelines.
Technical Contribution
The paper introduces a geometry-aware neural rendering pipeline combining a feed-forward Gaussian scene representation with a one-step pixel flow generator, enabling real-time high-quality panoramic RGB-D synthesis. It integrates collision-aware path planning and automatic instruction synthesis, forming an end-to-end automated data engine. This system significantly reduces computational costs compared to optimization-based methods, supports large-scale scene construction, and enhances zero-shot transfer capabilities, pushing the frontier of neural embodied simulation.
Novelty
This is the first work to directly convert large-scale image and video collections into high-fidelity, interactive 3D environments via a fully neural, real-time pipeline. The combination of a feed-forward Gaussian scene model with a geometry-aware pixel flow renderer, along with integrated motion and instruction generation, represents a novel paradigm that overcomes traditional scalability and fidelity tradeoffs, setting a new standard for neural simulation in embodied AI.
Limitations
- Despite high robustness, the system struggles with extreme occlusions and highly dynamic environments, where scene completion may be incomplete or inaccurate. Computational resource demands remain high, limiting deployment on edge devices. The scene construction relies on RGB-D data quality, which may vary across datasets, affecting generalization. Handling real-time dynamic changes remains an open challenge.
- The current approach is primarily tested on static indoor scenes; extending to outdoor, dynamic, or large-scale outdoor environments requires further development. Incorporating temporal consistency and dynamic scene understanding is a future direction.
Future Work
未来将结合多模态传感器数据，提升动态环境适应能力。探索模型压缩与高效推理策略，降低硬件依赖。计划引入强化学习优化路径与交互策略，增强自主导航的鲁棒性与泛化能力，推动神经模拟在实际机器人中的应用落地。
AI Executive Summary
Embodied navigation has long faced the challenge of balancing data fidelity and scalability. Traditional real-world scans offer high realism but are costly and limited in scope, while synthetic simulators scale easily but suffer from significant sim-to-real gaps. To address this, Zihan Wang and colleagues introduce Image2Sim, a neural simulation framework that constructs high-quality interactive environments directly from large-scale RGB-D images and videos.
The core innovation lies in decoupling 3D scene construction from observation synthesis. Using a feed-forward feature Gaussian model, the system efficiently lifts RGB-D observations into persistent 3D representations without optimization bottlenecks. For rendering, a geometry-aware one-step pixel flow model transforms sparse, noisy projections into panoramic RGB-D images at 40 FPS, maintaining high fidelity even under noisy conditions. This approach ensures real-time performance and robustness, crucial for training embodied agents.
Beyond rendering, Image2Sim integrates a collision-aware motion engine that generates physically valid trajectories, respecting obstacle constraints. It also employs a large vision-language model to automatically generate navigation instructions aligned with trajectories, enabling large-scale data synthesis—over 20,000 scenes and 10 million samples. The system's effectiveness is validated through extensive experiments, where models trained solely in these neural environments outperform those trained in traditional simulators, achieving superior zero-shot transfer to real-world tasks.
These advancements mark a significant step toward scalable, realistic neural simulation for embodied AI. By bridging the gap between real-world fidelity and synthetic scalability, Image2Sim opens new horizons for autonomous navigation, robotics, and virtual training, promising more adaptable and robust intelligent agents in diverse environments.
Deep Analysis
Background
Recent progress in embodied navigation has been driven by high-fidelity real-world scans and synthetic simulators. Datasets like Matterport3D, Gibson, and ScanNet provide realistic indoor environments, but their high acquisition cost limits scale. Synthetic environments such as Habitat and procedural worlds offer scalability but introduce a significant sim-to-real gap due to unrealistic assets and rendering statistics. Neural scene representations like NeRF and 3D Gaussian splatting have advanced visual fidelity but are computationally expensive and not suitable for large-scale data generation. Existing neural simulators often rely on optimization-based scene reconstruction, which hampers real-time scalability. This paper aims to leverage feed-forward neural models to construct scalable, high-fidelity environments suitable for embodied navigation training.
Core Problem
The core challenge is to develop a neural simulation system that combines the visual realism of real-world scans with the scalability of synthetic environments, while maintaining physical grounding and interaction fidelity. Existing methods either lack real-time performance or suffer from structural inaccuracies, limiting their utility for training robust navigation policies. Moreover, generating diverse, large-scale datasets with accurate annotations remains a bottleneck. Addressing these issues requires a system capable of rapid scene construction, high-quality rendering, and physically plausible motion simulation, all integrated into an automated pipeline.
Innovation
The paper introduces several key innovations: 1) a feed-forward Gaussian scene representation that encodes persistent 3D geometry without optimization, enabling real-time scene construction; 2) a geometry-aware one-step pixel flow renderer that synthesizes panoramic RGB-D images conditioned on scene geometry, handling noise robustly; 3) an integrated collision-aware motion engine that generates physically valid trajectories respecting obstacle constraints; 4) an automated vision-language instruction generation pipeline that produces diverse navigation data at scale. These innovations collectively enable scalable neural environment construction with high fidelity and physical realism, surpassing prior neural scene methods in efficiency and robustness.
Methodology
- �� Input：RGB-D视频和对应位姿信息。• 场景构建：利用双流编码器提取语义和几何特征，融合成密集高斯场，避免优化。• 投影：将高斯场投影到目标视角，生成全景RGB-D、深度和语义图。• 监督：通过重建损失和语义对齐，确保几何和语义一致。• 渲染：引入几何感知像素流模型，结合条件生成，补全稀疏投影。• 训练：采用连续时间MeanFlow损失和动量自蒸馏，提升稳定性。• 运动：基于高斯场进行碰撞检测和路径规划，生成符合物理的导航轨迹。• 指令：在神经模拟器中重放轨迹，自动生成自然语言指令，支持大规模数据合成。
Experiments
使用19,936个场景（真实与合成），涵盖Matterport3D、Structured3D等。模型在400K训练轮次中，分阶段优化几何初始化、像素流训练和细节增强。评估指标包括PSNR、SSIM、FPS，比较DiT360、AnySplat等。在导航任务中，模型在R2R、REVERIE等基准上超越传统模拟器，零样本迁移效果显著。渲染速度达41FPS，噪声环境下保持17.43 PSNR，验证了系统的鲁棒性和实用性。
Results
模型在R2R和REVERIE上的SPL提升超过10%，在复杂噪声环境中仍保持优异性能。渲染质量方面，PSNR达23.88，SSIM为0.591，FPS达41.3。大规模数据（超千万样本）显著改善导航模型的泛化能力，模型在不同模拟器和真实环境中表现出强迁移性。这些结果验证了神经模拟器在embodied navigation中的潜力，推动了从有限真实数据向大规模神经生成数据的转变。
Applications
该技术可广泛应用于自主机器人训练、虚拟现实、增强现实等领域。通过自动生成丰富的交互场景，降低数据采集成本，提升模型泛化能力。未来可结合动态环境和多模态传感器，推动机器人自主学习与适应复杂环境的能力，助力智能系统在实际场景中的部署。
Limitations & Outlook
目前系统在极端遮挡和复杂几何环境下仍存在生成不完整的问题，尤其在稀疏噪声环境中表现有限。模型对硬件资源依赖较大，训练成本较高，未来需优化模型压缩和推理效率。场景构建主要依赖RGB-D视频，可能受制于数据质量和多样性，动态环境适应性有待提升。
Plain Language Accessible to non-experts
想象你在一家工厂里工作，工厂里有很多不同的机器和材料。以前，要让机器人学会在工厂里找到东西、避开障碍，需要用很多真实的照片和复杂的模型，既费钱又慢。现在，科学家们发明了一种新方法，就像用一台特别聪明的相机，把工厂的照片变成虚拟的场景，这个虚拟场景既真实又可以快速生成。它用一种叫“神经模拟器”的技术，把工厂的图片变成一个可以自己动的虚拟环境，机器人可以在里面练习导航、避障，就像在玩一款超级逼真的游戏。这种方法不用每次都重新建模，只要给它一些图片，它就能快速生成很多不同的场景，让机器人学得更快、更好。未来，这个技术还能帮机器人在真实工厂中自主工作，节省很多时间和成本。
ELI14 Explained like you're 14
想象你在玩一个超级逼真的虚拟游戏，你的任务是带着角色在复杂的迷宫里找到出口。以前，要设计这个迷宫需要花很多时间画地图、建模型，还要确保每个房间都符合真实的比例和布局。现在，有了新技术，就像用一台神奇的相机，把你拍的迷宫照片变成一个可以自己走动的虚拟世界。这个系统用一种叫“神经模拟器”的聪明方法，把照片变成3D环境，机器人可以在里面练习导航、避障，就像在真实世界中一样。它还能自动生成不同的迷宫场景，让机器人学会应对各种复杂情况。这就像用一台超级快的相机，把照片变成虚拟的游戏世界，不仅省时间，还能让机器人变得更聪明、更强大。未来，这项技术可能让机器人在工厂、仓库甚至城市里自由行动，帮人们做很多事情。
Abstract
Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into nearly 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale.