GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models
GaussianWAM uses 3D Gaussian fields to enhance WAM's geometric and semantic representations, boosting success rates from 52.05% to 71.29%.
Key Findings
Methodology
GaussianWAM constructs a 3D Gaussian teacher from synchronized multi-view observations, integrating depth, camera parameters, and dense semantic features from pretrained models VGGT and CLIP. These signals are bound to shared Gaussian primitives, which are rendered into dense supervision signals via differentiable Gaussian rendering. During training, these signals guide the model to learn geometry- and semantics-aware latent representations. After training, all teacher modules and Gaussian components are removed, leaving the original WAM inference pipeline unchanged. Experiments on datasets like LIBERO-Plus and RoboTwin demonstrate significant improvements in task success rates, especially under distribution shifts.
Key Results
- On LIBERO-Plus, GaussianWAM boosts FastWAM success rate from 52.05% to 71.29%, and Cosmos Policy from 71.52% to 77.30%. The improvements are most prominent in camera, lighting, noise, and robot variation scenarios, indicating enhanced robustness.
- Combining CLIP and VGGT distillation established a strong baseline, which was further improved by Gaussian field unification, demonstrating the effectiveness of spatially organizing heterogeneous signals.
- In real-world robotic tasks, GaussianWAM shows positive transfer and robustness, validating training-time Gaussian distillation as a practical approach for enhancing spatial understanding without inference overhead.
Significance
This work addresses the long-standing challenge of integrating explicit 3D geometry and object semantics into WAMs. By leveraging a training-time Gaussian field, it significantly improves the spatial coherence and semantic relevance of learned representations. This advancement enhances robotic manipulation precision, robustness under environmental variations, and paves the way for more reliable autonomous systems. The approach offers a scalable, inference-efficient solution that bridges the gap between 2D visual prediction and 3D spatial understanding, with broad implications for robotics, AR/VR, and autonomous driving.
Technical Contribution
The core technical innovation is the use of a 3D Gaussian field as a unified spatial carrier that binds heterogeneous geometric and semantic signals. This enables joint optimization of geometry and semantics during training, with signals rendered densely onto the WAM latent space. The method employs differentiable Gaussian rendering and loss functions for depth, semantic cosine similarity, and coverage, facilitating end-to-end training. Importantly, the Gaussian modules are only used during training, ensuring inference efficiency remains unchanged. This approach offers a novel, scalable way to incorporate explicit 3D structure and rich semantics into WAMs.
Novelty
This is the first work to unify multi-view geometric cues and dense semantic features within a shared 3D Gaussian field for training-time supervision of WAMs. Unlike prior methods that treat geometry and semantics separately or require complex 3D reconstruction modules, GaussianWAM employs a simple yet effective Gaussian rendering-based supervision, maintaining inference efficiency. This innovation bridges the gap between 2D visual prediction and explicit 3D understanding, setting a new direction for spatially grounded robotic representations.
Limitations
- The method relies heavily on the accuracy of pretrained models VGGT and CLIP; errors in depth or semantic predictions can affect Gaussian supervision quality.
- Optimization of Gaussian parameters in highly complex or dynamic scenes may face convergence issues, requiring further algorithmic refinement.
- In scenarios with severe occlusion or rapid scene changes, the Gaussian field may struggle to accurately capture scene geometry and semantics.
Future Work
Future research will explore integrating self-supervised signals to reduce dependence on pretrained models, extending the approach to dynamic scenes with moving objects, and scaling to larger, more diverse environments. Additionally, improving the robustness of Gaussian fitting in challenging scenarios and reducing training time are key directions. Combining this with multi-modal data and reinforcement learning could further enhance autonomous manipulation capabilities.
AI Executive Summary
Robotic manipulation and autonomous control increasingly demand precise understanding of scene geometry and object semantics. Traditional world-action models (WAMs) excel at visual prediction but lack explicit spatial structure, limiting their effectiveness in tasks requiring accurate spatial reasoning. To address this, GaussianWAM introduces a novel training-time enhancement using a 3D Gaussian field that unifies geometric and semantic supervision.
This approach leverages multi-view synchronized observations, employing pretrained models VGGT for depth and camera parameters, and CLIP for dense semantic features. These signals are bound to shared Gaussian primitives, which are rendered into dense supervision signals via differentiable Gaussian rendering. During training, these signals guide the model to learn geometry- and semantics-aware latent representations. Crucially, all Gaussian modules are removed after training, leaving the inference pipeline unchanged, ensuring deployment efficiency.
Extensive experiments on datasets like LIBERO-Plus and RoboTwin demonstrate that GaussianWAM significantly improves task success rates, especially under challenging distribution shifts. For example, success rates for FastWAM increase from 52.05% to 71.29%, and Cosmos Policy from 71.52% to 77.30%. These results highlight the method's robustness and practical value in real-world robotic manipulation.
This work advances the field by providing a scalable, inference-efficient way to embed explicit 3D geometry and rich semantics into WAM representations. It opens new avenues for more reliable, spatially grounded autonomous systems, with broad implications for robotics, AR/VR, and beyond. Future directions include extending to dynamic scenes, reducing training costs, and integrating multi-modal data for even richer scene understanding.
Deep Analysis
Background
机器人自主控制的发展历程中，空间理解和语义感知一直是核心难题。早期方法主要依赖二维图像和潜在空间，难以保证空间结构的准确性。近年来，深度学习引入点云、体素等3D表示技术，提升了场景重建能力，但在机器人任务中仍存在空间一致性不足的问题。代表性工作如GQN、NeRF等，改善了场景的连续性和细节，但在复杂环境中的空间理解仍有限。随着基础模型如CLIP、VGGT的出现，结合多模态信息成为趋势，但如何高效融合异构信号仍是难点。
Core Problem
现有WAM在视觉预测方面表现优异，但在空间几何和对象语义的保持上存在明显不足。多视角几何信息未被充分整合，导致模型在空间交互和精确操控中表现受限。异构信号缺乏统一的空间载体，难以实现空间结构和语义信息的有效结合。此外，传统方法引入复杂的3D重建或多模态融合模块，增加了训练和推理的复杂度，限制了实际应用的推广。
Innovation
GaussianWAM提出创新性解决方案，利用3D高斯场作为统一的空间载体，将多视角几何和语义信息绑定到共享的高斯原语上，实现异构信号的空间对齐。通过可微渲染生成密集监督信号，训练模型学习到空间结构和语义的联合表示。该方法在训练阶段引入高斯场，推理时完全移除，保持了原有推理路径的高效性。相较于传统多模态融合和3D重建技术，该方案简化了模型结构，显著提升了空间理解能力。
Methodology
- �� 利用预训练VGGT模型提取深度和相机参数，构建几何有效掩码。• 使用CLIP提取密集语义特征，将其与几何信息绑定到同一空间网格。• 初始化高斯场，将几何和语义信息绑定到每个有效像素对应的高斯原语。• 通过可微渲染将高斯场投影到多视角图像空间，生成深度、语义和覆盖信号。• 设计多重损失函数（深度、语义相似度、覆盖）优化高斯参数。• 将渲染信号缓存，作为训练密集监督，指导WAM学习空间结构和语义信息。
Experiments
在LIBERO、LIBERO-Plus、RoboTwin等数据集上，采用FastWAM和Cosmos Policy作为基线模型。训练分别用70k和5k步，设置多GPU加速，调节超参数。对比无蒸馏、CLIP VGGT蒸馏和高斯场蒸馏效果，评估成功率、鲁棒性和迁移能力。通过消融实验验证高斯场的贡献，分析不同场景下的性能变化。
Results
GaussianWAM在LIBERO-Plus上将FastWAM成功率从52.05%提升至71.29%，在摄像头、光照、噪声和机器人变化场景中表现出显著优势。结合CLIP和VGGT的蒸馏，建立了强大基线，性能持续提升。真实机器人任务中表现出良好的迁移和鲁棒性，验证了训练时高斯蒸馏的实用性。整体上，该方法显著改善了WAM的空间理解和操控能力，展现出广阔的应用前景。
Applications
该技术适用于自主机器人、工业自动化及虚拟现实等场景，能显著提升空间交互的精度和鲁棒性。无需在推理时引入额外模块，便于实际部署。未来可结合多模态信息和自监督学习，推动自主系统在复杂环境中的应用，增强空间理解和操作能力。
Limitations & Outlook
目前依赖预训练模型的准确性，若模型在特定场景表现不佳，可能影响蒸馏效果。高斯参数优化在复杂或动态场景中可能面临收敛困难。极端遮挡或快速变化场景下，空间几何和语义的联合建模仍需改进。未来需提升算法稳定性和训练效率，增强模型在极端环境中的表现。
Plain Language Accessible to non-experts
想象你在厨房准备一顿饭。每次你会观察每样食材的位置（空间信息），记住它们的味道和用途（语义信息），还会用手触摸确认它们的状态。如果你能把所有这些信息都放在一个大篮子里，随时知道每样食材在哪里、它们的味道和用法，那么做饭就会变得更快更顺利。GaussianWAM就像这样，把场景中的空间位置和语义信息都装到一个“高斯篮子”里，训练时让模型学会用这个篮子记住场景的空间和语义，推理时不用这个篮子了，但模型已经学会了如何快速理解场景中的物体。这让机器人在复杂环境中变得更聪明、更可靠，能更好地完成任务。
ELI14 Explained like you're 14
想象你在学校操场玩捉迷藏。你知道每个朋友藏在哪里（空间位置），他们穿的衣服代表谁（语义信息），你用这些线索很快就能找到他们。现在，假如你有一张神奇的地图，把朋友们藏的地方和他们的衣服信息都画在一起（用高斯场），你就能更快找到他们。GaussianWAM就像这个神奇的地图，它在训练时用很多照片和信息，把场景的空间和语义都画在一起，学会了怎么用这些信息理解场景。等到真正玩的时候，它就不用地图了，但已经学会了怎么用这些线索找到目标。这让机器人在复杂环境中变得更聪明、更快，能更好地完成任务。
Abstract
World-Action Models (WAMs) jointly learn future visual prediction and action generation, using video dynamics as a representation-learning signal for robotic manipulation. However, their video latents are primarily optimized for visual prediction and are not explicitly encouraged to preserve cross-view geometric structure or spatially localized, object-relevant semantics. We propose \textbf{GaussianWAM}, a training-time representation-enhancement framework that organizes geometric and semantic supervision through a 3D Gaussian field. Given synchronized multi-view observations, frozen geometry and vision foundation models provide depth, camera parameters, and dense semantic features. GaussianWAM binds these heterogeneous signals to shared Gaussian primitives and renders spatially aligned semantic, depth, and coverage targets, which are distilled into the current-observation representations of the WAM. All teacher models, Gaussian components, and auxiliary prediction heads are removed after training, leaving the original WAM inference path without additional modules or forward computation. On LIBERO-Plus, GaussianWAM improves FastWAM from 52.05\% to 71.29\% and Cosmos Policy from 71.52\% to 77.30\%. Direct CLIP and VGGT distillation already establishes a strong FastWAM baseline of 69.37\%, while Gaussian-field unification further improves it to 71.29\%, supporting the benefit of spatially organizing heterogeneous teacher signals. GaussianWAM also improves performance on standard LIBERO and shows positive transfer trends on RoboTwin and real-world manipulation. These results suggest that training-time Gaussian distillation provides a practical way to inject geometry- and semantics-related supervision into WAM representations without changing their deployment architecture.