GeoWAM: Visual Geometry World Action Models for Autonomous Driving
GeoWAM predicts future scene geometry with point clouds, significantly improving autonomous driving trajectory planning.
Key Findings
Methodology
GeoWAM employs a point-cloud-based geometric representation, pretraining a transformer-based model to forecast dense future scene geometry. The model consists of a multiview geometry encoder (based on DVGT-2) extracting multi-scale geometric and ego information, and a future geometry decoder that uses causal self-attention to predict future point clouds. Supervised by dense point map and confidence losses, the model learns to predict scene evolution without RGB reconstruction. During fine-tuning, an inverse-dynamics-inspired trajectory head maps geometric dynamics to ego paths, enabling end-to-end planning.
Key Results
- On nuScenes, GeoWAM achieves an average absolute relative error (Abs Rel) of 0.257 over 4 seconds, outperforming Epona+DVGT’s 0.274, and maintains a δ<1.25 accuracy of 0.754, demonstrating superior long-term geometric forecasting.
- In NAVSIM closed-loop planning, GeoWAM scores 90.2 on the Extended Predictive Driver Model Score (EPDMS), surpassing multiple baselines, especially in complex scenarios, indicating robust planning performance.
- The model generalizes well across multi-view, multi-time-step data, validating the effectiveness of geometric pretraining for dynamic scene understanding in autonomous driving.
Significance
This work shifts scene modeling from pixel-based image prediction to explicit geometric forecasting, addressing the limitations of indirect scene representation. By capturing spatial structure and scene dynamics directly via point clouds, GeoWAM enhances path planning accuracy, robustness, and safety. It bridges the gap between scene understanding and motion control, providing a more natural and effective state space aligned with the physical environment. The approach offers promising avenues for safer, more reliable autonomous vehicles and advances the theoretical foundation of scene dynamics modeling.
Technical Contribution
The paper introduces a novel geometric world action model integrating dense point cloud forecasting with transformer-based sequence modeling. It leverages a pretraining strategy that avoids depth annotation reliance, using dense point map supervision instead. The inverse-dynamics-inspired trajectory head directly maps predicted geometric evolution to ego paths, enabling end-to-end planning. The architecture combines multi-scale geometric encoding, causal self-attention, and confidence supervision, establishing a unified framework for scene understanding and trajectory prediction in 3D space.
Novelty
This is the first work to treat scene geometry as the primary prediction space for autonomous driving, moving away from pixel-based scene generation. It introduces a dense point cloud forecasting paradigm combined with a geometric inverse-dynamics approach, enabling explicit modeling of scene evolution and direct trajectory inference. This approach outperforms existing image-based and voxel occupancy models, setting a new standard for scene dynamic modeling.
Limitations
- The model's robustness in highly non-rigid, occluded, or extreme lighting conditions remains limited; integrating multi-modal sensors like LiDAR could help.
- Computational cost is high, especially for dense point cloud prediction and transformer inference, challenging real-time deployment.
- Handling complex non-rigid scene deformations and dynamic occlusions requires further research to improve generalization.
Future Work
Future directions include integrating LiDAR and semantic segmentation for richer scene understanding, optimizing model architectures for real-time inference, and exploring reinforcement learning for adaptive decision-making. Extending the framework to multi-agent scenarios and urban environments will further enhance its practical deployment potential.
AI Executive Summary
Autonomous driving demands precise scene understanding and reliable path planning in complex environments. Traditional pixel-based models, while visually rich, struggle to explicitly capture the three-dimensional structure and dynamic transformations of the scene, limiting their effectiveness in long-term planning and robustness. Addressing this challenge, the present work introduces GeoWAM, a geometric world action model that predicts future scene geometry directly in the point cloud domain.
GeoWAM’s core innovation lies in replacing pixel prediction with dense point cloud forecasting, leveraging transformer architectures to model scene evolution across multiple scales and time steps. The model is pretrained on large-scale driving datasets, learning to predict dense future point maps without requiring depth annotations, thus enhancing generalization. During fine-tuning, an inverse-dynamics-inspired trajectory head maps these geometric predictions into ego vehicle paths, enabling end-to-end planning.
Experimental results demonstrate that GeoWAM outperforms existing methods in both future geometry prediction and closed-loop path planning. On nuScenes, it achieves an average Abs Rel of 0.257 over 4 seconds, surpassing pixel-based baselines. In NAVSIM, it attains a top EPDMS score of 90.2, indicating superior safety and efficiency in complex scenarios. These findings confirm that explicit geometric modeling significantly advances autonomous driving capabilities.
Despite its strengths, the approach faces challenges in handling highly non-rigid scenes and real-time deployment. Future work will focus on integrating multi-modal sensors, optimizing computational efficiency, and extending the framework to urban multi-agent environments. Overall, GeoWAM paves the way for safer, more robust autonomous vehicles by fundamentally rethinking scene representation and action prediction in three-dimensional space.
Deep Analysis
Background
随着深度学习在自主驾驶中的广泛应用，场景理解逐渐成为核心问题。早期方法依赖激光雷达和高清地图实现环境感知，随后出现基于深度神经网络的像素级未来帧预测模型，如DriveDreamer、GEM等，能够生成逼真的未来场景视频，但难以直接反映空间结构。近年来，点云和体素等几何表示逐渐兴起，为场景的三维理解提供了更自然的表达方式。代表性工作如DVGT、VGGT等，利用多视角几何编码实现稠密场景重建，为路径规划提供几何基础。然而，这些方法多依赖深度标注或稠密点云，数据采集成本高，泛化能力有限。
Core Problem
现有模型多集中于像素空间的未来帧生成，难以直接捕获场景的空间结构和动态变换，导致路径规划在复杂环境中表现不佳。像素表示的局限在于其对几何信息的间接编码，难以实现高精度的空间理解。长时序预测误差累积严重，遮挡和非刚性变形场景下表现不足。如何利用点云几何信息，建立更自然、更高效的场景表示，成为亟待解决的问题。
Innovation
本研究提出以点云几何为核心场景表示，利用预训练模型预测未来场景几何，避免深度标注依赖。引入Transformer架构实现多尺度空间-时间建模，提升几何预测精度。采用逆动力学思想，将场景几何演变映射到未来ego路径，增强路径规划的合理性。这一方法实现了几何信息与动作决策的端到端融合，为自主驾驶提供了全新视角。
Methodology
- �� 使用DVGT-2编码器对多视角图像序列进行多尺度几何编码，提取空间结构和ego信息，形成历史几何记忆。• 设计未来几何解码器，利用Transformer的因果自注意力机制，预测未来点云，捕获场景演变。• 训练目标包括未来点云的欧几里得回归和置信度监督，确保几何预测的准确性。• 在微调阶段，利用逆动力学思想，将预测的几何动态映射到未来ego路径，提升路径规划效果。• 采用密集点云目标监督，避免深度标注依赖，增强模型泛化。• 最终实现几何预测与路径规划的端到端融合。
Experiments
在nuScenes数据集上，评估未来几何预测的误差指标（Abs Rel）和阈值指标（δ<1.25），验证模型在不同预测时长（1-4秒）内的性能。与Epona+DVGT和VGGT-World等基线模型比较，GeoWAM在所有时长段表现优异，尤其在4秒预测中保持误差最低（0.257）和阈值准确率最高（0.754）。在NAVSIM路径规划任务中，采用闭环和两阶段评估，GeoWAM获得最高的EPDMS（90.2），显示其在复杂交通场景中的鲁棒性。模型训练包括Transformer层数、隐藏维度、学习率调度等参数，确保模型性能。
Results
GeoWAM在未来几何预测中显著优于像素生成模型，尤其在长时预测中表现出更低的误差和更高的准确率。在路径规划中，模型在复杂环境中保持高安全性和效率，EPDMS得分达90.2，优于多种先进模型。模型对遮挡和非刚性变形的适应性较强，验证了几何表示的优势。实验还揭示了预训练几何预测对提升路径鲁棒性的关键作用，为自主驾驶系统提供了坚实基础。
Applications
该模型可应用于自动驾驶车辆的路径规划与决策系统，依赖多视角摄像头和点云数据，提升环境理解的空间精度。未来结合激光雷达和语义分割信息，增强在复杂交通环境中的表现。长远来看，GeoWAM有望推动智能交通系统的自动化和安全性，减少交通事故，提升交通效率，为智慧城市建设提供技术支撑。
Limitations & Outlook
模型在极端光照、遮挡或非刚性变形场景下表现仍有限，需引入多模态信息和更鲁棒的几何表达。训练和推理成本较高，实时性有待优化。未来应结合强化学习和模型压缩技术，提升系统的实用性和普适性。
Plain Language Accessible to non-experts
想象你在玩一个用点点和线条画的地图游戏，没有颜色也没有细节，但你可以通过观察这些点点的变化，知道前方的道路和障碍物会怎么变。这个模型就像用点云来画出场景的地图，然后预测未来点云会变成什么样，就像提前知道下一幕会发生什么。它不用看彩色图片，只用几何信息，就能更清楚地理解空间关系和运动变化，就像用地图和路线图代替照片一样。这样，汽车可以根据这些点点的变化，提前规划出安全的行驶路线，避免碰撞。
ELI14 Explained like you're 14
想象你在玩一个超级酷的迷宫游戏，你看不到彩色的墙壁，只看到一些线条和点点，但你可以用这些点点画出迷宫的轮廓，然后猜测未来迷宫会变成什么样。这个模型就像用点点画出场景的地图，然后预测未来的变化。它不用彩色图片，只用几何信息，就像用地图和路线图帮你提前知道下一步怎么走。这样，汽车就能提前知道前方会发生什么，避开障碍物，安全到达目的地。它用点云做“地图”，让汽车更聪明、更安全！
Glossary
Point Cloud（点云）
由空间中大量点组成的三维数据，用于表示场景的几何形状。技术上，它由激光雷达或深度相机采集，反映环境的空间结构。
在论文中，点云作为场景几何的主要表示，用于预测未来场景的空间结构。
Transformer（变换器）
一种基于自注意力机制的深度学习架构，擅长建模序列中的长距离依赖关系。广泛应用于自然语言处理和视觉任务。
GeoWAM中的未来几何解码器采用Transformer实现多尺度空间-时间建模。
Inverse Dynamics（逆动力学）
通过观察运动结果反推作用力和运动状态的过程，常用于运动分析和控制。
模型中利用逆动力学思想，将场景几何演变映射到未来ego路径。
Dense Point Map（稠密点图）
场景中每个像素对应的三维点集合，表示场景的详细几何结构。
模型通过预测未来点云密集点图，实现场景几何的高精度预测。
Open Questions Unanswered questions from this research
- 1 在极端复杂场景（如非刚性变形、遮挡严重）中保持几何预测鲁棒性仍是挑战，未来需结合多模态信息提升模型适应性。
- 2 模型在实时性方面仍需优化，尤其在高密度点云和多视角融合情况下，需提升推理速度以满足工业应用。
Applications
Immediate Applications
路径规划增强
利用几何预测结果，提升自动驾驶车辆在复杂环境中的路径规划准确性和安全性，适用于高精度自动驾驶系统。
环境感知优化
结合多视角传感器，增强场景几何理解，为感知系统提供更稳固的空间结构信息，提升整体性能。
Long-term Vision
智慧交通系统
未来将几何场景理解融入城市交通管理，实现自动驾驶车辆的协同与优化，推动智慧交通和自动驾驶普及。
Abstract
World action models (WAMs) have recently gained increasing attention as a framework for jointly modeling scene evolution and ego actions in autonomous driving. Most existing WAMs learn scene dynamics in pixel space by combining a video-generation backbone for future-observation prediction with an action head for ego-trajectory prediction. Pixels, however, provide only an indirect representation of these dynamics: they entangle geometry and motion with appearance, texture, and illumination, forcing the model to infer three-dimensional transformations from two-dimensional observations. We argue that geometry, represented by point clouds, offers a more natural state space for driving because it explicitly captures spatial structure and the rigid and non-rigid transformations that govern scene evolution while directly aligning with the space in which driving actions are executed. Building on this insight, we introduce \textbf{GeoWAM}, a visual geometry world action model for autonomous driving. Rather than predicting future images, GeoWAM is pretrained to forecast future scene geometry, yielding representations that jointly encode spatial structure and temporal evolution. A geometry-conditioned action head then leverages these learned geometric dynamics to predict future ego trajectories. Extensive open-loop and closed-loop evaluations show that visual geometry world modeling yields substantially stronger driving policies than image-based alternatives, establishing future-geometry prediction as an effective pretraining objective for autonomous driving.