DrivingScene: A Multi-Task Online Feed-Forward 3D Gaussian Splatting Method for Dynamic Driving Scenes
DrivingScene employs a static-to-dynamic two-stage training with residual scene flow, achieving real-time high-fidelity 4D scene reconstruction from two images.
Key Findings
Methodology
DrivingScene utilizes a two-stage training process: first, a static scene prior is learned via depth and Gaussian parameter networks from two frames; second, the static model is frozen, and a residual flow network with a pyramidal architecture predicts non-rigid motion residuals. The residual flow combines a shared backbone with per-camera decoders, enabling stable, detailed dynamic modeling. The entire framework is trained with self-supervised losses including geometric consistency and rendering quality, allowing end-to-end optimization. This approach efficiently reconstructs 4D scenes in real-time, generating high-quality depth, scene flow, and Gaussian point clouds from minimal input, surpassing prior static or offline methods.
Key Results
- On nuScenes, DrivingScene achieves a PSNR of 28.76 for novel view synthesis, outperforming DrivingForward (26.06) and Driv3R (26.10). Depth prediction errors are lower with an Abs Rel of 0.227, confirming better geometric understanding.
- Inference time per frame is 0.21 seconds with only 0.117GB parameters, demonstrating high efficiency. The model also excels in scene flow accuracy, with clear separation of rigid and residual flows visualized.
- Ablation studies show removing residual flow or flow warping loss degrades performance, validating their importance. Overall, the method significantly advances dynamic scene reconstruction quality and speed.
Significance
This work addresses the critical challenge of real-time, high-fidelity dynamic scene reconstruction using only monocular images, a key requirement for autonomous driving. By decoupling static and dynamic components, it overcomes the limitations of static assumptions and offline optimization, enabling scalable, efficient perception. The framework’s ability to generate detailed depth, scene flow, and 3D point clouds on-the-fly paves the way for more robust environment understanding and safer autonomous systems, with broad implications for robotics, AR/VR, and intelligent surveillance.
Technical Contribution
The paper introduces a novel static-to-dynamic training paradigm, leveraging a pre-trained static scene prior and a lightweight residual flow network with a pyramidal architecture. This design ensures stable, detailed dynamic modeling while maintaining real-time inference. The multi-task self-supervised learning framework integrates geometry, rendering, and flow consistency losses, enabling end-to-end training without manual annotations. The hybrid-shared residual flow architecture efficiently captures large displacements and complex non-rigid motions, setting a new standard for online 4D scene reconstruction.
Novelty
This is the first work to explicitly decouple static and dynamic scene modeling via a residual scene flow network trained in a two-stage manner. Unlike prior static-only or end-to-end dynamic methods, it combines a static scene prior with a lightweight residual flow, enabling accurate, real-time reconstruction of complex, non-rigid motions from minimal input. Its multi-task self-supervised approach and hybrid architecture represent a significant innovation in neural scene reconstruction.
Limitations
- The model relies on a pre-trained static prior, which may limit performance in highly dynamic or occluded scenarios. Extreme motions or rapid scene changes can still challenge the residual flow prediction.
- Long-term temporal consistency is not explicitly modeled, potentially affecting performance over extended sequences.
- Computational costs, while low, may still be prohibitive for extremely large scenes or high-resolution inputs, requiring further optimization.
Future Work
Future research could incorporate multi-frame sequences to improve temporal coherence and robustness. Developing more expressive deformation models for non-rigid objects and integrating multi-modal data like LiDAR could further enhance scene understanding. Extending the framework to handle longer sequences and more complex dynamic phenomena remains an important direction.
AI Executive Summary
DrivingScene introduces a cutting-edge, real-time framework for reconstructing dynamic driving scenes from just two consecutive surround-view images. Its core innovation lies in a two-stage training process: first, learning a robust static scene prior through depth and Gaussian parameter networks, establishing a geometrically consistent foundation; second, freezing this static model and training a residual scene flow network with a pyramidal architecture to predict non-rigid motions. This static-to-dynamic decoupling effectively addresses the instability and complexity challenges faced by previous end-to-end methods.
The residual flow network, with its hybrid-shared architecture, captures large displacements and complex non-rigid motions efficiently, enabling high-fidelity dynamic scene modeling. The entire system is trained with self-supervised losses, including geometric consistency, flow warping, and rendering quality, ensuring stable, accurate, and coherent reconstructions. Experiments on nuScenes demonstrate that DrivingScene outperforms state-of-the-art methods in novel view synthesis, depth accuracy, and scene flow estimation, while maintaining a lightweight model size and fast inference speed.
This approach signifies a major step forward in autonomous driving perception, providing a scalable, efficient, and detailed understanding of complex environments. Its ability to generate high-quality depth, scene flow, and 3D point clouds online from minimal input opens new avenues for safer, more reliable autonomous systems. Future extensions could involve multi-frame integration, more expressive deformation models, and multi-modal sensor fusion, further pushing the boundaries of real-time scene understanding.
Deep Dive
Abstract
Real-time, high-fidelity reconstruction of dynamic driving scenes is challenged by complex dynamics and sparse views, with prior methods struggling to balance quality and efficiency. We propose DrivingScene, an online, feed-forward framework that reconstructs 4D dynamic scenes from only two consecutive surround-view images. Our key innovation is a lightweight residual flow network that predicts the non-rigid motion of dynamic objects per camera on top of a learned static scene prior, explicitly modeling dynamics via scene flow. We also introduce a coarse-to-fine training paradigm that circumvents the instabilities common to end-to-end approaches. Experiments on nuScenes dataset show our image-only method simultaneously generates high-quality depth, scene flow, and 3D Gaussian point clouds online, significantly outperforming state-of-the-art methods in both dynamic reconstruction and novel view synthesis.