DLGStream: Dynamic Language-embedded Guassian Splatting for Open-vocabulary Enabled Free-viewpoint Video Streaming
DLGStream introduces dual-opacity language Gaussian representation with deformation fields, achieving 43KB/frame and 60FPS in open-vocabulary free-viewpoint video streaming.
Key Findings
Methodology
DLGStream integrates a dual-opacity dynamic language Gaussian model with an interpolation-based deformation field. It leverages CLIP for extracting multimodal language features, encoding them as Gaussian attributes. The dual-opacity mechanism manages separate transparency parameters for color and language features, addressing performance degradation during joint optimization. The deformation field reduces temporal redundancy, enabling frame interpolation and FPS boosting from 20 to 60. The system streams Gaussian attributes and language features simultaneously, supporting real-time interaction, scene editing, and spatial intelligence. The core components include Gaussian point cloud reconstruction, multimodal feature encoding, dual-opacity management, and deformation-based temporal interpolation, all optimized for low-latency streaming.
Key Results
- On ShapeNet and ScanNet datasets, DLGStream achieved IoUs of 85.3% and 78.9%, respectively, outperforming existing methods. The average frame size was only 43KB, with FPS increased from 20 to 60 via deformation interpolation. Multilingual and open-vocabulary support enhanced scene interaction. Ablation studies confirmed the importance of dual-opacity and deformation modules. The system maintained high reconstruction quality and interaction robustness across complex scenes, demonstrating its practicality for real-time applications.
- Compared to baseline models, DLGStream improved open-vocabulary segmentation IoU by 12%, compressed data by 30%, and increased FPS significantly. The multimodal features enabled flexible scene descriptions, supporting diverse downstream tasks such as scene editing, spatial reasoning, and multi-user interaction. The results validate the effectiveness of the proposed mechanisms for scalable, high-fidelity, real-time free-viewpoint streaming.
- Ablation experiments indicated that removing dual-opacity or deformation components led to performance drops of over 10%, highlighting their critical roles. The system also showed resilience to varying scene complexity and vocabulary size, confirming its generalization ability. Overall, DLGStream offers a comprehensive solution for open-vocabulary, high-rate, low-storage free-viewpoint video streaming with rich semantic and spatial capabilities.
Significance
This work advances the state-of-the-art in real-time 3D scene reconstruction and streaming by addressing the limitations of existing Gaussian-based methods in open-vocabulary and high-FPS scenarios. Its ability to stream multimodal language features alongside Gaussian attributes enables richer scene understanding and interaction, essential for immersive VR/AR applications, digital twins, and remote collaboration. The low storage footprint combined with high FPS makes it suitable for bandwidth-constrained environments, broadening accessibility. The integration of deformation-based interpolation further enhances temporal consistency and user experience, paving the way for scalable, intelligent virtual environments. This research bridges the gap between high-fidelity 3D reconstruction and practical, interactive streaming, influencing future standards in digital content delivery.
Technical Contribution
DLGStream's key technical innovation lies in the dual-opacity mechanism, which separates color and language feature transparency, preventing joint optimization bottlenecks. The deformation field for temporal interpolation reduces redundancy, enabling high FPS with minimal storage. The framework combines multimodal feature extraction via CLIP, Gaussian point cloud modeling, and efficient stream synchronization. These innovations collectively enable open-vocabulary, high-rate, low-latency free-viewpoint streaming, surpassing prior methods limited by single-attribute Gaussian models or fixed transparency strategies. The approach also offers theoretical guarantees on temporal consistency and semantic expressiveness, opening new avenues for scalable 3D scene streaming.
Novelty
This is the first work to combine dual-opacity Gaussian representations with deformation-based temporal interpolation for open-vocabulary free-viewpoint video streaming. Unlike traditional single-attribute Gaussian models, this method manages separate transparency for color and language features, significantly improving joint optimization performance. The integration of CLIP multimodal features with a deformation field for frame interpolation is novel, enabling high FPS and semantic richness simultaneously. These innovations collectively push the boundary of real-time, scalable 3D scene streaming, especially in open-vocabulary contexts, marking a significant step forward in the field.
Limitations
- The current deformation interpolation may struggle with rapid scene changes or highly dynamic environments, leading to potential artifacts or reduced accuracy.
- Dependence on CLIP's language understanding introduces semantic biases and may affect scene consistency if descriptions are inaccurate or ambiguous.
- Computational costs, especially for large-scale scenes with dense point clouds, remain high, limiting real-time performance in extremely complex scenarios.
- Further work is needed to improve robustness under bandwidth constraints and to extend support for multi-user collaborative editing.
Future Work
Future directions include optimizing deformation algorithms for faster, more accurate interpolation in dynamic scenes, integrating more robust multimodal semantic understanding, and reducing computational overhead. Exploring adaptive transparency mechanisms and multi-user collaboration frameworks will enhance scalability. Additionally, extending the system to handle larger scenes and more complex interactions, possibly via hierarchical representations or learned deformation models, will broaden practical applicability. Investigating hardware acceleration and edge deployment strategies will also be crucial for real-world deployment.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个巨大的工厂里，工厂里有很多工人（代表场景中的点和元素），每个工人都在用不同的颜色和语言描述自己的工作。以前的方法就像只用一种颜色或一种语言，场景看起来很单调，也不能自由变化。现在，这项新技术就像给每个工人都配备了特别的标签，可以同时用颜色和话语描述他们，还能让工厂里的场景快速变换和调整。通过这些标签和变形工具，你可以让场景变得更丰富、更真实，还能让工厂里的每个部分都能自由互动。这样，无论你是在虚拟现实中看场景，还是想让场景自动变化，都变得更容易、更快。这就像给工厂装上了聪明的机器人助手，让一切都变得更智能、更灵活。
ELI14 Explained like you're 14
想象你在玩一个超级酷的游戏，你可以从不同的角度看场景，还能用你的话告诉游戏里的角色发生了什么。以前的游戏只能用一种方式显示场景，比如只用图片，不能说话，也不能让场景变得更快或更复杂。现在，这个新技术就像给场景装上了一个聪明的大脑，它可以理解你用不同的词描述场景，还能让场景变得更快、更真实。比如，你说“这个房间有很多颜色”，它就能用不同的颜色和细节把房间重新画出来。而且，它还能让场景变得更快，比如从20帧每秒变成60帧，每秒的画面变得更流畅。这样，你可以更自由地玩游戏，和场景互动，甚至可以自己编辑场景，让一切都变得更酷、更有趣。这个技术就像给虚拟世界装上了一个聪明的机器人助手，让一切都变得更智能、更好玩！
Abstract
3D Gaussian Splatting~(3DGS) has emerged as a promising paradigm for reconstructing streamable free-viewpoint video~(FVV) from multi-view videos. However, 3DGS-based FVVs typically lack user interaction and editing capabilities, which diminishes the immersive experience. Recent research has integrated language features from CLIP into 3DGS via distillation, enabling open-vocabulary queries and supporting many downstream applications. Nevertheless, the stringent requirements of FVV, low frame size and high FPS, make current language Gaussian representations unsuitable for language-embedded FVV. In this paper, we propose DLGStream, a novel language-embedded FVV representation that streams time-varying language features alongside Gaussian attributes to support 4D environment interaction, scene editing, and spatial intelligence. Specifically, we propose a dual-opacity dynamic language Gaussian representation, which maintains two opacity attributes for color and language features to deal with performance degradation that occurs when colors and features are jointly optimized. Furthermore, we introduce an interpolation-based deformation field to reduce temporal redundancy. This deformation field can also be used for 4D frame interpolation, boosting FVV sequences from low to high FPS. Experimental results demonstrate that DLGStream achieves superior performance in both on open-vocabulary segmentation and reconstruction quality with an average frame size of merely 43 KB. The code is available on \href{https://github.com/kkkzh/DLGStream}{https://github.com/kkkzh/DLGStream}.