OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams

Visual agents are increasingly deployed in real-time streaming environments, from camera surveillance to augmented reality devices, requiring agents to update their beliefs online under tight latency and memory budgets. Traditional vision foundation models often focus on image semantic perception, offline temporal modeling, or spatial geometry, leading to fragmentation within the field. In recent years, language models have achieved generality with a single autoregressive backbone trained under next-token prediction. However, vision tasks differ not only in supervision but also in the nature of their outputs, such as discrete labels, segmentation masks, dense depth, 3D geometry, and temporally evolving predictions. This has led to the emergence of specialized foundation models, such as image encoders, video models, and geometric experts. While effective in-domain, these models typically learn representations tailored to a narrow objective, making it difficult to transfer across tasks.

The core problem is how to unify perception, reconstruction, and action in the vision domain to achieve general-purpose visual understanding. Current vision foundation models are fragmented in semantic, temporal, and spatial geometry, making it challenging to generalize across tasks without fine-tuning the backbone. Additionally, existing models often rely on expensive re-training, re-tokenization of outputs, or architectural adjustments to the generative head, complicating the realization of unified visual representations. Therefore, the key research question is whether a representation can be learned that supports multiple downstream tasks without modifying or fine-tuning the backbone.

OmniStream's core innovations include proposing a unified streaming visual backbone capable of generalizing across semantic, spatial, and temporal reasoning without fine-tuning the backbone. • Introducing causal spatiotemporal attention ensures strict temporal causality and enables efficient frame-by-frame inference via a persistent KV-cache, avoiding re-computation over past frames. • Proposing 3D Rotary Positional Embeddings (3D-RoPE) extends 2D RoPE to the spatiotemporal domain, enhancing the model's ability to handle long video streams. • Adopting a unified multi-task pre-training framework that couples static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment, encouraging representations that are temporally coherent, geometrically grounded, and language-aligned.

OmniStream's methodology includes the following key steps: • Employing causal spatiotemporal attention with a persistent KV-cache for efficient frame-by-frame inference, avoiding re-computation over past frames. Input: current frame and historical context; Output: composite output state. • Using 3D Rotary Positional Embeddings (3D-RoPE) to extend 2D RoPE to the spatiotemporal domain, enhancing the model's ability to handle long video streams. Input: non-overlapping patches per frame; Output: dense spatiotemporal feature maps and global semantics. • Adopting a unified multi-task pre-training framework that couples static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment. Input: set of views (global/local crops from an image or video clip); Output: global semantic consistency and patch-level discriminative features.

The experimental design includes multi-task pre-training on 29 datasets, covering static images, dynamic videos, and geometric 3D/4D scenes. Baselines used include DINOv3, V-JEPA, CUT3R, etc. Evaluation metrics include image classification accuracy, video action recognition accuracy, video depth estimation absolute relative error, etc. Key hyperparameters include sequence length T=16, optimizer Adam, learning rate 1e-4. Ablation studies analyze the impact of causal spatiotemporal attention and 3D-RoPE.

Results analysis shows that OmniStream excels in multiple tasks. It achieves 68.5% accuracy on the SSv2 dataset, significantly outperforming DINOv3's 54.0%. In online 3D reconstruction tasks, OmniStream achieves absolute relative errors of 0.314, 0.072, and 0.136 on the Sintel, BONN, and KITTI datasets, respectively. Additionally, in VLM and VLA tasks, OmniStream demonstrates strong spatial reasoning capabilities, achieving a leading score of 70.6% on the VSI-Bench benchmark.

OmniStream's application scenarios include: • Real-time video surveillance: enabling efficient frame-by-frame processing for real-time monitoring and analysis of dynamic scenes. • Augmented reality devices: supporting real-time updates and interactions from the user's perspective, enhancing user experience. • Robotic manipulation: supporting complex robotic manipulation tasks through a unified visual stream representation, achieving more efficient task execution.

OmniStream's limitations and outlook include: • Performance degradation when handling very long video sequences, as its pre-training temporal window is fixed at 16 frames. • In certain complex geometric reasoning tasks, OmniStream may not fully replace specialized geometric expert models. • Due to the model's complexity, training and inference are computationally expensive, potentially unsuitable for resource-constrained environments. Future research directions include optimizing OmniStream's performance on long sequences, exploring more efficient causal spatiotemporal attention mechanisms, and applications in resource-constrained environments.