MVTrack: Ultrafast Appearance-Free Moving Object Tracking from Compressed Bitstreams
MVTrack uses H.264 motion vectors for ultrafast, appearance-free moving object tracking; achieves high accuracy with minimal parameters and FLOPs.
Key Findings
Methodology
MVTrack integrates a lightweight MVDet detector and MVLink association module, operating directly on H.264 motion vectors. MVDet employs a modified CenterNet architecture to detect moving targets from motion fields, while MVLink uses a kinematic model based on Kalman filtering to link stationary and moving targets, addressing fragmentation issues. The system foregoes pixel reconstruction, enabling real-time performance on CPU hardware. The pipeline involves extracting motion vectors, encoding block structure, and end-to-end training with multi-task losses including heatmap, size, offset, and IoU. This design ensures efficient, privacy-preserving multi-object tracking suitable for edge deployment.
Key Results
- On VIRAT dataset, MVTrack achieves a detection mAP50:95 of 72.33%, surpassing YOLO26n by approximately 10%. It maintains a HOTA score of 64.36%, with only 41K parameters and a latency of 5.21ms on CPU, which is 8.6 times faster than RGB baselines and reduces FLOPs by 40 times. The system remains robust at low bitrates (1 Mbps) with detection mAP of 73.12%. Ablation studies confirm the importance of temporal modeling and block structure features, with cross-scene tests showing strong transferability.
- Compared to RGB detectors, MVTrack with ByteTrack achieves comparable detection accuracy and superior identity preservation, especially in scenarios with static or occluded targets. The model's efficiency enables real-time multi-stream processing on consumer hardware, making it suitable for large-scale surveillance. The use of motion vectors also enhances privacy by avoiding pixel-level reconstruction, addressing privacy concerns in surveillance applications.
- Ablation results indicate that temporal aggregation and encoding block metadata significantly improve detection robustness. Cross-scene validation demonstrates good generalization across different environments and compression settings. The system's reliance on motion cues makes it resilient to appearance variations but sensitive to target overlaps and static target fragmentation, suggesting directions for future integration of appearance features.
Significance
This work fundamentally shifts the paradigm of video object tracking by demonstrating that compressed domain information alone can enable accurate, real-time, privacy-preserving surveillance. It addresses key industry challenges such as computational cost, latency, and privacy concerns, especially relevant for edge devices and large-scale deployments. By eliminating the need for pixel reconstruction, MVTrack reduces hardware requirements and energy consumption, paving the way for scalable, low-cost surveillance solutions. Its success suggests a new research direction where motion vectors become primary cues, opening opportunities for further innovations in compressed video analytics.
Technical Contribution
The main technical contribution is the development of a fully compressed-domain tracking framework that leverages motion vectors as the primary representation, bypassing RGB decoding entirely. The system employs a modified CenterNet detection architecture tailored for motion vector inputs, combined with a novel kinematic association module (MVLink) based on Kalman filtering to handle stationary and fragmented targets. The approach introduces end-to-end training with multi-task losses, achieves high detection and tracking accuracy with minimal parameters, and demonstrates real-time CPU performance. This work bridges the gap between traditional pixel-based methods and efficient compressed domain analysis, offering theoretical guarantees of robustness and scalability.
Novelty
This is the first framework to perform complete moving object detection and tracking solely using H.264 motion vectors, without any pixel-level information. Unlike prior works that rely on RGB reconstruction or intermediate features, MVTrack directly exploits the compressed domain signals, significantly reducing computational load and privacy risks. Its integration of a lightweight detection network with a kinematic association module tailored for motion vectors represents a novel approach, establishing a new paradigm in surveillance video analysis that emphasizes efficiency, privacy, and scalability.
Limitations
- In scenarios with highly overlapping objects or similar motion patterns, the coarse spatial resolution of motion vectors can cause localization ambiguities and merging errors. Static or long-stationary targets pose challenges due to fragmentation, requiring further integration of appearance cues.
- The system's performance depends on encoding parameters; extreme compression or low frame rates degrade detection accuracy. It is less effective in scenes with complex background motion, such as swaying trees or dynamic backgrounds.
- While highly efficient, the current model may struggle with fine-grained localization and identity preservation over extended static periods, indicating the need for hybrid approaches combining appearance features.
Future Work
未来将结合外观特征和多模态信息，提升静止目标的连续追踪能力。探索更鲁棒的运动模型，适应不同编码参数和极端压缩环境。还将优化模型结构，支持更低码率和更长GOP长度的编码参数，推动压缩域目标追踪技术的广泛应用。
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在看一场足球比赛，但你不用看球员的脸，只看他们跑动的轨迹。只要有人在场上跑，系统就能知道是谁在动，不用看具体长什么样。这就像MVTrack一样，它不用看图片，只用看运动的线条来追踪目标。这样既快又省事，还能保护隐私，因为不用看脸或车牌。即使有人静止不动一会儿，只要开始跑，系统就能立刻发现。它就像用风的方向判断风有多大，不需要看风吹到的树叶。只用运动信息，就能找到所有在动的东西，非常聪明又实用。
Abstract
Deploying modern video trackers at scale is bottlenecked by the computational cost of RGB-based object detectors. To this end, we present MVTrack, an ultrafast tracker for moving objects that operates directly on H.264 bitstreams. MVTrack combines MVDet, a lightweight detector for motion vector fields, with MVLink, a minimalist kinematic association module. On VIRAT, MVTrack outperforms YOLO26n while using 60$\times$ fewer parameters, requiring 40$\times$ fewer FLOPs, and reducing CPU latency by 8.6$\times$. These results demonstrate that compressed video data alone can enable accurate and scalable surveillance tracking, thereby bypassing the need for pixel reconstruction.