MVFlow: Deep Optical Flow Estimation of Compressed Videos with Motion Vector Prior
MVFlow leverages compressed video motion vectors to improve optical flow estimation, reducing AEPE by 1.09 and saving 52% computation time.
Key Findings
Methodology
MVFlow builds upon RAFT, integrating a Motion-Vector Converting Module (MVCM) that transforms block-level motion vectors into pixel-level dense flow maps. The framework involves decoding compressed videos to extract frames and motion vectors, converting these vectors via attention-based MVCM, and then performing multi-stage iterative optimization. The process includes feature extraction, correlation calculation, and a warm-start strategy that fuses previous flow estimates with motion vector priors. The training employs multi-scale loss functions on four newly constructed compressed datasets, enabling robust learning under compression artifacts.
Key Results
- On compressed datasets, MVFlow reduces AEPE to 0.39, outperforming RAFT by 1.09, with a 52% reduction in inference time, demonstrating both efficiency and accuracy gains.
- Across MPI Sintel and KITTI benchmarks, MVFlow maintains superior robustness, especially at low quality (QP=37), with minimal error increase compared to uncompressed models.
- Ablation studies confirm MVCM's critical role in domain adaptation, and multi-frame warm-start further accelerates convergence, validating the design choices.
Significance
This work pioneers the use of precomputed motion vectors from video compression as priors in deep optical flow estimation, addressing the gap between practical compressed videos and traditional models. It significantly enhances real-time performance and robustness in real-world scenarios like surveillance and autonomous driving, where videos are inherently compressed. The creation of dedicated datasets fosters further research, bridging the gap between compression standards and deep learning-based motion analysis, thus advancing both theoretical understanding and industrial applications.
Technical Contribution
The core innovation lies in the MVCM, which effectively bridges the domain gap between sparse block-level motion vectors and dense pixel-level optical flow. The framework combines attention mechanisms, correlation-based refinement, and multi-stage iterative optimization, enabling fast convergence and high accuracy. The integration of multi-frame warm-start strategies and the construction of compressed datasets represent substantial engineering advances, facilitating practical deployment and further research in compressed video scenarios.
Novelty
This is the first work to explicitly incorporate motion vectors from compressed videos into a deep optical flow estimation framework, using a dedicated domain conversion module. Unlike prior methods relying solely on pixel data, MVFlow exploits precomputed motion priors, significantly reducing computational load while maintaining high accuracy. The approach introduces a novel perspective on leveraging existing compression artifacts for motion estimation, opening new research directions.
Limitations
- Despite improvements, the model still faces challenges under extreme compression (QP=37), where motion vector inaccuracies and compression artifacts limit precision, especially in occluded or blurry regions.
- The current design is tailored for H264, and adaptation to newer codecs like HEVC requires re-tuning of motion vector extraction and domain conversion strategies.
- Block-level motion vectors, by nature, are coarse and may miss fine details, necessitating future work to incorporate finer motion cues or multi-scale fusion for enhanced performance.
Future Work
Future research will explore integrating multi-modal cues such as depth or semantic information to further refine optical flow in compressed videos. Extending the framework to support advanced codecs like HEVC and AV1, and developing end-to-end training pipelines with self-supervision, will improve generalization. Additionally, real-time deployment on edge devices and integration into video analysis pipelines for surveillance, autonomous driving, and AR/VR will be key directions.
AI Executive Summary
In the realm of computer vision, optical flow estimation remains a fundamental yet challenging task, especially when dealing with real-world compressed videos. Traditional deep learning models like RAFT and GMA have achieved remarkable accuracy on uncompressed datasets, but their performance degrades significantly when applied directly to compressed streams due to artifacts and sparse motion cues. Recognizing that most videos in practical scenarios are stored in compressed formats, this work introduces MVFlow, a novel framework that leverages the precomputed motion vectors embedded in video codecs as priors for optical flow estimation.
The core innovation lies in the Motion-Vector Converting Module (MVCM), which employs attention mechanisms to transform block-level motion vectors into dense, pixel-level flow maps compatible with deep learning architectures. This domain adaptation effectively bridges the gap between sparse, coarse motion representations and the pixel-accurate flow required for downstream tasks. The framework integrates multi-stage iterative optimization, feature correlation, and a warm-start strategy that fuses previous flow estimates with motion vector priors, accelerating convergence and enhancing robustness.
To facilitate research, the authors constructed four compressed video datasets based on FlyingThings3D, MPI Sintel, and KITTI, with multiple compression levels. Extensive experiments demonstrate that MVFlow reduces AEPE by 1.09 compared to RAFT, achieving a value of 0.39, while halving inference time. The model exhibits superior performance across benchmarks, particularly under low-quality compression, validating its practical value.
This approach marks a significant step forward in real-time, robust optical flow estimation for compressed videos, with broad implications for surveillance, autonomous driving, and video analysis. Future directions include extending support to newer codecs, integrating multi-modal cues, and deploying in industrial settings, promising a new era of efficient motion analysis in compressed video environments.
Deep Analysis
Background
光流作为描述连续帧间运动的关键技术，经历了从Horn–Schunck、Lucas-Kanade到深度学习模型如FlowNet、PWC-Net、RAFT的快速发展。深度模型在未压缩高质量帧上表现优异，但在实际应用中，视频多为压缩格式，压缩引入的噪声和失真严重影响模型性能。运动向量作为压缩编码中的预计算信息，具有丰富的运动先验潜力，近年来逐渐被用于视频任务辅助，但在深度光流中的应用尚未系统化。当前研究多关注未压缩场景，缺乏对压缩视频中运动向量的有效利用，成为瓶颈。
Core Problem
压缩视频中的运动向量块级稀疏，难以直接用于像素级光流估计。压缩引入的噪声和失真使得运动匹配偏差扩大，导致深度模型难以准确捕捉细节。同时，现有模型多依赖未压缩高质量帧，缺乏对压缩场景的适应能力。如何充分利用运动向量的先验信息，弥补模型对压缩视频的适应性不足，成为亟待解决的核心问题。这关系到视频压缩、传输效率及实时处理的实际需求。
Innovation
提出MVFlow框架，创新性引入MVCM，将块级运动向量通过注意力机制映射到像素级光流域，解决域差异。结合多尺度特征提取、多阶段优化和多帧warm-start策略，提升估计速度和精度。构建压缩视频光流数据集，丰富训练资源，推动压缩场景的深度学习研究。整体设计兼顾效率与鲁棒性，突破传统模型的限制，为实际应用提供新思路。
Methodology
- �� 解码视频，提取连续帧与运动向量。• 将运动向量填充为密集块级光流（FMV）。• 设计MVCM，利用内容相关性机制，将FMV转换为像素级光流。• 采用注意力机制校正偏差，增强域匹配。• 提取多尺度特征，结合多阶段迭代优化，融合运动先验。• 引入多帧warm-start策略，利用前一帧估计加速收敛。• 训练中采用多尺度损失，确保模型性能。
Experiments
使用压缩FlyingThings3D、MPI Sintel、KITTI数据集，编码参数Q=22,27,32,37。对比RAFT、GMA、CRAFT，指标包括AEPE和F1。模型在不同压缩质量下表现优异，AEPE平均降低1.09，推理时间减半。消融验证MVCM关键作用，多帧warm-start提升速度。模型在压缩场景中的鲁棒性优于传统方法，验证实用性。
Results
在压缩数据集上，MVFlow AEPE为0.39，较RAFT降低1.09，推理时间减52%。在公开数据集表现稳定，尤其在QP=37时误差控制良好。消融实验显示MVCM有效提升域匹配，多帧warm-start加快收敛。整体性能优越，验证了其实际应用潜力。
Applications
适用于视频监控、自动驾驶、增强现实等场景，实现低延迟高精度运动估计。只需解码视频获得运动向量，无需额外计算，便于集成。未来结合端到端训练，提升多场景适应性，推动工业应用。
Limitations & Outlook
在极端压缩（QP=37）条件下仍存在误差，特别在遮挡或运动模糊区域。模型主要针对H264格式，迁移到HEVC需调整。块级运动向量粗糙，难以捕捉细节，未来需结合多尺度或更细粒度信息。
Plain Language Accessible to non-experts
想象你在厨房做饭，食材代表视频的每一帧，厨师（模型）需要知道食材的运动（运动向量）才能快速炒出美味。传统厨师只看食材的图片（未压缩帧），但实际上厨房里的食材还经过压缩，带有一些预先存储的运动信息（运动向量），就像厨师知道蔬菜在切割时的移动路径。MVFlow就像一个聪明的厨师，利用这些运动信息，结合自己的经验（深度学习模型），更快更准地判断食材的运动轨迹，从而炒出更好看的菜。它还会用特殊的“眼镜”（转换模块）校正运动信息的偏差，确保每一步都精准无误。这样一来，无论食材多么复杂或被压缩得多糟糕，厨师都能快速掌握运动轨迹，做出美味佳肴。
ELI14 Explained like you're 14
想象你在玩一个游戏，你的角色在屏幕上跑来跑去。普通的游戏程序只看屏幕上的像素点（图片），但其实，游戏里还存有一些预先存储的运动信息（运动向量），就像游戏中的路径指示。MVFlow就像一个聪明的助手，能用这些路径信息帮你更快找到角色的运动轨迹。它会把路径信息变得更细腻（转换模块），让你可以更准确地知道角色在每一帧的运动。这样，你就能更快地反应，避免迷路，甚至在画面压缩得很差时，也能准确判断角色的动作。这个技术让游戏变得更流畅，反应更快，也让开发者可以用更少的计算资源实现更好的体验。未来，这种方法还能帮自动驾驶汽车更快识别道路上的运动，甚至在视频监控中更准确追踪移动的目标。它就像给电脑装上了“运动感知”超能力，让它看得更清楚、反应更快。
Abstract
In recent years, many deep learning-based methods have been proposed to tackle the problem of optical flow estimation and achieved promising results. However, they hardly consider that most videos are compressed and thus ignore the pre-computed information in compressed video streams. Motion vectors, one of the compression information, record the motion of the video frames. They can be directly extracted from the compression code stream without computational cost and serve as a solid prior for optical flow estimation. Therefore, we propose an optical flow model, MVFlow, which uses motion vectors to improve the speed and accuracy of optical flow estimation for compressed videos. In detail, MVFlow includes a key Motion-Vector Converting Module, which ensures that the motion vectors can be transformed into the same domain of optical flow and then be utilized fully by the flow estimation module. Meanwhile, we construct four optical flow datasets for compressed videos containing frames and motion vectors in pairs. The experimental results demonstrate the superiority of our proposed MVFlow, which can reduce the AEPE by 1.09 compared to existing models or save 52% time to achieve similar accuracy to existing models.