GMFlow: Learning Optical Flow via Global Matching
GMFlow replaces local regression with global matching, reaching Sintel EPE 1.08 after one refinement, better than 31-step RAFT.
Key Findings
Methodology
GMFlow reformulates dense optical flow as global correspondence estimation. A shared convolutional backbone extracts 1/8-resolution features, which are enhanced by a Transformer containing self-attention, cross-attention, and FFN blocks. The model computes C=F1F2ᵀ/√D, applies softmax over all candidate pixels, and obtains sub-pixel correspondences through a coordinate-weighted average. Self-attention propagates reliable flow to occluded or out-of-boundary pixels, while a 1/4-resolution reuse stage predicts residual motion.
Key Results
- With only one refinement, GMFlow obtains Sintel train EPEs of 1.08 on clean and 2.48 on final, outperforming 31-refinement RAFT at 1.41 and 2.69. It also runs in 151 ms versus 170 ms for RAFT, with only 4.7M parameters.
- The advantage is strongest for large motion: on Sintel clean pixels moving more than 40 pixels, GMFlow reaches EPE 6.26 compared with 8.83 for 31-refinement RAFT. On Things validation, its zero- and one-refinement EPEs are 3.48 and 2.80.
- Ablations show that cross-attention is crucial: removing it raises Things EPE from 6.67 to 10.84. Removing flow propagation increases unmatched-pixel EPE on Sintel clean from 10.39 to 15.54.
Significance
The work addresses the long-standing tension between large-displacement accuracy and inference efficiency. RAFT gradually expands its effective search through many sequential updates, whereas GMFlow compares all candidate locations directly. The results show that dense optical flow can benefit from explicit matching ideas developed in SuperGlue and LoFTR, while a propagation module handles occlusion. This creates a promising foundation for real-time vision, robotics, and video analysis.
Technical Contribution
The paper makes three main technical contributions. First, it replaces local cost-volume regression with differentiable global correlation and softmax matching. Second, it uses a six-block Transformer with positional encoding, local-window self-attention, and cross-attention to create discriminative cross-frame features. Third, it propagates reliable flow using feature self-similarity and reuses the same architecture for 1/4-resolution residual refinement. Backward flow can also be obtained by transposing the correlation matrix, avoiding a second network pass.
Novelty
Unlike PWC-Net, RAFT, and related local-regression systems, GMFlow changes the formulation rather than merely enlarging the search range or adding iterations. GLU-Net uses global correlation but still relies on convolutional regression; GMFlow directly converts global similarities into correspondences through Transformer-enhanced features and softmax. The novelty is therefore both conceptual and architectural: optical flow becomes explicit dense matching.
Limitations
- The global correlation matrix has quadratic complexity in the number of feature locations. Although 1/8 features and windowed attention reduce cost, high-resolution images still impose substantial memory and computation demands.
- Softmax matching assumes that a visible counterpart exists. Occlusions, out-of-boundary pixels, repeated textures, and severe non-rigid motion can produce unreliable distributions; propagation and consistency checks only partially repair these cases.
Future Work
Future research could develop sparse or linear global attention, hierarchical candidate pruning, and explicit occlusion probabilities. Lighter Transformers and hardware-optimized matrix multiplication may improve deployment at high resolution. The same formulation could also be extended to stereo, depth, scene flow, video correspondence, and multimodal motion understanding.
AI Executive Summary
Optical flow asks where each image pixel moves between two video frames. Established systems such as PWC-Net and RAFT rely on local cost volumes followed by convolutional regression. Large motion is difficult because the search range is local; RAFT solves this by repeatedly updating the field, but 31 sequential refinements increase latency.
GMFlow adopts a different view: flow is dense global matching. A shared convolutional backbone produces features, and six Transformer blocks use self-attention and cross-attention to make them more discriminative. The model then computes all-pairs correlations, converts them into a softmax matching distribution, and obtains flow by averaging candidate coordinates. Self-attention propagates reliable estimates into unmatched regions, while one higher-resolution pass predicts residual flow.
On Sintel, one-refinement GMFlow achieves EPE 1.08 on clean and 2.48 on final, compared with 1.41 and 2.69 for 31-refinement RAFT. It is also faster: 151 ms versus 170 ms. For motions above 40 pixels, clean EPE is 6.26 versus RAFT’s 8.83. The price is quadratic global-correlation cost and sensitivity to occlusion and repeated texture. Nevertheless, the study suggests a new path from repeated local regression toward direct, efficient correspondence estimation.
Deep Analysis
Background
Since FlowNet, learned optical flow has commonly encoded candidate correspondences in cost volumes and regressed motion with convolutions. PWC-Net uses coarse-to-fine estimation, while RAFT maintains high resolution and iteratively refines flow, achieving strong benchmark results. Yet local search struggles with large displacement and RAFT’s sequential updates increase latency. Inspired by explicit matching in SuperGlue and LoFTR, GMFlow recasts dense flow as global correspondence estimation.
Core Problem
Given consecutive frames I1 and I2, the model must assign a 2D displacement to every pixel while handling large motion, occlusion, out-of-boundary locations, and sub-pixel movement. A local cost volume fixes a search radius R; RAFT uses R=4 and expands effective reach through repeated updates. Direct argmax matching is not differentiable, while plain softmax cannot naturally represent invisible pixels.
Innovation
- �� Global matching: compare every feature in one frame with all features in the other and use softmax for differentiable correspondence.
- �� Transformer enhancement: six self/cross-attention blocks with DETR-style 2D sine-cosine positional encoding.
- �� Flow propagation: use feature self-similarity to transfer reliable motion to unmatched pixels.
- �� Residual refinement: reuse GMFlow at 1/4 resolution with local windows to recover fine motion details.
Methodology
- �� Feature extraction: a weight-sharing convolutional backbone generates F1,F2∈R^(H×W×D), mainly at 1/8 image resolution, with D=128.
- �� Feature enhancement: positional encoding is added and six Transformer blocks are applied; 2×2 shifted local windows reduce attention cost.
- �� Matching: C=F1F2ᵀ/√D, M=softmax(C), correspondence Ĝ=MG, and flow V=Ĝ−G.
- �� Propagation: softmax(F̂1F̂1ᵀ/√D)V̂ spreads flow from reliable matches.
- �� Refinement and loss: upsample 1/8 flow, warp the second feature, perform 9×9 local matching at 1/4 resolution, and train with multi-stage L1 loss using γ=0.9.
Experiments
Training uses FlyingChairs for 100K iterations and FlyingThings3D for 200K–800K iterations, followed by evaluation on Sintel and KITTI. Metrics include EPE, KITTI F1-all, and motion bins of 0–10, 10–40, and over 40 pixels. Baselines include RAFT and cost-volume-plus-convolution models. Ablations vary Transformer depth, window splits, attention components, global versus local matching, propagation, and weight sharing.
Results
The six-block model reaches Things validation EPE 6.67 and Sintel clean/final EPE 2.28/3.44 before refinement. One refinement improves these to 2.80, 1.08, and 2.48. Against 31-step RAFT, GMFlow is better on Sintel clean/final, 1.08/2.48 versus 1.41/2.69, and faster, 151 versus 170 ms. Removing cross-attention raises Things EPE to 10.84, confirming that cross-frame reasoning is essential.
Applications
GMFlow is relevant to video tracking, motion segmentation, robotics, autonomous driving, and frame interpolation. Deployment benefits from GPU matrix multiplication and moderate input resolution because global correlation consumes memory. Its low parameter count and single refinement suit latency-sensitive systems. Transposed correlation also provides backward flow efficiently for forward-backward occlusion checks.
Limitations & Outlook
The principal bottleneck is quadratic all-pairs correlation, which remains expensive at high resolution. Repeated textures, severe occlusion, fast non-rigid motion, and out-of-boundary pixels can make softmax assignments unreliable; propagation only partially corrects them. The paper mainly evaluates standard benchmarks, so robustness to camera noise, real-world domain shift, and extreme motion remains uncertain. Sparse matching, hierarchical search, and explicit uncertainty modeling are natural next steps.
Plain Language Accessible to non-experts
Imagine two enormous classroom seating charts. A traditional method lets each student inspect only a few nearby seats. If a student suddenly runs across the room, the method must repeatedly guess a direction, which is slow. GMFlow lets every student compare themselves with every seat in the second chart at once, using clues such as clothing, backpack, and nearby classmates. The most similar seats receive the highest scores; turning all scores into probabilities and averaging their positions gives the movement.
Some students may be behind a wall or outside the room, so no true seat exists. The system observes nearby students and passes trustworthy movement patterns into those gaps. It then enlarges the chart and checks small details. This direct global search is powerful for long jumps, but comparing everyone with everyone is computationally expensive, and identical-looking students can still be confused.
ELI14 Explained like you're 14
Suppose you are finding the same character in two video-game screenshots. A normal method checks only a few nearby squares. If the character teleports far away, it must keep trying corrections again and again, like RAFT. GMFlow opens the whole map and compares each spot in the first screenshot with every spot in the second one.
Similarity is not just color. The model also looks at the background and at information from the other screenshot. Its Transformer is like a smart teammate who notices the whole scene before deciding where the character went. After finding most characters, it studies nearby characters’ movements to guess the path of someone hidden or partly off-screen.
The results are strong: on Sintel, one refinement gives clean EPE 1.08 and final EPE 2.48, while 31-refinement RAFT gets 1.41 and 2.69. GMFlow is faster too, 151 milliseconds versus 170. For objects moving more than 40 pixels, its error is especially smaller.
The catch? Looking everywhere costs memory, especially for large images. Similar-looking objects can still fool it. If researchers invent a faster way to narrow the search, this idea could become very useful for phones, robots, and self-driving cars!
Glossary
Optical Flow
Optical flow describes the two-dimensional motion of image pixels between consecutive frames. It is commonly evaluated by endpoint error, the Euclidean distance between predicted and ground-truth motion vectors.
The paper formulates it as dense pixel correspondence.
Global Matching
Global matching compares a feature with every candidate location in the other image rather than searching only a local neighborhood. It handles large displacement directly but increases computation.
It is GMFlow’s central reformulation.
Cost Volume
A cost volume stores similarity or matching scores for candidate displacements. Conventional optical-flow systems use convolutions to regress motion from a local cost volume.
GMFlow replaces this dominant local-regression pipeline.
Cross-Attention
Cross-attention uses one feature set as queries and another as keys and values, explicitly modeling relationships between images. It improves feature discrimination for matching.
Ablations identify it as the most important Transformer component.
EPE
Endpoint error is the mean Euclidean distance between predicted and ground-truth two-dimensional flow vectors. Lower EPE indicates more accurate flow.
It is the primary metric on Sintel and Things.
Open Questions Unanswered questions from this research
- 1 How can global matching preserve its large-displacement advantage on high-resolution video without quadratic memory and computation? Sparse, hierarchical, or approximate correspondence mechanisms are needed.
- 2 Occlusion and repeated texture remain difficult because propagation only repairs unreliable softmax matches indirectly. Stronger uncertainty estimation and explicit visibility modeling remain open problems.
Applications
Immediate Applications
Video motion analysis
Video systems can use GMFlow for motion segmentation, object tracking, and frame interpolation. Its single refinement offers strong accuracy with limited sequential computation, but deployment should control image resolution and use consistency checks for occlusions.
Robotics and autonomous driving
Robots can use flow for obstacle avoidance, camera-motion estimation, and target tracking. GMFlow is attractive for fast-moving scenes, although real deployment must assess GPU memory, sensor noise, and domain shift from synthetic training data.
Long-term Vision
A unified visual-correspondence engine
The global-matching paradigm could extend to stereo, depth, scene flow, cross-view matching, and multimodal motion understanding. Major obstacles are high-resolution efficiency, reliable occlusion reasoning, and stable cross-task training.
Abstract
Learning-based optical flow estimation has been dominated with the pipeline of cost volume with convolutions for flow regression, which is inherently limited to local correlations and thus is hard to address the long-standing challenge of large displacements. To alleviate this, the state-of-the-art framework RAFT gradually improves its prediction quality by using a large number of iterative refinements, achieving remarkable performance but introducing linearly increasing inference time. To enable both high accuracy and efficiency, we completely revamp the dominant flow regression pipeline by reformulating optical flow as a global matching problem, which identifies the correspondences by directly comparing feature similarities. Specifically, we propose a GMFlow framework, which consists of three main components: a customized Transformer for feature enhancement, a correlation and softmax layer for global feature matching, and a self-attention layer for flow propagation. We further introduce a refinement step that reuses GMFlow at higher feature resolution for residual flow prediction. Our new framework outperforms 31-refinements RAFT on the challenging Sintel benchmark, while using only one refinement and running faster, suggesting a new paradigm for accurate and efficient optical flow estimation. Code is available at https://github.com/haofeixu/gmflow.