GeCo: Evaluating Geometric Consistency for Video Generation via Motion and Structure
GeCo evaluates geometric consistency in video generation by fusing motion and depth priors to reveal distortions and occlusion inconsistencies.
Key Findings
Methodology
GeCo fuses residual motion and depth priors to create interpretable dense consistency maps, detecting geometric deformation and occlusion inconsistencies in static scenes. It uses optical flow, depth, and camera pose estimations to measure geometric consistency through motion and structure cues.
Key Results
- On WarpBench, GeCo's motion cue achieved 64.90% AP on CO3D-Warp, significantly outperforming MEt3R's 16.26%.
- On OccluBench, the fused cue achieved 83.48% AP, outperforming the structure cue's 62.36%.
- GeCo's noise level on real videos is lower than generated videos, confirming its robustness.
Significance
GeCo provides a new perspective for evaluating geometric consistency in video generation, addressing the limitations of existing methods that cannot simultaneously detect geometric deformation and occlusion inconsistencies. It enhances model evaluation and offers guidance for optimizing geometric consistency during generation.
Technical Contribution
GeCo introduces a differentiable geometric consistency metric that combines motion and structure cues into a unified error map. This method can optimize geometric consistency during inference without requiring training.
Novelty
GeCo is the first to jointly detect geometric deformation and occlusion inconsistencies, overcoming the limitations of existing methods that only detect one type of artifact.
Limitations
- GeCo relies on the accuracy of optical flow and depth estimation, which may affect results when estimation errors are large.
- Performance may degrade in complex dynamic scenes.
Future Work
Future work could explore GeCo's application in dynamic scenes and integrate more geometric information to enhance its robustness and accuracy.
AI Executive Summary
Video generation models can produce photorealistic videos but often violate multiview geometry, resulting in non-rigid deformations and occlusion inconsistencies. Existing evaluation methods struggle to capture both types of artifacts simultaneously. GeCo introduces a geometric consistency metric by fusing residual motion and depth priors, creating interpretable dense consistency maps that reveal these artifacts. Experiments show GeCo's superior performance on WarpBench and OccluBench, accurately localizing geometric deformation and occlusion inconsistencies. GeCo serves not only as an evaluation tool but also as a guidance loss during inference to reduce geometric artifacts in generation. Future work will explore its application in dynamic scenes.
Deep Analysis
Background
Video generation technology has made significant progress in recent years, capable of producing realistic videos. However, these models often exhibit geometric deformation and occlusion inconsistencies when generating static scenes. These artifacts not only affect the visual quality of videos but also hinder downstream applications such as video world models and 3D asset creation.
Core Problem
Existing evaluation methods struggle to detect both geometric deformation and occlusion inconsistencies. General video benchmarks lack explicit 3D geometric evaluation, and sparse point matching methods perform poorly in occluded regions.
Innovation
GeCo provides a unified geometric consistency metric by fusing motion and depth cues. It uses optical flow and depth reprojection errors to detect geometric deformation and occlusion inconsistencies, creating interpretable dense consistency maps.
Methodology
- �� Use optical flow estimation and depth reprojection to calculate motion and structure consistency. • Predict depth maps and camera poses using a geometry foundation model. • Fuse motion and structure cues into a unified error map. • Serve as a guidance loss during inference to optimize geometric consistency in generation.
Experiments
Experiments were conducted on WarpBench and OccluBench to evaluate geometric deformation and occlusion inconsistencies, respectively. WarpBench simulates non-rigid deformation by injecting thin-plate-spline warps, while OccluBench evaluates artifact localization ability through controlled occlusion inconsistencies.
Results
On WarpBench, GeCo's motion cue achieved 64.90% AP on CO3D-Warp. On OccluBench, the fused cue achieved 83.48% AP. GeCo's noise level on real videos is lower than generated videos, confirming its robustness.
Applications
GeCo can be used for evaluating and optimizing video generation models, especially in applications requiring geometric consistency, such as video world models and 3D asset creation.
Limitations & Outlook
GeCo relies on the accuracy of optical flow and depth estimation, which may affect results in complex dynamic scenes. Future work could integrate more geometric information to enhance its robustness.
Plain Language Accessible to non-experts
Imagine watching a movie where objects sometimes distort or disappear and reappear. GeCo acts like a movie critic, pinpointing these unnatural spots. By analyzing the motion and depth of objects, it judges whether the movie follows real-world rules. Just like using a magnifying glass to observe details, GeCo helps filmmakers fix these issues, making the movie look more realistic.
ELI14 Explained like you're 14
Imagine playing a game where characters sometimes distort or suddenly disappear and reappear. GeCo is like a super detective, spotting these strange places. By analyzing the characters' movements and positions, it judges whether the game follows real-world rules. This way, game developers can fix these issues, making the game more fun!
Glossary
Geometric Consistency
The ability of video generation to maintain consistent object appearance across different viewpoints.
Used to evaluate geometric deformation and occlusion inconsistencies in video generation models.
Optical Flow
A vector field representing pixel motion in images, reflecting object movement.
Used to calculate motion consistency.
Depth Reprojection
The process of transforming depth information from one viewpoint to another.
Used to calculate structure consistency.
Occlusion Inconsistency
Changes or errors in objects when they reappear after being occluded.
Detected by GeCo to identify such artifacts.
Residual Motion
The difference between optical flow and camera-induced rigid motion.
Used to evaluate motion consistency.
Open Questions Unanswered questions from this research
- 1 How to improve GeCo's robustness in dynamic scenes? Current methods perform poorly under complex motion, requiring more geometric information.
- 2 How to reduce GeCo's reliance on the accuracy of optical flow and depth estimation?
- 3 How to better detect occlusion inconsistencies across multiple viewpoints?
Applications
Immediate Applications
Video Generation Evaluation
GeCo can be used to evaluate the geometric consistency of video generation models, helping developers identify and fix artifacts.
Long-term Vision
3D Asset Creation
By improving geometric consistency in video generation, GeCo can facilitate the creation of more realistic 3D assets.
Abstract
We introduce GeCo, a geometry-grounded metric for jointly detecting geometric deformation and occlusion-inconsistency artifacts in static scenes. By fusing residual motion and depth priors, GeCo produces interpretable, dense consistency maps that reveal these artifacts. We use GeCo to systematically benchmark recent video generation models, uncovering common failure modes, and further employ it as a training-free guidance loss to reduce deformation artifacts during video generation.