Motion-o: Trajectory-Grounded Video Reasoning
Motion-o enhances video reasoning by making motion explicit, improving trajectory consistency.
Key Findings
Methodology
Motion-o introduces Motion Chain of Thought (MCoT) to make object motion explicit. It densifies sparse spatio-temporal annotations into object tracks and derives motion descriptors from centroid displacement and box-area change. Training involves complementary rewards for trajectory consistency and visual grounding, including a perturbation-based signal penalizing unchanged motion descriptions when temporal evidence is removed.
Key Results
- Motion-o consistently improves trajectory-faithful reasoning across multiple video understanding benchmarks, notably increasing mAM and mLGM by 2.9 and 3.9 points on the V-STAR benchmark.
- On VideoMME and WorldSense benchmarks, Motion-o improves by 6.1 and 4.0 points, respectively, demonstrating enhanced video understanding through explicit trajectory reasoning.
- Ablation studies confirm the importance of visual grounding rewards for consistent and accurate motion descriptions.
Significance
Motion-o complements existing vision-language model pipelines by converting implicit dynamics into verifiable evidence, addressing the challenge of supervising, verifying, or penalizing dynamic and trajectory-dependent claims, thus enhancing interpretability and accuracy in video reasoning.
Technical Contribution
Without architectural changes, Motion-o enhances trajectory consistency reasoning through an explicit motion interface. It introduces Motion Chain of Thought (MCoT) as a structured intermediate evidence that can be parsed, supervised, rewarded, and inspected.
Novelty
Motion-o is the first to make motion an explicit and rewardable component of the model's reasoning trace, unlike previous work that treated motion as a low-level visual signal.
Limitations
- Motion-o may inaccurately describe motion in complex scenes, particularly when objects move rapidly or are occluded.
- The model's reliance on motion descriptions may lead to incorrect reasoning when insufficient visual evidence is available.
Future Work
Future research directions include extending Motion-o to handle more complex scenes and object interactions, and exploring its applications in other fields such as robotics and autonomous driving.
AI Executive Summary
In the field of video reasoning, existing models often rely on spatio-temporal evidence chains to localize objects at specific timestamps. However, these models frequently overlook the motion connecting observations, the 'how' of motion. Motion-o addresses this by introducing Motion Chain of Thought (MCoT), making trajectories explicit and verifiable. MCoT uses discrete <motion/> tags to summarize object motion in terms of direction, speed, and scale change.
Motion-o significantly improves trajectory-faithful reasoning performance across multiple video understanding benchmarks. Notably, on the V-STAR benchmark, Motion-o increases mAM and mLGM by 2.9 and 3.9 points, respectively. This suggests that an explicit motion interface can complement existing vision-language model pipelines by converting implicit dynamics into verifiable evidence.
However, Motion-o may inaccurately describe motion in complex scenes, particularly when objects move rapidly or are occluded. Future research directions include extending Motion-o to handle more complex scenes and object interactions, and exploring its applications in other fields such as robotics and autonomous driving.
Deep Analysis
Background
Video reasoning is a crucial area in computer vision, aiming to understand and infer dynamic changes in scenes by analyzing video content. With the advancement of deep learning, vision-language models (VLMs) have been widely applied in video reasoning. However, these models often rely on spatio-temporal evidence chains to localize objects at specific timestamps, neglecting the motion information connecting observations.
Core Problem
Existing video reasoning models struggle with dynamic and trajectory-dependent claims because they often lack an explicit representation of object motion between different timestamps, making it difficult to supervise, verify, or penalize unsupported claims.
Innovation
Motion-o introduces Motion Chain of Thought (MCoT) to make object motion explicit. MCoT uses discrete <motion/> tags to summarize object motion in terms of direction, speed, and scale change, making trajectories explicit and verifiable. This innovation allows models to more accurately reason about dynamic changes in videos.
Methodology
- �� Motion-o densifies sparse spatio-temporal annotations into object tracks.
- �� Derives motion descriptors from centroid displacement and box-area change.
- �� Uses complementary rewards for trajectory consistency and visual grounding.
- �� Includes a perturbation-based signal penalizing unchanged motion descriptions when temporal evidence is removed.
Experiments
Experiments were conducted on multiple video understanding benchmarks, including V-STAR, VideoMME, and WorldSense. The benchmarks used include datasets like STGR and PLM. The experimental design includes ablation studies to verify the importance of visual grounding rewards for consistent and accurate motion descriptions.
Results
On the V-STAR benchmark, Motion-o increases mAM and mLGM by 2.9 and 3.9 points, respectively. On VideoMME and WorldSense benchmarks, Motion-o improves by 6.1 and 4.0 points. Ablation studies show that visual grounding rewards are crucial for consistent and accurate motion descriptions.
Applications
Motion-o can be applied in video surveillance, autonomous driving, and robotics, helping systems more accurately understand and predict dynamic changes. These applications require high-precision motion descriptions and trajectory reasoning.
Limitations & Outlook
Motion-o may inaccurately describe motion in complex scenes, particularly when objects move rapidly or are occluded. Additionally, the model's reliance on motion descriptions may lead to incorrect reasoning when insufficient visual evidence is available.
Plain Language Accessible to non-experts
Imagine watching a soccer game. Traditional video analysis is like a referee who only focuses on where players are at each moment, ignoring how they move from one place to another. Motion-o is like a smart coach who not only sees where players are but also analyzes their movement paths, speed, and direction. This allows it to better understand the dynamic changes of the game and make more accurate judgments.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super cool game where the characters not only appear on the screen but also move around. Traditional game analysis is like only focusing on where the characters are at each moment, ignoring how they move. Motion-o is like a super smart assistant that not only sees where the characters are but also analyzes their movement paths, speed, and direction. This helps it understand the game's dynamic changes better and helps you make better game decisions!
Glossary
Motion Chain of Thought
A structured reasoning step that summarizes object motion using discrete <motion/> tags for direction, speed, and scale change.
Used in Motion-o to make object motion explicit.
Vision-Language Model
A model that combines visual and language information for reasoning, commonly used in video and image analysis.
Motion-o is an extension of vision-language models.
Trajectory Consistency
Refers to the model's ability to maintain consistency in object motion trajectories during reasoning.
Motion-o enhances trajectory consistency through reward signals.
Visual Grounding
Refers to the model's ability to match and locate language descriptions with visual evidence.
Motion-o uses visual grounding rewards to improve motion description accuracy.
Perturbation Signal
A signal used to penalize unchanged motion descriptions when temporal evidence is removed.
Used in Motion-o's training process.
Open Questions Unanswered questions from this research
- 1 How to improve motion description accuracy in complex scenes, especially when objects move rapidly or are occluded.
- 2 How to reduce the model's reliance on motion descriptions to avoid incorrect reasoning when insufficient visual evidence is available.
Applications
Immediate Applications
Video Surveillance
Motion-o can enhance the dynamic analysis capabilities of video surveillance systems, helping to identify abnormal behaviors and events.
Long-term Vision
Autonomous Driving
By providing more accurate trajectory reasoning, Motion-o can help autonomous driving systems better understand and predict dynamic changes on the road.
Abstract
Recent video reasoning models increasingly produce spatio-temporal evidence chains that localize objects at specific timestamps. While these traces improve interpretability by grounding \emph{where} and \emph{when} evidence appears, they often leave the motion connecting observations, the \textit{how}, implicit. This makes dynamic and trajectory-dependent claims difficult to supervise, verify, or penalize when unsupported by the video. We formalize this missing component as Spatial-Temporal-Trajectory (STT) reasoning and introduce \textbf{Motion-o}, a motion-centric extension to vision-language models (VLMs) that makes trajectories explicit and verifiable. Motion-o augments evidence chains with Motion Chain of Thought (MCoT), a structured pathway that represents object motion through a discrete \texttt{<motion/>} tag summarizing direction, speed, and scale change. To supervise MCoT, we densify sparse spatio-temporal annotations into object tracks and derive motion descriptors from centroid displacement and box-area change. We then train with complementary rewards for trajectory consistency and visual grounding, including a perturbation-based signal that penalizes motion descriptions that remain unchanged when temporal evidence is removed. Across multiple video understanding benchmarks, Motion-o consistently improves trajectory-faithful reasoning without architectural modifications. These results suggest that an explicit motion interface can complement existing VLM pipelines by converting implicit dynamics into verifiable evidence. Code is available at~\href{https://github.com/ostadabbas/Motion-o}{\faGithub\ \texttt{ostadabbas/Motion-o}}.