MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs
MotionBlind reveals Video-LLMs' inability to accurately understand motion in videos; only Gemini3.1 Pro passes.
Key Findings
Methodology
MotionBlind uses a self-recorded video contrastive benchmark to evaluate physical motion attributes like speed, magnitude, and direction. Each instance consists of two nearly identical video clips differing only in motion, paired with four yes/no questions, requiring models to answer all correctly to score.
Key Results
- Gemini3.1 Pro is the only model to pass MotionBlind, achieving 60% accuracy, while other open models hover near the 6.25% chance level.
- Experiments show that video and correct temporal order are necessary for solving MotionBlind; text or shuffled video cannot solve the task.
- Even increasing frame numbers or using smarter frame selection strategies, models fail to significantly improve motion understanding.
Significance
MotionBlind challenges Video-LLMs' ability to understand physical motion, revealing deficiencies in speed, magnitude, and direction recognition. This has significant implications for supervision and evaluation in world models relying on video understanding.
Technical Contribution
Introduced a strict contrastive benchmark, MotionBlind, focusing on physical motion attributes, complementing TimeBlind's semantic reasoning. Ensures motion is the only signal source by controlling variables.
Novelty
MotionBlind is the first to center physical motion attributes in a contrastive benchmark, differing from previous semantic reasoning benchmarks by emphasizing independent evaluation of speed, magnitude, and direction.
Limitations
- Open models completely fail in recognizing speed and magnitude, indicating current techniques cannot effectively encode these motion attributes.
- Even the most advanced model, Gemini3.1 Pro, performs poorly in speed recognition, revealing a representational gap.
- Existing frame selection strategies fail to address the impact of temporal order on motion understanding.
Future Work
Future research should focus on improving models' understanding of physical motion attributes, exploring new frame selection strategies and temporal sequence modeling methods to enhance Video-LLMs' motion recognition capabilities.
AI Executive Summary
MotionBlind is an innovative contrastive benchmark designed to evaluate video large language models (Video-LLMs) on their ability to understand physical motion. Existing Video-LLMs perform well in recognizing static content but show significant deficiencies in motion understanding. MotionBlind uses self-recorded video contrastive instances to reveal these models' shortcomings in speed, magnitude, and direction recognition.
Experimental results show that only the Gemini3.1 Pro model passes the MotionBlind evaluation, while other open models fail to achieve expected accuracy. This indicates a significant gap in current models' understanding of physical motion, rendering them unreliable for supervision and evaluation in world models.
Future research directions include developing new frame selection strategies and temporal sequence modeling methods to improve Video-LLMs' understanding of physical motion attributes. This is crucial for enhancing AI performance in dynamic environments.
Deep Analysis
Background
Video large language models (Video-LLMs) have been increasingly used as perceptual front ends for world models, assuming they can read motion information. However, existing research shows significant deficiencies in these models' motion understanding, particularly in recognizing speed, magnitude, and direction.
Core Problem
The core problem is Video-LLMs' inability to accurately recognize motion attributes in videos, such as speed, magnitude, and direction. This poses challenges for supervision, reward, and evaluation in world models relying on video understanding.
Innovation
MotionBlind uses a self-recorded video contrastive benchmark focusing on physical motion attributes. Each instance consists of two nearly identical video clips differing only in motion, ensuring motion is the only signal source.
Methodology
- �� Use self-recorded video contrastive benchmark to evaluate physical motion attributes
- �� Each instance consists of two nearly identical video clips paired with four yes/no questions
- �� Require models to answer all correctly to score, ensuring motion is the only signal source
Experiments
Experimental design includes a controlled study of six open and two frontier Video-LLMs, evaluating whether the video is present, whether frames are shown in correct temporal order, and frame sampling strategies.
Results
Experimental results show that only the Gemini3.1 Pro model passes the MotionBlind evaluation, while other open models fail to achieve expected accuracy, indicating a significant gap in current models' understanding of physical motion.
Applications
MotionBlind can be used to evaluate Video-LLMs' ability to understand physical motion, aiding in the development of more reliable world model perceptual front ends.
Limitations & Outlook
Existing models completely fail in recognizing speed and magnitude, indicating current techniques cannot effectively encode these motion attributes. Even the most advanced model, Gemini3.1 Pro, performs poorly in speed recognition, revealing a representational gap.
Plain Language Accessible to non-experts
Imagine watching two video clips of a person walking across a room. While you can clearly see what they are wearing and the furniture in the room, you can't tell how fast they are walking or in which direction. This is the problem MotionBlind reveals. It's like a test that requires models not only to recognize static content but also to understand motion. Just like watching a movie, you need to know not only who the characters are but also understand their actions and speed.
ELI14 Explained like you're 14
Imagine playing a game where two characters are walking in the same room. While you can see what clothes they're wearing and the furniture in the room, you can't tell which character is walking faster or in a different direction. That's the problem MotionBlind reveals. It's like a test that requires models not only to recognize static content but also to understand motion. Just like watching a movie, you need to know not only who the characters are but also understand their actions and speed. Isn't that interesting?
Glossary
Video-LLMs
Large language models designed to process video content, assuming they can read motion information.
Used in the study to evaluate models' motion understanding capabilities.
MotionBlind
A contrastive benchmark for evaluating models' understanding of physical motion attributes.
Used to reveal deficiencies in Video-LLMs' motion understanding.
Instance Accuracy
Evaluates a model's ability to answer all questions correctly in a contrastive benchmark.
Used to measure model performance on MotionBlind.
Gemini3.1 Pro
The only model to pass the MotionBlind evaluation, outperforming other open models.
Displayed high instance accuracy in experiments.
TimeBlind
A contrastive benchmark focusing on semantic reasoning rather than physical motion.
Complementary to MotionBlind, used to evaluate models' temporal reasoning capabilities.
Open Questions Unanswered questions from this research
- 1 Existing models cannot effectively encode speed and magnitude attributes, requiring exploration of new temporal sequence modeling methods.
- 2 Frame selection strategies fail to address the impact of temporal order on motion understanding, necessitating new strategies.
Applications
Immediate Applications
Video Analysis
MotionBlind can be used to evaluate Video-LLMs' ability to understand physical motion, aiding in the development of more reliable world model perceptual front ends.
Long-term Vision
AI in Dynamic Environments
Improving Video-LLMs' understanding of physical motion attributes will enhance AI performance in dynamic environments.
Abstract
Video large language models (Video-LLMs) are increasingly used as the perceptual front end of world models, a role that assumes they can read motion: how fast something moves, which way it travels, how hard it is pushed. We show they cannot. A Video-LLM can watch two clips of the same person in the same room, name every object in both, and still fail to say which clip moves faster. We introduce MotionBlind, a contrastive benchmark of self-recorded video for physically grounded motion(speed, magnitude, and direction), the variables a world model must predict. Each instance is a pair of near-identical clips that differ only in motion. Each clip carries two complementary yes/no questions, giving four items per instance, and a model earns credit only if all four are correct. We report Instance Accuracy(IAcc), which has a 6.25% chance floor. Single-frame, appearance, and language-only shortcuts all collapse to it. MotionBlind complements the recent TimeBlind benchmark. We run a controlled study of six open and two frontier Video-LLMs, varying whether the video is present, whether frames are shown in the correct temporal order, and how frames are sampled (1 to 24 frames, four selection strategies). Open models sit near the 6.25% floor, and scale does not help. Removing the video drops every model to zero IAcc, and shuffling frames collapses IAcc to chance, so the task genuinely needs video in order. Neither more frames nor smarter frame selection closes the gap, because these change which frames are seen, not whether motion is read. Only Gemini3.1 Pro clears the benchmark overall, and even it fails on speed. A frontend that cannot tell two speeds of the same action apart is not yet a trustworthy source of supervision, reward, or evaluation for a world model.