Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs

Video Large Language Models (Video-LLMs) integrate visual encoders with large language models to enable understanding and generation of language grounded in video content. Recent advances, including models like LLaVA-Video and Gemini2.5-Flash, have demonstrated impressive capabilities in temporal reasoning, action recognition, and long-form video comprehension. Benchmarks such as Something-Something V2 (SSv2), KTH action dataset, and TOMATO evaluate these temporal and motion-related abilities. Despite these advances, foundational perceptual primitives like signed image-plane motion direction recognition remain underexplored. Motion direction is critical for visual navigation, physical interaction, and temporal reasoning, serving as a building block for complex video understanding. The gap in evaluating and improving this primitive limits the overall effectiveness and reliability of Video-LLMs in dynamic scene interpretation.

The core problem addressed is the inability of current Video-LLMs to accurately recognize basic signed motion directions—left, right, up, and down—in videos featuring a single moving object. Although seemingly trivial, experiments reveal that most models perform near random chance (~25%) on this task. The bottleneck is not the absence of motion direction information—linear probes confirm its presence in visual encoder outputs, projector outputs, and LLM hidden states—but rather the failure of the final language output token to bind this internal motion signal to the correct prompt-specific answer option. This 'direction binding gap' prevents the model from verbalizing its perceptual understanding correctly, undermining downstream tasks requiring precise motion comprehension.

This work introduces several key innovations: 1) The identification and formalization of the 'direction binding gap,' a novel conceptual framework pinpointing the failure mode in Video-LLMs' motion direction recognition as a language readout binding issue rather than perceptual absence. 2) The creation of the MoDirect dataset family, comprising four controlled subsets combining synthetic and real foregrounds and backgrounds, designed as randomized multiple-choice questions to rigorously test motion direction binding. 3) The use of concept vector analysis to reveal that while motion direction representations share orientation across domains, their magnitude diminishes with increasing visual complexity, explaining out-of-domain generalization failures. 4) The design of DeltaDirect, a training-only auxiliary objective applied at the projector output layer that predicts normalized 2-D motion vectors from adjacent-frame feature deltas, directly reinforcing signed displacement cues before language decoding. 5) A training regime integrating standard next-token prediction with motion vector prediction loss, improving motion direction accuracy without altering inference-time architecture or input formats.

• �� Dataset Construction: MoDirect comprises four subsets—Primitive-on-Syn (geometric shapes on uniform backgrounds), Cutout-on-Syn (real object cutouts on uniform backgrounds), Primitive-on-Real (geometric shapes on natural backgrounds), and Cutout-on-Real (real object cutouts on natural backgrounds). Each video contains a single object moving in one of four directions.

• �� Task Design: Multiple-choice questions with randomized answer option orders require models to bind perceived motion direction to the correct textual answer, preventing fixed label shortcuts.

• �� Diagnostic Analysis: Linear probing classifiers are trained on frozen representations from the vision encoder, projector, and LLM hidden states to assess motion direction information accessibility.

• �� Concept Vector Analysis: Difference-in-means vectors are computed per motion direction and domain to assess orientation alignment and magnitude differences across domains.

• �� DeltaDirect Auxiliary Objective: At the projector output, adjacent-frame feature differences are spatially pooled to form motion-change descriptors. A lightweight prediction head regresses normalized 2-D motion vectors representing signed displacement directions.

• �� Training Procedure: The total loss combines standard next-token cross-entropy with mean squared error for motion vector prediction. Only the projector and prediction head are updated; the vision encoder and LLM weights remain frozen.

• �� Inference: The auxiliary prediction head is removed, preserving original input formats, model architecture, and decoding procedures.

Experiments are conducted on MoDirect synthetic and real subsets, as well as real-world benchmarks including Something-Something V2 (SSv2), TOMATO, and KTH datasets. Baseline models include LLaVA-Video-7B, Gemini2.5-Flash, and others. Evaluation metric is multiple-choice question accuracy on motion direction recognition. Models are fine-tuned with LoRA adapters on the projector and LLM, with the vision encoder frozen. Ablation studies compare supervision applied at different layers (vision encoder, pre/post-projector, LLM hidden states, final readout) and different motion signal formulations (feature concatenation, delta equivariance, normalized motion vectors). Hyperparameters and training details are provided in the appendix. The impact on general video understanding benchmarks is also assessed to verify no degradation.

DeltaDirect instruction tuning improves motion direction accuracy on MoDirect-SynBench from 25.9% to 85.4%, a dramatic gain over baselines. On MoDirect-RealBench, it achieves a 21.9-point improvement without real-world motion labels, demonstrating strong zero-shot transfer. Ablations show that applying supervision at the projector output yields the best performance, while readout layer supervision is ineffective despite the binding gap there. Predicting normalized 2-D motion vectors outperforms other motion signal targets. Fine-tuned models maintain or slightly improve performance on standard and fine-grained video understanding benchmarks, indicating that motion direction supervision complements rather than compromises general video comprehension.

The proposed method directly benefits video question answering systems by enabling accurate motion direction comprehension, improving answer correctness and user experience. It enhances intelligent surveillance systems' ability to detect and interpret object motion, aiding anomaly detection and event prediction. In robotics, improved motion direction perception supports visual navigation and obstacle avoidance, critical for autonomous operation. The approach's inference-time transparency facilitates integration into existing Video-LLMs deployed in autonomous driving, augmented reality, and other dynamic scene understanding applications.

Despite significant improvements, DeltaDirect's accuracy decreases in highly complex real-world scenes due to diminished motion signal magnitude, limiting generalization. The reliance on synthetic motion direction annotations constrains applicability where real-world labeled data is scarce. The current scope is limited to four cardinal motion directions, excluding complex trajectories and 3D motion patterns. Further work is needed to address these limitations and extend the framework's applicability.