Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval
Self-SiMS uses video self-similarity for zero-shot proposals, reaching 42.2 [email protected] on QVHighlights test.
Key Findings
Methodology
Self-SiMS builds temporal self-similarity matrices from frame features and MLLM-generated caption features, detects boundaries with a contrastive kernel, and proposes candidate spans. It combines top-k query-caption matching with intra-span consistency, then applies query-aware Yes/No reranking using LLaMA-3.2-11B-Vision-Instruct.
Key Results
- On QVHighlights test, Self-SiMS obtains [email protected]=59.7, [email protected]=42.2, [email protected]=59.2, and average mAP=38.3, surpassing Moment-GPT's 58.3, 37.7, 55.1, and 35.0.
- On QVHighlights validation, it reaches 61.0, 43.2, 60.5, and 39.3 for the same metrics. The consistently stronger Inner-to-Outer Ratio indicates more discriminative temporal signals than query-frame or query-caption similarity.
- On Charades-STA, Self-SiMS achieves mIoU=41.9 versus Moment-GPT's 36.5. Its [email protected] is 21.0, slightly below Moment-GPT's 21.6, showing that gains vary by metric and localization strictness.
Significance
The paper reframes zero-shot retrieval as a representation-alignment problem with two distinct failures: visual-language modality mismatch and stylistic mismatch between human queries and generated captions. By making proposal generation query-independent, Self-SiMS reduces a major source of instability without temporal supervision. This is relevant to academia because it isolates a previously underemphasized bottleneck, and to industry because it offers a practical route to indexing large video collections without expensive span annotation.
Technical Contribution
The system integrates dual temporal self-similarity matrices, contrastive-kernel boundary detection, variability-aware thresholds, top-k query-caption scoring, self-matching span scoring, and MLLM reranking. It defines M=1/2(M^f+M^c), S_n=(1-α)S^Q_n+αS^S_n, and S*= (1-β)S+βS^R. The default settings are α=0.1, β=0.5, kS=kR=3, and kC=5.
Novelty
The authors present this as the first training-free ZMR method to use temporal self-similarity directly for both candidate generation and span scoring. Unlike CLIP-style query-frame matching or Moment-GPT-style query-caption matching, it separates structural segmentation from semantic selection, using query-independent video relations to produce more stable proposals.
Limitations
- Self-similarity assumes that an event is internally coherent. It can fail when one semantic event crosses a hard scene cut, when rapid editing creates false boundaries, or when visually similar but semantically different events occur repeatedly.
- The pipeline still depends on MLLM captioning, feature extraction, and final visual question answering. LLaMA-3.2-11B-Vision-Instruct introduces latency, memory cost, and model-specific biases, which may hinder real-time deployment.
- The supplied main text does not report a complete ablation table, so the independent marginal contribution of each module is not fully established.
Future Work
Future work should develop lighter MLLM rerankers, adaptive fusion between frame and caption self-similarity, and hierarchical processing for long videos. Important extensions include multilingual and cross-domain queries, weakly supervised calibration of dynamic thresholds, explicit modeling of long-range event continuity, systematic error analysis, and testing transfer across different VLMs and MLLMs.
AI Executive Summary
Finding one relevant moment in an untrimmed video is harder than recognizing its general topic. Conventional video moment retrieval relies on costly temporal annotations. Zero-shot systems using CLIP, BLIP, or multimodal large language models avoid training labels, but commonly compare a query directly with frames or generated captions. Query-frame scores can remain low and flat because text and vision occupy different representational spaces. Query-caption scores reduce that problem but become noisy when generated captions describe different aspects of the same scene.
The paper proposes Self-SiMS, or Self-Similarity-based Moment Proposal and Scoring. It constructs frame and caption temporal self-similarity matrices, M^f and M^c, and averages them as M=1/2(M^f+M^c). A contrastive kernel rewards similarity within the two sides of a boundary and penalizes similarity across it. Dynamic thresholds then produce non-overlapping candidate spans. Each span combines the mean of its top-three query-caption scores with a self-matching score measuring internal consistency. The top five candidates are finally reranked by LLaMA-3.2-11B-Vision-Instruct through query-aware Yes/No judgments.
On QVHighlights test, Self-SiMS reaches [email protected]=42.2 and [email protected]=59.2, compared with Moment-GPT's 37.7 and 55.1; average mAP is 38.3 versus 35.0. On Charades-STA, mIoU reaches 41.9 versus 36.5. The broader contribution is architectural: first identify coherent video units, then verify query relevance. However, scene cuts, repeated visual patterns, MLLM cost, and incomplete reported ablations leave important questions for future research.
Deep Analysis
Background
VMR evolved through annotated datasets such as ActivityNet and QVHighlights, with supervised systems including Moment-DETR and UMT. These models achieve strong localization but require expensive temporal labels. Zero-shot approaches based on CLIP, BLIP-2, Moment-GPT, and instruction-tuned MLLMs remove that requirement. Their remaining weakness is precise temporal proposal generation under unreliable cross-modal similarity.
Core Problem
Given an untrimmed video V={v_i} and query Q, the system must predict the relevant temporal interval without training on span annotations. Direct query-frame matching suffers from modality gaps and weak, flat signals. Query-caption matching is affected by language-style gaps: captions may emphasize objects, actions, or sub-events differently from the human query. Thus both boundary discovery and candidate ranking become unstable.
Innovation
Self-SiMS introduces four connected innovations. First, it generates proposals from query-independent intra-video structure. Second, it fuses frame and caption self-similarity to capture complementary visual and semantic continuity. Third, it replaces unreliable span-wide averaging with top-k query-caption scores and adds Self-Matching Span Score to penalize internally inconsistent spans. Fourth, it uses direct MLLM query-frame verification for final reranking, rather than relying only on independently computed embeddings.
Methodology
- �� Features: L2-normalize frame features F^f and caption features F^c; compute M^f=Fhat^f(Fhat^f)^T and M^c=Fhat^c(Fhat^c)^T; fuse them into M.
- �� Boundaries: extract a local patch P_i around each diagonal position and compute b_i=P_i⊙K, where K has positive within-side and negative cross-side quadrants.
- �� Proposals: use context-dependent thresholds, retain local maxima, add frames 1 and L, and form successive non-overlapping spans E_n.
- �� Initial scoring: compute cosine query-caption scores; QMS is the mean of the top three scores within a span.
- �� Structural scoring: select the highest-scoring key frame κ_n and average M_{κ_n,j} over the span to obtain SMS; combine scores with S=(1-α)S^Q+αS^S.
- �� Reranking: sample 10–30 representative frames from the top five spans, obtain MLLM Yes probabilities, average the top three, and combine using β=0.5.
Experiments
The evaluation covers QVHighlights, Charades-STA, ActivityNet-Captions, and TVR. Frame rates are 0.5, 1, 1, and 1 fps respectively. LLaMA-3.2-11B-Vision-Instruct is used for captioning and reranking. QVHighlights uses [email protected], [email protected], [email protected], and average mAP; the other datasets use [email protected]/0.5/0.7 and mIoU. Baselines include Moment-DETR, TFVTG, and Moment-GPT. The reported setup uses kS=kR=3, kC=5, α=0.1, and β=0.5.
Results
On QVHighlights test, Self-SiMS scores 59.7/42.2/59.2/38.3 for [email protected], [email protected], [email protected], and average mAP, exceeding Moment-GPT's 58.3/37.7/55.1/35.0. Validation results are 61.0/43.2/60.5/39.3. On Charades-STA, results are 62.7, 39.7, 21.0, and 41.9; on ActivityNet-Captions, 49.9, 28.2, 13.8, and 34.7. The gains across datasets support improved generalization, although strict [email protected] is not uniformly higher.
Applications
The method can support natural-language search in news archives, sports footage, online courses, corporate training, and user-generated video. It is especially attractive for offline indexing because it does not require temporal-span training labels. A deployment needs frame sampling, a vision-language encoder, caption generation, and MLLM verification. For real-time use, the top-candidate limit and reranking model would need aggressive compression or caching.
Limitations & Outlook
The approach assumes meaningful temporal coherence, so rapid edits, cross-shot events, and repeated visual motifs may yield incorrect boundaries. Two self-similarity matrices plus captioning and MLLM verification increase memory, latency, and energy use. Dynamic thresholds reduce manual tuning but depend on video variability estimates. The provided main text does not include a full ablation table, limiting causal interpretation of module contributions. Future work should address efficiency, multilingual retrieval, long-range continuity, and stronger robustness evaluation.
Plain Language Accessible to non-experts
Imagine an archivist organizing a huge library of video tapes. A conventional archivist reads the visitor's request and compares it with every picture. That can fail because a picture and a sentence speak different languages. If the archivist asks another AI to describe every picture, the descriptions may jump between “someone holding a hammer” and “someone walking toward a shelf,” even though the same event is happening.
Self-SiMS first studies the tape itself, without the request. It notices that neighboring pictures usually belong to the same chapter: their people, objects, and actions change slowly. When the scene suddenly changes, the pictures before and after the change no longer resemble each other. Those locations become possible chapter boundaries.
Only after making these chapters does the archivist read the request. The system selects the few pictures that look most relevant, checks whether the whole chapter is coherent, and then asks a powerful visual assistant, “Does this picture answer the request?” In QVHighlights, this organization produced [email protected]=42.2, above Moment-GPT's 37.7. It can still split one event across a scene change and requires substantial computing power.
ELI14 Explained like you're 14
Suppose you want to find “the first time the streamer defeats the boss” inside a three-hour gaming video. Comparing that sentence with every screenshot sounds reasonable, but computers may struggle because words and pictures are not the same kind of information. Asking an AI to write a caption for every screenshot helps, yet the AI might describe one frame as “swinging a sword” and another as “standing near a door.” The score can become jumpy even when the same battle continues.
Self-SiMS takes a smarter route. It first watches how the video changes by itself. Screenshots from one continuous battle tend to look related; entering a new room or finishing the fight creates a noticeable change. The method marks those changes and cuts the long video into likely event chapters before using the question.
Then it checks each chapter: which few frames best match the request, and do the other frames still belong to the same event? A multimodal AI finally answers a simple question: “Is this frame relevant?” The system combines all these clues and chooses the best chapter.
The results are impressive: on QVHighlights, the strict [email protected] score is 42.2, compared with Moment-GPT's 37.7. On Charades-STA, mIoU is 41.9 versus 36.5. But scene changes can fool it, and repeatedly asking a large AI for judgments can be slow and expensive!
Glossary
Zero-Shot Video Moment Retrieval
Locating a time interval in a video from a natural-language query without training on temporal span annotations. It uses pretrained vision-language models instead.
This is the paper's target task.
Temporal Self-Similarity Matrix
A matrix whose entries measure how similar video moments are to one another. Its local patterns reveal continuity and transitions inside the video.
Used for boundary detection and span consistency.
Modality Gap
A mismatch between visual and linguistic representation spaces that makes direct similarity unreliable. It can produce weak or flat query-frame scores.
A central failure mode identified by the paper.
Language-Style Gap
A mismatch between human-written queries and automatically generated captions in wording, focus, or granularity. Semantically valid captions may therefore receive unstable scores.
Motivates top-k scoring and MLLM reranking.
Inner-to-Outer Ratio
A diagnostic ratio comparing similarity inside the ground-truth moment with similarity in surrounding regions. Values above 1 indicate stronger discrimination.
Used to compare query-based and self-similarity signals.
Query-Aware MLLM Reranking
Directly asking a multimodal language model whether sampled frames match a query. This differs from comparing independently generated embeddings.
Applied to the top five initial candidates.
Open Questions Unanswered questions from this research
- 1 How can self-similarity preserve one event across hard camera or scene cuts? Explicit long-range modeling of actors, objects, and actions may be required.
- 2 The main text lacks a complete ablation table, so the separate effects of matrix fusion, dynamic thresholds, SMS, and reranking remain uncertain.
- 3 Transfer across MLLMs, languages, and extremely long videos is not established, especially for low-resource languages and multi-event queries.
Applications
Immediate Applications
Natural-language media archives
News organizations can sample footage, generate captions, and use Self-SiMS to index segments such as “the interview with the minister.” No new temporal training labels are required, making offline archive search a practical first deployment.
Course and corporate-video search
Education and training platforms can retrieve moments such as laboratory demonstrations or safety explanations. Required components are frame sampling, a vision-language encoder, caption generation, and a limited MLLM reranking budget.
Long-term Vision
A general interface for video knowledge
With efficient models, multilingual queries, and hierarchical long-video indexing, people could search meetings, personal recordings, and public footage as easily as webpages. Privacy, compute cost, scene continuity, and reliable evaluation remain major obstacles.
Abstract
Zero-shot video moment retrieval aims to overcome the limitations of traditional approaches that require large-scale datasets annotated with text and its relevant temporal spans. Despite advances in pre-trained vision-language models and multimodal large language models, existing ZMR methods still heavily depend on query-to-video content similarity, making them vulnerable to modality and language-style gaps. These gaps lead to unreliable span proposals and unstable moment retrieval results. To address this issue, we propose Self-Similarity-based Moment Proposal and Scoring that instead exploits intrinsic relationships within videos, enabling robust span generation and scoring. By deriving self-similarity only from the video content, we circumvent the noisy and mismatched patterns of query-frame or query-caption similarities, thereby mitigating both modality and language-style gaps. Furthermore, we introduce a query-aware MLLM-based reasoning stage to further sharpen alignment between text and video. Extensive experiments demonstrate that Self-SiMS achieves state-of-the-art performance across ZMR benchmarks.