GT-SVJ: Generative-Transformer-Based Self-Supervised Video Judge For Efficient Video Reward Modeling
GT-SVJ transforms video generative models into energy-based discriminators using contrastive learning, reducing human annotation by 6-65x while improving temporal sensitivity.
Key Findings
Methodology
This work converts a pretrained video generative model (CogVideoX) into an energy-based discriminator via contrastive learning. It employs latent space perturbations—frame shuffling, feature swapping, frame dropping—to generate challenging negative samples that mimic subtle temporal degradations. The energy function Eθ assigns low energy to high-quality videos and high energy to degraded ones, trained with a contrastive loss (Eq.4). The discriminative model is then fine-tuned into a reward model by incorporating human preference data through aspect-wise prediction and preference ranking, enabling precise alignment with human judgments. Experiments on GenAI-Bench and MonteBench demonstrate superior performance, with only 30K human annotations, outperforming VLM-based methods by 6-65x in data efficiency.
Key Results
- GT-SVJ exceeds prior models by 24.63% on GenAI-Bench and 3.68% on MonteBench, ranking second on VideoReward-Bench with only 30K annotations, showing excellent generalization and data efficiency.
- Latent space perturbations produce challenging negatives that help the model learn fine-grained temporal cues, resulting in smooth energy trajectories for real videos and erratic fluctuations for fake ones.
- The approach significantly reduces annotation costs while maintaining high accuracy in preference prediction, demonstrating its practical viability for large-scale video evaluation.
Significance
This approach addresses the core challenge of capturing subtle temporal dynamics in video quality assessment, surpassing the limitations of static frame-based metrics and VLMs. By leveraging the inherent temporal modeling of generative models, it achieves high fidelity and data efficiency, paving the way for scalable, human-aligned video evaluation systems. Such advancements are crucial for applications like content moderation, video editing, and autonomous content curation, where understanding nuanced motion and coherence is vital. The reduction in human annotation requirements further accelerates deployment in real-world scenarios, making automated video quality assessment more accessible and reliable.
Technical Contribution
The main technical innovation lies in reformulating a video generative model as an energy-based discriminator trained via contrastive learning. The latent space perturbations generate challenging negatives that compel the model to learn meaningful spatiotemporal features rather than superficial cues. The subsequent fine-tuning into a reward model with aspect-wise preference prediction and pairwise ranking enhances alignment with human judgments. This framework reduces reliance on large-scale annotations, improves stability, and extends energy-based modeling to the video domain, opening new avenues for efficient, temporally-aware reward modeling.
Novelty
This is the first systematic effort to convert a video generative model into an energy-based discriminative reward model, leveraging latent space perturbations for challenging negatives. Unlike prior work focused on static or frame-based metrics, this approach emphasizes temporal coherence and subtle motion cues, significantly reducing annotation costs while maintaining high accuracy. It innovatively combines generative modeling, contrastive learning, and preference optimization, setting a new standard for data-efficient, temporally sensitive video evaluation.
Limitations
- The approach depends heavily on the quality of the pretrained generative model; poor generation quality limits the discriminative power.
- Latent space perturbations, while effective, may not capture all real-world degradations, especially in complex scenarios.
- Training still requires substantial computational resources, and further efficiency improvements are needed for large-scale deployment.
Future Work
Future directions include integrating multimodal cues (audio, text) for richer preference modeling, developing adaptive perturbation strategies for more realistic negatives, and extending the framework to other video understanding tasks like summarization and content moderation. Additionally, exploring more efficient training algorithms and model architectures will facilitate broader adoption in industry.
AI Executive Summary
The rapid growth of video content demands automated, accurate evaluation methods that can understand subtle temporal nuances. Traditional metrics like PSNR and SSIM fall short in capturing motion coherence and perceptual quality, while existing learned models such as VLMs often treat videos as collections of static frames, missing critical temporal cues. Recent advances in generative modeling have produced highly realistic videos, but evaluating their quality remains challenging.
This paper introduces GT-SVJ, a novel framework that leverages pretrained video generative models—like CogVideoX—as the backbone for reward modeling. The key idea is to reformulate these models as energy-based discriminators trained via contrastive learning. By applying controlled latent space perturbations—such as frame shuffling, feature swapping, and frame dropping—the model learns to distinguish subtle degradations that mimic real-world imperfections. These challenging negatives force the model to focus on meaningful spatiotemporal features rather than superficial cues.
Once trained, the discriminative model is fine-tuned into a reward model by incorporating human preference data through aspect-wise prediction and pairwise ranking. This approach significantly reduces the need for extensive human annotations, requiring only 30K samples—6 to 65 times fewer than VLM-based methods—yet achieving state-of-the-art performance on benchmarks like GenAI-Bench and MonteBench. The results demonstrate that the model effectively captures temporal coherence and motion dynamics, outperforming existing approaches in preference alignment.
The impact of this work is substantial: it offers a data-efficient, temporally sensitive evaluation framework that bridges the gap between generative modeling and human perception. Its ability to learn nuanced video preferences with minimal supervision opens new avenues for scalable content moderation, video editing, and autonomous content assessment. Despite its strengths, the approach depends on the quality of the pretrained generator and still requires considerable computational resources. Future research will focus on multimodal integration, adaptive perturbations, and broader application to complex video understanding tasks, aiming to make automated video evaluation more robust, accurate, and accessible.
Deep Dive
Methodology
How is the approach implemented?
Key Results
What do the experiments show?
Abstract
Aligning video generative models with human preferences remains challenging: current approaches rely on Vision-Language Models (VLMs) for reward modeling, but these models struggle to capture subtle temporal dynamics. We propose a fundamentally different approach: repurposing video generative models, which are inherently designed to model temporal structure, as reward models. We present the Generative-Transformer-based Self-Supervised Video Judge (\modelname), a novel evaluation model that transforms state-of-the-art video generation models into powerful temporally-aware reward models. Our key insight is that generative models can be reformulated as energy-based models (EBMs) that assign low energy to high-quality videos and high energy to degraded ones, enabling them to discriminate video quality with remarkable precision when trained via contrastive objectives. To prevent the model from exploiting superficial differences between real and generated videos, we design challenging synthetic negative videos through controlled latent-space perturbations: temporal slicing, feature swapping, and frame shuffling, which simulate realistic but subtle visual degradations. This forces the model to learn meaningful spatiotemporal features rather than trivial artifacts. \modelname achieves state-of-the-art performance on GenAI-Bench and MonteBench using only 30K human-annotations: $6\times$ to $65\times$ fewer than existing VLM-based approaches.