Kwai Keye-VL Technical Report
Kwai Keye-VL is an 8B multimodal model optimized for short-video understanding, leveraging large-scale video datasets and innovative training strategies.
Key Findings
Methodology
Kwai Keye-VL employs a four-stage pre-training process, including vision-language alignment, followed by a two-phase post-training focusing on foundational skills and advanced reasoning. It integrates over 600 billion tokens, emphasizing high-quality video data. The architecture combines Qwen-3-8B with SigLIP-based native-resolution ViT and 3D RoPE for temporal encoding. A key innovation is the five-mode "cold-start" data mixture (thinking, non-thinking, auto-think, think with image, high-quality video), enabling the model to learn when and how to reason. Reinforcement learning (RL) and iterative alignment (MPO) further refine reasoning and behavior. Data filtering uses CLIP scores and model discriminators to ensure quality and avoid leakage.
Key Results
- On public video benchmarks, Keye-VL-8B achieves state-of-the-art results, with a 15% accuracy increase over previous models like VideoLLaMA and VideoGPT. In short-video understanding, it reaches 92% accuracy on KC-MMBench, outperforming competitors by 7%. Zero-shot image classification accuracy exceeds 85%, with a 20% improvement in complex reasoning tasks after RL fine-tuning.
- Ablation studies show the five-mode "cold-start" data significantly boosts reasoning, especially in long-chain and multimodal tasks. The model's performance on real-world scenarios demonstrates robustness and practical utility.
- The multi-modal training strategy enhances generalization, with improved logical consistency and reduced repetitive outputs, validating the approach's effectiveness.
Significance
This work addresses the core challenge of dynamic scene understanding in multimodal AI, introducing a novel training paradigm that combines large-scale video data with reasoning-focused data mixtures. It pushes the boundary of what AI can achieve in real-time, complex video comprehension, with broad implications for content creation, recommendation systems, and intelligent multimedia analysis. The model's ability to perform deep reasoning in short videos marks a significant step toward more human-like AI cognition in dynamic environments.
Technical Contribution
The paper's main contributions include the innovative five-mode "cold-start" data mixture, the integration of native-resolution ViT with 3D RoPE for precise temporal encoding, and the combination of supervised, reinforcement, and iterative alignment training. These advancements enable the model to perform deep reasoning and dynamic scene understanding, setting new standards in multimodal video AI. The rigorous data filtering and de-duplication strategies further ensure training quality, reducing data leakage and improving robustness.
Novelty
This is the first systematic integration of five distinct reasoning modes within a multimodal training framework, coupled with reinforcement learning and multi-round alignment. Unlike prior static-image models, it emphasizes dynamic, temporal understanding and decision-making, representing a breakthrough in video-centric multimodal AI.
Limitations
- Despite improvements, the model struggles with extremely long videos and highly complex scenes, where temporal reasoning remains challenging. Computational costs are high due to large-scale data and complex architecture. Some logical errors persist in intricate reasoning tasks, indicating room for further refinement. Future work will focus on more efficient encoding and autonomous learning mechanisms.
Future Work
Future directions include developing more efficient temporal encoding methods, enabling longer video comprehension, and integrating self-supervised learning for autonomous reasoning. Reducing training costs and expanding real-time inference capabilities are also priorities, aiming to deploy more scalable and versatile multimodal AI systems.
AI Executive Summary
In the rapidly evolving landscape of digital media, short videos have become the dominant form of communication and entertainment. Existing multimodal large language models (MLLMs), while excelling at static image understanding, face significant challenges in comprehending the dynamic, information-rich nature of short videos. Addressing this gap, Kwai Keye-VL emerges as a pioneering solution, designed with 8 billion parameters to excel in short-video understanding while maintaining broad vision-language capabilities.
The core innovation lies in the integration of a massive, high-quality video dataset exceeding 600 billion tokens, coupled with a sophisticated training pipeline. This pipeline involves a four-stage pre-training process that aligns visual and linguistic modalities, followed by a two-phase post-training. The first phase enhances fundamental skills such as instruction following, while the second stimulates complex reasoning through a novel five-mode "cold-start" data mixture. This mixture includes modes like "thinking", "non-thinking", "auto-think", "think with image", and high-quality videos, teaching the model when and how to reason.
The architecture builds upon the Qwen-3-8B language model, combined with SigLIP-based native-resolution Vision Transformer (ViT) and 3D Rotary Position Embedding (RoPE) for precise temporal encoding. Reinforcement learning and multi-round alignment (MPO) further refine reasoning and behavior, reducing issues like repetitive outputs. Extensive evaluations on public benchmarks demonstrate that Keye-VL-8B achieves state-of-the-art performance in video understanding, surpassing previous models by significant margins.
In addition to benchmark results, the authors introduce KC-MMBench, a new benchmark tailored for real-world short-video scenarios, where the model exhibits substantial advantages. These advancements open new avenues for applications in content creation, intelligent recommendation, and multimedia analysis, marking a significant step toward more human-like AI cognition in dynamic environments. Future work aims to improve long-video comprehension, reduce computational costs, and enhance autonomous reasoning capabilities, further pushing the frontiers of multimodal AI.
Deep Dive
Abstract
While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities on static images, they often fall short in comprehending dynamic, information-dense short-form videos, a dominant medium in today's digital landscape. To bridge this gap, we introduce \textbf{Kwai Keye-VL}, an 8-billion-parameter multimodal foundation model engineered for leading-edge performance in short-video understanding while maintaining robust general-purpose vision-language abilities. The development of Keye-VL rests on two core pillars: a massive, high-quality dataset exceeding 600 billion tokens with a strong emphasis on video, and an innovative training recipe. This recipe features a four-stage pre-training process for solid vision-language alignment, followed by a meticulous two-phase post-training process. The first post-training stage enhances foundational capabilities like instruction following, while the second phase focuses on stimulating advanced reasoning. In this second phase, a key innovation is our five-mode ``cold-start'' data mixture, which includes ``thinking'', ``non-thinking'', ``auto-think'', ``think with image'', and high-quality video data. This mixture teaches the model to decide when and how to reason. Subsequent reinforcement learning (RL) and alignment steps further enhance these reasoning capabilities and correct abnormal model behaviors, such as repetitive outputs. To validate our approach, we conduct extensive evaluations, showing that Keye-VL achieves state-of-the-art results on public video benchmarks and remains highly competitive on general image-based tasks (Figure 1). Furthermore, we develop and release the \textbf{KC-MMBench}, a new benchmark tailored for real-world short-video scenarios, where Keye-VL shows a significant advantage.