EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation
EgoVid-5M provides 5M curated clips, while EgoDreamer jointly controls egocentric generation with text and kinematics.
Key Findings
Methodology
The paper builds EgoVid-5M from Ego4D, estimating camera translation and rotation with ParticleSfM, IMU filtering, least-squares optimization, and a Kalman Filter. LLaVA-NeXT-Video-32B-Qwen and Qwen2 produce detailed action captions. EgoVideo, CLIP, RAFT, and DOVER score action alignment, frame consistency, motion, and clarity. EgoDreamer jointly injects text and kinematic conditions through a Unified Action Encoder and Adaptive Alignment.
Key Results
- EgoVid-train contains 4.9M clips and EgoVid-val 1.2K clips, with 1080p source videos averaging about 120 frames. EgoVid-65K supplies accurate IMU-based kinematic labels and is approximately five times larger than the previously largest kinematic-control dataset.
- EgoVid fine-tuning reduces CD-FVD for SVD from 591.61 to 548.32, DynamiCrafter from 243.63 to 236.82, and OpenSora from 809.46 to 718.32. Semantic consistency, action consistency, clarity, smoothness, and motion strength improve across all three baselines.
- DynamiCrafter’s action consistency rises from 0.481 to 0.494 and motion strength from 9.357 to 18.329. Cleaning ablations show that overly strict filtering removes local hand motion, whereas weak filtering produces excessive motion; the balanced strategy preserves both.
Significance
EgoVid-5M addresses a major data bottleneck for egocentric video generation. Ego4D and Ego-Exo4D primarily target perception and contain noisy motion or simplistic narrations. By combining video, action language, camera motion, and cleaning metadata, EgoVid-5M offers a reusable foundation for VR, AR, gaming, and embodied-agent simulation.
Technical Contribution
The work turns VIO into generation-ready control signals: ParticleSfM provides scale-ambiguous poses, IMU integration contributes metric dynamics, Butterworth filters suppress gravity and high-frequency noise, least squares estimates velocity, extrinsics, and scale, and Kalman fusion produces trajectories. EgoDreamer uses Plücker embedding for geometry, CLIP and cross-attention for text, and multi-scale Adaptive Alignment rather than simple ControlNet zero initialization.
Novelty
The authors present EgoVid-5M as the first publicly released, high-quality dataset explicitly designed for egocentric video generation. Its fundamental novelty is not merely scale, but the joint release of low-level kinematics, high-level action descriptions, and per-video cleansing signals. EgoDreamer further models the coupling between these conditions, unlike methods that separately encode text or mainly control smooth camera motion.
Limitations
- Reliable IMU data are unavailable for the full corpus; accurate kinematic annotations cover only 65K clips. Large-scale training may therefore rely on estimated trajectories whose quality can degrade in weak-texture, occluded, or rapidly rotating scenes.
- Because the corpus is derived from Ego4D, device, participant, and scenario biases remain. Automatically generated captions may contain MLLM errors, while cross-dataset generalization and demographic or hardware robustness are not extensively evaluated.
- Experiments emphasize short-video fine-tuning and offline metrics. Long-horizon physical consistency, closed-loop interaction, and deployment in real VR or robotic systems remain unverified.
Future Work
Future research should expand synchronized IMU and multi-view annotations, improve caption verification, and model long-horizon trajectories, object states, and action planning. The released cleansing metadata enables systematic studies of thresholds, data mixtures, and transfer across datasets, while closed-loop evaluation can test whether metric gains translate into useful simulation.
AI Executive Summary
Video generation is increasingly viewed as a world simulator, yet the first-person setting exposes a severe data gap. The camera moves with the wearer, hands create fine-grained local motion, and scenes range from kitchens to sports fields. General datasets such as WebVid-10M and Panda-70M lack reliable egomotion labels, while Ego4D and Ego-Exo4D were built mainly for perception and contain noisy motion and weak narrations.
Wang and colleagues introduce EgoVid-5M, a curated collection of 5 million egocentric clips with detailed action descriptions and kinematic controls. Their pipeline combines ParticleSfM with IMU signals, Butterworth filtering, least-squares calibration, and Kalman fusion. LLaVA-NeXT-Video-32B-Qwen and Qwen2 generate captions, while EgoVideo, CLIP, RAFT, and DOVER assess action alignment, semantic consistency, optical flow, and clarity. The accompanying EgoDreamer uses a Unified Action Encoder and Adaptive Alignment to combine language with camera motion.
The dataset improves three representative generators. CD-FVD falls from 591.61 to 548.32 for SVD, 243.63 to 236.82 for DynamiCrafter, and 809.46 to 718.32 for OpenSora. DynamiCrafter’s action consistency increases from 0.481 to 0.494, and motion strength from 9.357 to 18.329. The contribution is substantial, but accurate IMU labels cover only 65K clips, and long-term physical consistency and closed-loop interaction remain open challenges.
Deep Analysis
Background
Diffusion models and DiT architectures have rapidly advanced video synthesis; Sora and related systems suggest that generators may serve as world simulators. General datasets such as WebVid-10M and Panda-70M support text-to-video learning, while Ego4D and Ego-Exo4D support egocentric understanding. However, perception datasets are not necessarily generative datasets: they often contain unstable motion, weak temporal coherence, and captions that do not precisely describe visible actions.
Core Problem
Egocentric generation must specify both what happens and how the viewpoint moves. Noisy IMU signals, unknown camera–IMU extrinsics, ambiguous scale, unknown initial velocity, rapid rotations, and occlusion complicate motion recovery. Simple narrations fail to align with frames. Aggressive cleaning removes useful hand motion; insufficient cleaning preserves shake, blur, and semantic discontinuity. The model therefore needs a large corpus with aligned language, geometry, and quality control.
Innovation
- ��EgoVid-5M provides 5M clips and releases annotation and cleansing metadata.
- ��VIO combines ParticleSfM and IMU to estimate translation and rotation.
- ��LLaVA-NeXT-Video-32B-Qwen plus Qwen2 generate structured action descriptions.
- ��EgoVideo, CLIP, RAFT, and DOVER form an egocentric-specific filtering pipeline.
- ��EgoDreamer introduces a Unified Action Encoder for coupled text–kinematic representation and Adaptive Alignment for multi-scale diffusion control.
Methodology
- ��Motion recovery: ParticleSfM estimates scale-ambiguous camera poses; IMU signals are processed with low- and high-pass Butterworth filters to suppress gravity and high-frequency noise.
- ��Quality filtering: clips satisfy reconstructed-point and IMU-variance thresholds, removing unreliable trajectories.
- ��Calibration: least squares minimizes ||TI PI(T−1)−λPc||² to estimate initial velocity, extrinsics, and scale; a Kalman Filter fuses signals.
- ��Captioning: LLaVA describes foreground, background, subjects, and actions; Qwen2 summarizes egocentric action text and extracts noun/verb categories.
- ��Generation: Plücker embedding represents camera rays; CLIP and cross-attention encode language; Adaptive Alignment injects multi-scale action features using Li=αLi+(Ai−μL)/σL.
Experiments
The corpus is split into EgoVid-train with 4.9M clips and EgoVid-val with 1.2K curated samples. EgoVid-65K contains reliable IMU annotations for kinematic training. Baselines include U-Net-based SVD and DynamiCrafter and DiT-based OpenSora, initialized from pretrained weights and continuously trained at 480p. Evaluation uses CD-FVD, CLIP semantic consistency, EgoVideo action consistency, DOVER clarity, frame-interpolation smoothness, and RAFT motion strength; cleaning strategies are also compared.
Results
EgoVid improves every baseline. SVD’s CD-FVD decreases from 591.61 to 548.32 and semantic consistency rises from 0.258 to 0.266. DynamiCrafter improves from 243.63 to 236.82 CD-FVD and from 0.481 to 0.494 action consistency. OpenSora improves from 809.46 to 718.32. DynamiCrafter’s motion strength nearly doubles from 9.357 to 18.329. Ablations show that balanced cleaning best preserves fine hand motion without sacrificing coherence.
Applications
The dataset can support first-person VR and AR prototyping, game-level and interaction-video generation, and visual imagination for embodied agents. Developers can condition generation on language and camera trajectories to preview actions before implementation. Practical use requires reliable input controls, object-state reasoning, collision and physics validation, and human review for safety-critical or immersive applications.
Limitations & Outlook
The corpus inherits Ego4D’s hardware, participant, and scenario biases, and automatically generated captions may contain hallucinated details. Accurate IMU labels cover only 65K clips, while VIO can fail under blur, occlusion, weak texture, or rapid rotation. Cleaning and training millions of clips are computationally expensive. Current evaluation focuses on short, offline videos; long-horizon dynamics, causal interaction, real-time inference, and cross-dataset transfer need stronger evidence.
Plain Language Accessible to non-experts
Imagine training a first-person film director. Ordinary internet videos are like scripts that say only “someone washes a car”; they rarely tell the director exactly how the camera moved, whether a hand entered the frame, or whether two neighboring frames belong together. EgoVid-5M is a much stricter film school: it collects 5 million wearable-camera clips, attaches a readable action description, records the viewpoint’s movement, and removes footage that is blurry, unrelated, violently shaky, or visually inconsistent.
The researchers first act like instrument technicians. They combine camera images with motion-sensor readings, filtering out unwanted vibration and correcting missing information about scale and alignment. Then an AI observer describes the scene, the people, the objects, and the action. Several inspectors check whether the words match the pictures, whether frames remain similar enough, whether movement is meaningful, and whether the video is clear.
EgoDreamer is the trained director. The words say what should happen; the movement track says how the viewpoint should move. Because both instructions are learned together, the generated result can include not only a camera turn but also detailed hand interaction. The system is still imperfect, especially during fast turns or occlusions, but the dataset makes this difficult training problem measurable and reproducible.
ELI14 Explained like you're 14
Suppose you are building a game where the player says, “Open the bathroom door,” and the computer creates a short movie from the player’s eyes. That is much harder than making a still picture. Your head may turn, your hand may reach forward, the door may move, and the background may wobble. A normal internet video may show the action, but it usually does not explain exactly how the camera moved.
EgoVid-5M is like a giant homework book with 5 million first-person video clips. Each example tries to provide two clues: a sentence describing the action and a movement trail describing how the viewpoint traveled. The researchers also act like strict teachers, throwing out clips where the sentence does not match the video, frames jump strangely, the camera shakes too much, or the picture is unclear.
EgoDreamer is like a game director who listens to both the script and the camera instructions. “Wash the red car” tells it what should happen; the movement signal tells it how the view should shift. Learning both together helps it show small hand movements instead of producing a frozen scene or a chaotic blur.
The results are promising! After training with EgoVid, SVD, DynamiCrafter, and OpenSora all score better. DynamiCrafter’s action score rises from 0.481 to 0.494, while its motion-strength score increases from 9.357 to 18.329. It still makes mistakes with fast turns, hidden objects, and long actions. But this is an important step toward smarter VR, games, and robots that can imagine what will happen before acting.
Glossary
EgoVid-5M
A 5-million-clip dataset designed specifically for egocentric video generation. It includes action text, kinematic signals, and cleansing metadata.
It is the paper’s main training and evaluation resource.
Visual-Inertial Odometry (VIO)
A method for estimating camera motion by combining visual observations with inertial measurements. It helps recover motion that vision alone cannot scale or stabilize reliably.
It produces translation and rotation controls for the dataset.
Unified Action Encoder (UAE)
A module that encodes language-level actions and low-level geometric motion together. Its attention mechanisms model the relationship between semantic intent and egomotion.
It is a central component of EgoDreamer.
Adaptive Alignment (AA)
A multi-scale feature-injection mechanism that normalizes and aligns action features with diffusion features. It uses a learnable scaling parameter rather than simple zero-initialized control.
It strengthens action control in the generation branch.
CD-FVD
A distributional metric for spatial and temporal video quality; lower values generally indicate closer correspondence to real videos. It evaluates generated-video quality beyond individual frames.
It is reported for all three baseline generators.
Open Questions Unanswered questions from this research
- 1 How can reliable synchronized IMU and object-state labels be acquired for millions of clips at low cost? Current VIO remains vulnerable to occlusion, weak texture, blur, and rapid rotation.
- 2 Do short-video metric gains translate into long-horizon physical consistency and useful closed-loop behavior? This requires cross-dataset, robotic, and interactive VR evaluation rather than offline scores alone.
Applications
Immediate Applications
VR/AR prototyping
VR and AR developers can generate first-person previews from action captions and viewpoint trajectories before building a full interactive scene. The approach may reduce prototyping time, but outputs still require checks for geometry, object motion, latency, and user comfort.
Game action and level design
Game studios can turn instructions such as “pick up a tool” or “open a drawer” into first-person video references for animation, camera design, and immersive transitions. Human review remains necessary for collisions, physics, and character continuity.
Long-term Vision
Embodied world simulation
An agent could predict the visual consequences of candidate actions before executing them, using EgoDreamer as a learned simulator. Major obstacles include long-term memory, object permanence, accurate dynamics, real-time inference, and safe closed-loop control.
Abstract
Video generation has emerged as a promising tool for world simulation, leveraging visual data to replicate real-world environments. Within this context, egocentric video generation, which centers on the human perspective, holds significant potential for enhancing applications in virtual reality, augmented reality, and gaming. However, the generation of egocentric videos presents substantial challenges due to the dynamic nature of egocentric viewpoints, the intricate diversity of actions, and the complex variety of scenes encountered. Existing datasets are inadequate for addressing these challenges effectively. To bridge this gap, we present EgoVid-5M, the first high-quality dataset specifically curated for egocentric video generation. EgoVid-5M encompasses 5 million egocentric video clips and is enriched with detailed action annotations, including fine-grained kinematic control and high-level textual descriptions. To ensure the integrity and usability of the dataset, we implement a sophisticated data cleaning pipeline designed to maintain frame consistency, action coherence, and motion smoothness under egocentric conditions. Furthermore, we introduce EgoDreamer, which is capable of generating egocentric videos driven simultaneously by action descriptions and kinematic control signals. The EgoVid-5M dataset, associated action annotations, and all data cleansing metadata will be released for the advancement of research in egocentric video generation.