TARS: Timestep-Aware Data Scaling for 3D-Free Video Re-Shooting
TARS learns camera motion at high-noise diffusion steps, reaching 83.18% R-Prec, 72.76% T-Prec, and 10.34° V-MPGE.
Key Findings
Methodology
TARS is a 3D-free video-reshooting framework that jointly conditions a video diffusion model on a source video, a Camera Grid trajectory, and textual viewpoint descriptions. Its Timestep-Aware Data Routing assigns global structure, subject motion, and camera changes to high-noise steps, while mid-to-low-noise steps preserve texture and identity. Stage 1 uses one million clips for self-supervised learning; Stage 2 uses limited cross-pair supervision only at t∈[0.95,1.0] to enforce motion synchronization.
Key Results
- On 1,021 online-video samples, TARS achieves 83.18% R-Prec and 72.76% T-Prec, versus 71.77% and 51.48% for CamClone. The corresponding relative gains are 15.90% and 41.34%. Its V-MPGE is 10.34°, compared with 21.67° for CamClone and 31.38° for TrajCrafter.
- For semantic viewpoint control, TARS scores 0.83, 0.81, and 0.98 on shot scale, viewing angle, and perspective, respectively, outperforming SD-2.0 at 0.69, 0.72, and 0.47. This supports close, side, high-angle, reverse-angle, and first-/third-person transformations.
- TARS obtains 83.92% FSCS, 0.91 CE, 3.48% FDR, and 12.57% VDR. These results indicate stronger content expansion and temporal coherence than TrajCrafter, whose FSCS is 47.33% and VDR is 91.22%, although SD-2.0 has lower distortion.
Significance
TARS addresses two persistent obstacles in video reshooting: geometric errors from explicit 3D reconstruction and the scarcity of real cross-trajectory video pairs. It shows that strict pairing is not necessary at every diffusion step: camera motion and coarse structure can be learned from large-scale self-supervised data, while a small high-noise intervention can recover temporal alignment. This reduces data-collection costs and makes natural-language cinematographic control practical for filmmaking, advertising, autonomous driving, and embodied AI.
Technical Contribution
The framework combines Camera Grid, Rectified Flow, and timestep-aware data routing. Camera Grid converts rotations R_i and translations t_i into dynamic projected grids through λu=K(R_ip+t_i). Rectified Flow uses z_t=(1−t)z_0+tε and conditional flow matching to learn a deterministic vector field. Self-supervised pairs are formed from temporal clips V1 and V2, with Qwen3-VL generating structured viewpoint text. Only the high-noise regime receives cross-pair supervision, efficiently injecting synchronized motion without sacrificing broad visual priors.
Novelty
The authors present TARS as the first framework to unify 3D-free video reshooting with text-driven semantic viewpoint control. Unlike TrajCrafter's explicit 3D rendering and CamClone's dependence on cross-pair training across denoising stages, TARS separates global camera dynamics from fine appearance generation and uses large-scale self-supervision to replace much of the expensive geometry and paired-data requirement.
Limitations
- TARS depends on a strong video-diffusion backbone, one million training clips, and reliable camera-pose or trajectory estimation. Severe occlusion, fast non-rigid motion, and extreme out-of-view synthesis may still produce implausible geometry or identity drift.
- Several viewpoint, content-expansion, and distortion measures are judged by Gemini 3.1 Pro, introducing evaluator bias. The paper also gives limited evidence about inference cost, calibration noise, human agreement, and systematic component ablations.
Future Work
Future research should improve automatic pose estimation, object-level and non-rigid motion modeling, and hierarchical timestep control. Larger real cross-view datasets could be combined with depth, optical flow, or event-camera signals. Stronger human and geometry-based evaluation, efficient inference, long-video modeling, and explicit uncertainty estimates are needed for reliable reverse-angle and heavily occluded reshooting.
AI Executive Summary
Video reshooting asks a generative system to preserve a scene, actor, and action while changing the camera trajectory. Existing 3D approaches such as TrajCrafter inherit reconstruction and scale errors, especially under dynamics and occlusion. Cross-pair methods such as CamClone require scarce videos of the same scene filmed along different trajectories. Both struggle to hallucinate unseen regions and to interpret cinematic instructions such as “medium side view” or “first-person perspective.”
TARS, introduced by the Kling team, removes the explicit 3D requirement. It represents the target trajectory as a Camera Grid and conditions a Rectified Flow video model on the source video, grid, and text. The central observation is timestep-dependent: high-noise denoising steps establish low-frequency layout, camera motion, and subject dynamics, whereas later steps mainly refine texture and identity. TARS therefore trains on one million self-supervised clips and routes only a small amount of the 60K cross-pair samples to t∈[0.95,1.0].
On a balanced 1,021-sample evaluation set, TARS reaches 83.18% R-Prec, 72.76% T-Prec, and 10.34° V-MPGE, outperforming CamClone and TrajCrafter. Its semantic scores for shot scale, viewing angle, and perspective are 0.83, 0.81, and 0.98, respectively. The method supports reverse-angle reshooting, large camera movements, and first-/third-person switching while maintaining high FSCS of 83.92%. Remaining challenges include extreme occlusion, long-range geometry, computational cost, and dependence on automated evaluators.
Deep Analysis
Background
Video diffusion has progressed from text-to-video generation to controllable camera motion, but reshooting is harder because each source frame exposes only one view. TrajCrafter reconstructs 3D point clouds before rendering, making it vulnerable to dynamic scenes, occlusions, and scale mismatch. CamClone learns from cross-pair data but suffers from limited real paired examples and synthetic-data bias. TARS reframes reshooting as joint semantic viewpoint and explicit trajectory control.
Core Problem
Given a source video Vsrc, target trajectory P′, and text T, the system must generate Vtgt with a new camera and viewpoint while preserving identity, appearance, action, and timing. The core bottlenecks are hallucinating unseen regions, separating camera motion from subject motion, and avoiding temporal drift or texture collapse when paired supervision is scarce.
Innovation
- ��Timestep analysis: high-noise steps learn global structure and motion; low-noise steps refine detail.
- ��Data routing: self-supervised data train most steps, while limited cross-pair data train only the high-noise regime.
- ��Semantic control: Qwen3-VL decomposes viewpoint into shot scale, viewing angle, and perspective.
- ��Unified conditioning: Camera Grid translates camera parameters into a dynamic visual representation compatible with diffusion models.
Methodology
- ��For P_i=[R_i|t_i], sample floor and ceiling grid points and project them with λu=K(R_ip+t_i).
- ��Encode videos with a 3D VAE and train Rectified Flow using z_t=(1−t)z_0+tε and ||vθ−(ε−z_0)||².
- ��Split an unlabeled video into V1 and V2; use V1 as source, V2 as target, and pair the target grid G2 with Qwen3-VL viewpoint text.
- ��Fine-tune with 60K cross-pair samples only for t∈[0.95,1.0], preserving self-supervised training elsewhere.
- ��At inference, jointly condition on Vsrc, G, and T to produce the target reshoot.
Experiments
Training uses one million self-supervised clips and 60K cross-pair samples: 10K high-quality real-world videos plus 50K Unreal Engine videos. The model is trained for 8K iterations with learning rate 5×10^-5. The evaluation set contains 1,021 online videos covering humans, objects, animals, and landscapes, balanced across simple/complex camera motion and scene dynamics. Baselines are CamClone, TrajCrafter, and SD-2.0. Metrics include R-Prec, T-Prec, V-MPGE, ArcFace, GCR, LSR, CE, FDR, and VDR.
Results
TARS reaches 83.18% R-Prec, 72.76% T-Prec, and 10.34° V-MPGE, versus 71.77%, 51.48%, and 21.67° for CamClone. It achieves 83.92% FSCS, 0.91 CE, 3.48% FDR, and 12.57% VDR. Semantic viewpoint scores are 0.83, 0.81, and 0.98, compared with SD-2.0's 0.69, 0.72, and 0.47. The timestep experiment further shows that source injection at high noise preserves layout and motion, whereas mid/low-noise injection mainly preserves identity.
Applications
Filmmakers and advertisers can generate reverse-angle, over-the-shoulder, high-angle, or first-person variants from existing footage using trajectory grids and natural-language prompts. Autonomous-driving and robotics researchers can synthesize alternative observation paths for data augmentation. Deployment requires a capable video-diffusion backbone, reliable trajectory estimation, and validation of generated labels and geometry before safety-critical use.
Limitations & Outlook
Because TARS is not an explicit geometric renderer, it does not guarantee metric depth, scale, or physically correct occlusion. Large reverse-angle changes, fast non-rigid motion, multiple interacting people, and long videos may cause hallucination or identity drift. Training requires substantial data and compute, while evaluation partly relies on Gemini 3.1 Pro. Future work should add depth or flow constraints, real paired data, longer temporal context, uncertainty estimation, and faster inference.
Plain Language Accessible to non-experts
Imagine a film kitchen preparing a new shot from an already cooked scene. The original video is a dish photographed from one side, but a customer asks for a closer view, a side view, or the cook’s own perspective. The kitchen must invent what was hidden behind the dish while keeping every actor, movement, and table position consistent.
TARS divides the work between an early planning crew and a finishing crew. The early crew decides the large arrangement: where the camera moves, where the person stands, and how the overall action unfolds. The finishing crew adds fine details such as faces, fabric, hair, and texture. The paper shows that camera movement is mostly decided during the early, noisy part of generation, so expensive paired demonstrations are unnecessary at every stage.
First, TARS learns from ordinary videos by using two clips and describing their viewpoint change with phrases such as “medium shot, side view, third person.” Then a small number of genuinely synchronized camera pairs teaches the system that the actor’s movements must remain aligned with the original. Most other training material can focus on appearance and general visual knowledge.
This lets TARS create high-angle, reverse-angle, over-the-shoulder, and first-person versions while filling in unseen regions plausibly. It can still guess incorrectly under total occlusion or extreme motion. Its central lesson is simple: plan the whole scene first, add detail later, and spend costly examples where they matter most.
ELI14 Explained like you're 14
Suppose you recorded a game character running from the front, but now you want the same clip from behind, from a rooftop, or through the character’s own eyes. The computer has never seen the character’s back in that moment. It must invent the missing view while keeping the legs, timing, clothes, and face from suddenly changing. That is video reshooting!
TARS handles this like a smart game editor. At the beginning, it decides the big things: where the camera goes, where the character is, and how the movement should look. Later, it fills in small details such as hair, fabric, and facial appearance. This is like drawing a sketch before coloring instead of polishing one shoelace while the whole picture is still wrong.
The system learns from lots of ordinary videos split into two parts. A language model labels the viewpoint with instructions like “close-up, side angle, first person.” Special videos filmed from two synchronized cameras are rare, so TARS uses only a small amount of them during the most important early stage. The rest of the time, it learns general appearance from huge amounts of video.
The results are impressive: rotation accuracy is 83.18%, translation accuracy is 72.76%, and the camera error is 10.34 degrees. Perspective accuracy reaches 0.98. It is not magic—fast action or completely hidden surfaces can still fool it—but it acts more like a director who follows both camera instructions and story continuity. Pretty useful, right?
Glossary
TARS
A 3D-free framework for video reshooting. It allocates different training data to different diffusion timesteps.
The paper's complete method, combining source video, Camera Grid, and text.
Camera Grid
A visual encoding of camera rotation and translation as projected spatial-grid motion. It is computed with λu=K(R_ip+t_i).
It provides the target camera trajectory to the diffusion model.
Rectified Flow
A deterministic generative formulation connecting noise and data through straight probability-flow paths. TARS trains it with conditional flow matching.
It serves as the video-generation backbone.
High-noise regime
The early denoising regime in which coarse structure and motion are formed. TARS defines it as t∈[0.95,1.0].
Limited cross-pair supervision is routed to this regime.
Cross-pair data
Video pairs showing the same scene, action, and appearance from different camera trajectories. They provide strong motion correspondence but are expensive to collect.
TARS uses 60K examples in Stage 2.
Semantic viewpoint control
Natural-language control over shot scale, viewing angle, and first-/third-person perspective. It maps cinematographic language to model conditions.
Qwen3-VL produces structured viewpoint descriptions.
Open Questions Unanswered questions from this research
- 1 Without depth or calibrated cameras, how well does TARS preserve long-range scale, occlusion ordering, and metric geometry? Current perceptual and trajectory metrics do not fully answer this question.
- 2 Generalization to fast non-rigid motion, crowded interactions, and long videos remains uncertain. Larger real cross-view datasets, object-level motion labels, and long-horizon benchmarks are needed.
- 3 The relationship between Gemini 3.1 Pro judgments and human cinematographer judgments remains underexplored, particularly for content expansion and visual distortion.
Applications
Immediate Applications
Film and advertising previsualization
Directors can provide a source clip, a Camera Grid, and prompts such as “medium side view” or “high-angle over-the-shoulder” to generate alternative shot drafts. This supports rapid storyboarding and asset reuse, but human review is still required for identity, occlusion, and brand-critical details.
Robotics and autonomous-driving augmentation
Researchers can transform a single observation trajectory into side, overhead, or reverse views for perception and policy training. The generated samples should be filtered with reliable trajectory metadata and checked for scene-label consistency before entering safety-critical datasets.
Long-term Vision
Interactive virtual cinematography
A future system could let viewers or directors drag the camera, switch between first- and third-person views, and preserve continuous action in real time. Achieving this requires faster diffusion inference, stronger 3D consistency, multi-person handling, and robust long-range occlusion modeling.
Abstract
Video re-shooting aims to regenerate videos with controllable camera motion and viewpoint. Existing methods rely on explicit 3D priors, which are limited by reconstruction quality and often perform poorly when synthesizing previously unseen regions, or on paired videos with different camera trajectories, whose scarcity hinders generalization. We revisit video re-shooting through text-driven semantic viewpoint specification, enabling control over shot scale, viewing angle, and first-/third-person perspective. To this end, we propose TARS, a 3D-free video re-shooting paradigm. Timestep-wise sensitivity analysis reveals that camera motion is primarily established during high-noise stages, where coarse spatiotemporal structures are formed. Based on this insight, we introduce self-supervised training to learn camera dynamics and fundamental visual representations without paired re-shooting data or 3D reconstruction. Through data scaling and joint textual-camera conditioning, TARS supports robust camera and viewpoint control, plausibly synthesizing regions beyond the source view under large camera motions while enabling reverse-angle re-shooting and perspective switching. Extensive experiments show that TARS provides more accurate and temporally consistent camera control than prior methods. Project Page: https://ymlinfeng.github.io/TARS.github.io/