UniVR: Thinking in Visual Space for Unified Visual Reasoning
UniVR uses VR-GRPO for native visual reasoning, improving VR-X over Emu3.5 by 18.4% overall and 25.2% in long-term planning.
Key Findings
Methodology
UniVR starts from the 34B-parameter Emu3.5 unified generator and models next-frame prediction, p(x_{t+1}|x_{1:t}), over visual trajectories. Cold-start SFT first supplies task priors; VR-GRPO then combines a global reward Rg for completion with a Step-Focal reward Rs. Rs uses inter-trajectory variance of CLIP embeddings to locate uncertain substeps. The final reward is Rreason=Rg−λ|Rg−Rs|, with λ=2 by default.
Key Results
- On VR-X, UniVR reaches 58.2 overall versus 39.8 for Emu3.5, an 18.4% absolute gain. Long-term planning rises from 42.4 to 63.8, a 25.2% gain, while general reasoning reaches 64.3 versus 46.2. Its JEPA distance falls from 33.62 to 13.01, indicating closer alignment with physical dynamics.
- Across Guidance, Robot, Editing, Spatial, Puzzle, and Search, UniVR scores 59.5, 68.0, 48.5, 46.5, 62.2, and 64.3. At 34B parameters it approaches or exceeds larger text-mediated pipelines, including Gemini-based systems, particularly on long-horizon manipulation.
- Ablations show that global reward alone yields 48.2 on VR-X, whereas adding Step-Focal and pairwise comparison reaches 63.8 with JEPA 13.01. UniVR also improves MMMU from 0.292 to 0.337 and MM-Vet from 28.0 to 35.6.
Significance
The work shifts world-model learning from language-mediated descriptions toward raw visual trajectories, targeting knowledge that is difficult to verbalize: contact, deformation, spatial continuity, and physical dynamics. It addresses a persistent weakness of MLLMs, which may produce excellent textual plans but visually inconsistent executions. VR-X and JEPA provide a broader evaluation vocabulary for the field. Industrially, the approach could reduce dependence on dense action-language annotation in robotics, although real-world validation remains essential.
Technical Contribution
The main contribution is VR-GRPO, a reward design tailored to multi-step visual generation. Unlike PPO, vanilla GRPO, HPSv3, or CLIP-based single-step rewards, it combines Qwen3-VL-30B global and local pairwise judgments with uncertainty-guided substep selection. At timestep t, CLIP features produce σ(t); the peak identifies divergent trajectories for focused evaluation. Rreason=Rg−λ|Rg−Rs| penalizes shortcut policies that reach a plausible endpoint through invalid intermediate states.
Novelty
The paper presents the first unified investigation, as claimed by the authors, of long-term planning, fine-grained physical dynamics, and general visual cognition learned directly in visual space without language supervision. Its fundamental novelty is treating visual state transitions as the reasoning trace itself, then allocating reward to disagreement-heavy substeps rather than relying only on terminal quality or task-specific rules.
Limitations
- The reward pipeline still depends on Qwen3-VL-30B and Qwen3.5-397B evaluators, so linguistic priors and visual hallucinations may affect supervision. CLIP variance identifies disagreement, not necessarily causal physical error.
- VR-X contains 1.8k evaluation samples and inherits the biases of 16 offline sources. Full-parameter 34B training, eight-way rollouts, 512-pixel videos, and a 20k-token limit also imply substantial computational cost.
Future Work
Future research should develop physics-aware visual critics, causal state representations, and closed-loop robot evaluation. Important extensions include real-time interaction, variable-length and multi-agent trajectories, broader cross-domain testing, online correction, and parameter-efficient RL. These directions could reduce reliance on VLM judges while improving safety and transfer.
AI Executive Summary
Humans often infer actions directly from changing scenes, yet current AI usually translates vision into language before reasoning. Gemini, GPT-5, and similar systems can write detailed plans, but their generated videos still contain skipped steps, broken object identities, and violations of physical continuity. Unified generators such as Emu3.5 provide a stronger visual interface, but remain heavily shaped by image-text supervision and artistic objectives.
UniVR proposes to “think in visual space.” Starting from Emu3.5, it receives an instruction and visual context, then autoregressively predicts the next frame. Cold-start supervised fine-tuning provides visual trajectory priors. VR-GRPO adds reinforcement learning: a global reward judges completion and visual quality, while Step-Focal reward inspects the time window where candidate trajectories diverge most. The combined objective, Rreason=Rg−λ|Rg−Rs|, discourages reaching a correct-looking endpoint through an incoherent process.
On VR-X, curated from 16 sources, UniVR scores 58.2 versus 39.8 for Emu3.5, improving overall performance by 18.4% and long-term planning by 25.2%. JEPA distance falls from 33.62 to 13.01. The same training improves MMMU from 0.292 to 0.337 and MM-Vet from 28.0 to 35.6. The results suggest that visual reasoning can improve both generation and multimodal understanding. However, the system still relies on VLM judges, offline data, and expensive 34B-scale training; causal physics, safety, and real-robot transfer remain open challenges.
Deep Analysis
Background
Modern MLLMs learn world knowledge mainly from text and use chain-of-thought reasoning. Systems such as GPT-5 and Gemini can pass textual plans to Nano Banana or GPT-image, but rendering often breaks continuity during manipulation. Unified models including Emu3.5 tokenize images, text, and video together, yet their training remains dependent on dense image-text data and entertainment-oriented objectives. UniVR instead treats raw visual trajectories as a source of action and physical knowledge.
Core Problem
The target is one model that can handle long-horizon manipulation, spatial puzzles, search, editing, and physical reasoning while preserving both terminal success and intermediate validity. The problem is difficult because task durations and domains vary widely, and subtle errors occur between the initial and final states. Global VLM scoring often rewards a clear endpoint while overlooking impossible contacts, skipped actions, or rule violations.
Innovation
First, UniVR performs next-frame reasoning directly in a unified visual space without dense language supervision. Second, VR-GRPO combines global completion reward with Step-Focal local assessment. Third, CLIP embedding variance automatically selects the most divergent subtrajectory instead of using task-specific heuristics. Fourth, VR-X unifies six task families from 16 sources and evaluates both logical quality through VLM scoring and physical similarity through JEPA.
Methodology
- �� Input: query image, instruction, and visual demonstration trajectory.
- �� Backbone: Emu3.5’s VQ-VAE-style unified discrete vocabulary, modeling p(x_{t+1}|x_{1:t}).
- �� Cold start: SFT on 310k visual-task samples using a standardized trajectory format.
- �� Rollouts: verl generates eight candidates per group; videos use a 512-pixel short side and up to 20k tokens.
- �� Step selection: compute σ(t)=sqrt(1/K∑||z_k(t)-z̄(t)||²) from CLIP embeddings; inspect a four-frame window around the peak, with segment alignment for variable lengths.
- �� Optimization: Qwen3-VL-30B performs global and local pairwise comparisons; rewards combine as Rg−λ|Rg−Rs|.
Experiments
VR-X begins with 1.5M raw samples from AgiBot, Action100M, EgoDex, VisualCoT, and 12 other sources, producing 310k cold-start, 3k RL, and 1.8k evaluation examples. Baselines include Qwen, Gemini, and GPT-5 paired with image generators, plus Janus-pro, Bagel, Emu3.5, and other unified models. Metrics are a 0–100 Qwen3.5-397B VLM score and JEPA distance. Ablations test reward components, text-based RL, joint training, and multimodal understanding transfer.
Results
UniVR scores 59.5, 68.0, 48.5, 46.5, 62.2, and 64.3 across Guidance, Robot, Editing, Spatial, Puzzle, and Search, for 58.2 overall; Emu3.5 scores 39.8. Long-term planning increases from 42.4 to 63.8, and JEPA falls from 33.62 to 13.01. The strongest reward configuration combines global and Step-Focal pairwise feedback. Gains also appear on all six reported multimodal-understanding metrics.
Applications
Immediate uses include robot imitation learning, household manipulation, cooking, navigation, craft assistance, and visual editing. A system can consume demonstrations plus a short instruction instead of dense step-by-step language. Deployment requires action controllers, collision checking, sensor feedback, and latency control so that generated frames correspond to executable behavior rather than merely plausible imagery.
Limitations & Outlook
The method treats VLM judgments as reward and may inherit language-derived biases or hallucinations. CLIP disagreement is only a proxy for error and cannot establish causal physical validity. VR-X is diverse but offline and relatively small in evaluation size. Full-parameter 34B training and eight-candidate rollouts are costly. Future systems should add causal physics critics, real-robot feedback, online correction, uncertainty calibration, and cross-environment generalization.
Plain Language Accessible to non-experts
Imagine teaching a robot to cook. A conventional system may read a long recipe, decide the steps in words, and ask another program to draw each action. It might know that the final dish should be a cake, yet spill the eggs, lose the bowl, or move a hand through the pan. Because the last picture looks attractive, the system may still receive a good score.
UniVR learns more like a person watching a cooking video. It observes how the pan, hand, ingredients, and tools change from moment to moment, then predicts what the next picture should look like. During training, one judge checks whether the whole dish is finished. Another judge looks closely at the few moments where different attempts disagree most, because those moments are likely to contain a mistake.
This means the robot is rewarded not only for a nice final picture but also for a believable process. VR-X tests this idea with tying knots, folding, cooking, robot control, puzzles, search, and spatial tasks. UniVR reaches 58.2 compared with 39.8 for its starting model, and its long plans improve by 25.2%. The approach still needs better judges and real-robot testing, but it demonstrates a simple principle: some knowledge is easier to learn by watching than by describing.
ELI14 Explained like you're 14
Think about a video game where you must grab a key, avoid traps, open a door, and rescue a character. A normal AI might write a brilliant walkthrough and ask an image generator to draw each step. But a written plan does not guarantee that the pictures connect: the key may vanish, the character may walk through a wall, and the final screenshot may still look awesome.
UniVR learns more like someone watching a skilled player’s replay. It does not need a paragraph explaining every move. It watches the screen change and predicts the next frame. It first studies lots of visual examples, then creates several possible replays. One referee asks, “Did you finish the mission?” Another referee zooms in on the moments where the replays disagree most: Did the hand really grab the object? Did the ball follow the route? Did anything teleport?
That is like checking whether you reached the game’s finish line without cheating through a wall. VR-X includes knot tying, cooking, robots, puzzles, search, and spatial reasoning. UniVR scores 58.2 while Emu3.5 scores 39.8, and long-term planning improves by 25.2%. It even helps other understanding tests such as MMMU and MM-Vet.
It is not magic yet! The referees can still be wrong, training is expensive, and most experiments use offline videos rather than real robots. The next step is an AI that watches, acts, notices mistakes, and fixes them in the real world. Then it would not merely talk about actions—it could perform them reliably!
Glossary
Visual reasoning
Reasoning directly from images or videos rather than translating everything into language first. Technically, it models and evaluates sequences of visual state transitions.
UniVR treats the visual trajectory as its reasoning trace.
VR-GRPO
A group-relative reinforcement-learning method designed for visual reasoning trajectories. It combines global task reward with local Step-Focal feedback.
It is UniVR’s central RL algorithm.
Step-Focal reward
A reward that concentrates evaluation on the substeps most likely to contain errors. UniVR selects them using high variance among CLIP trajectory embeddings.
It repairs the blind spots of global quality scoring.
JEPA similarity
A latent-space comparison between generated and reference trajectories. UniVR reports a maximum mean discrepancy, where lower values indicate closer physical-dynamic alignment.
It is VR-X’s physics-oriented metric.
VR-X
A benchmark spanning six visual-reasoning task families and 16 data sources, with cold-start, RL, and evaluation splits.
It trains and evaluates UniVR.
Open Questions Unanswered questions from this research
- 1 It remains unclear whether a VLM judge truly understands friction, contact, and causality. Physics simulators, sensor feedback, or human closed-loop assessment may be needed.
- 2 Offline video policies may fail in unseen environments, with new objects or real robots; the paper does not establish robust transfer.
- 3 Reducing the cost of 34B-parameter training and multi-sample rollouts while preserving the 18.4% gain remains an important engineering problem.
Applications
Immediate Applications
Robot imitation learning
Robots could learn grasping, folding, tying, and other procedures from videos plus short instructions, reducing action-language annotation. Deployment still requires controllers, collision checking, and physical validation.
Visual task planning
Home-service, navigation, cooking, craft, and editing systems could generate multi-step visual plans and inspect uncertain intermediate states. The main prerequisites are reliable sensors, execution feedback, and latency control.
Long-term Vision
Visual world models
Large-scale video learning could eventually unify perception, prediction, and action around executable world knowledge. Major obstacles include causal physics, safety certification, domain transfer, and real-time computation.
Abstract
Learning broad world knowledge directly from raw visual data is a fundamental capability of intelligence. We introduce UniVR, the first investigation into simultaneously learning complex reasoning, fine-grained physical dynamics, and long-term planning from pure visual demonstrations. At its core, UniVR features VR-GRPO, a reinforcement learning paradigm with complementary global and step-level rewards. This approach enforces logical coherence and physical consistency throughout the reasoning process without requiring task-specific heuristics or image-text pairs. To train and evaluate UniVR, we construct VR-X, a large-scale benchmark curated from 16 diverse sources spanning long-horizon manipulation, spatial puzzles, and physical reasoning. It is the first comprehensive suite to assess these heterogeneous capabilities under a purely visual protocol. Remarkably, UniVR achieves up to a 25% improvement on VR-X, and its superior visual reasoning also boosts performance on various multimodal understanding benchmarks. These findings underscore the vast potential of reasoning within visual spaces, with all code, data, and models are open-sourced for further research.