CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL
CreFlow employs a compositional LTL reward model with localized correction and success-guided reflow loss, boosting sparse-reward embodied video RL success by 23.8%.
Key Findings
Methodology
This approach introduces a reward model based on Linear Temporal Logic (LTL) that automatically formulates task requirements as compositional constraints. It integrates a perception-grounded state lifting process using SAM3, VLM, and IDM modules to extract entity states. The LTL monitor evaluates these states, producing binary success signals and localized violation traces. The framework employs a credit-aware NFT loss that confines gradient updates to reward-relevant regions, and a corrective reflow loss that leverages within-group successful samples to explicitly guide failed rollouts. These components enable high-dimensional video optimization by focusing training on task-critical regions, reducing gradient noise and improving stability.
Key Results
- On eight RoboTwin manipulation tasks, CreFlow outperforms baselines with a 23.8% increase in downstream success rate, aligning reward signals closely with human and simulator labels. Its localized training components significantly enhance training stability and efficiency, as demonstrated by ablation studies.
- Compared to DiffusionNFT and DanceGRPO, CreFlow achieves higher success rates and more consistent task success judgments, validating the effectiveness of combining logical constraints with localized reward optimization.
- Ablation results show that restricting updates to reward-relevant regions and using success samples for correction markedly improve training convergence and robustness, especially in high-dimensional video spaces.
Significance
This work addresses the challenge of sparse, non-decomposable rewards in embodied video generation for robotic manipulation. By integrating logical task specifications into the reward design, it provides a more faithful and interpretable success measure, advancing the state-of-the-art in task-aligned video synthesis. The approach bridges the gap between perceptual plausibility and physical correctness, enabling more reliable autonomous robot learning and deployment in complex environments. Its innovations in localized reinforcement and explicit correction open new avenues for scalable, interpretable, and physically consistent video-based robot training.
Technical Contribution
Key technical contributions include: 1) the automatic synthesis of compositional LTL constraints for task specification; 2) a localized, reward-relevant gradient restriction mechanism via a shared spatio-temporal mask; 3) a novel success-guided corrective reflow loss that explicitly pulls failed rollouts toward successful behavior averages; 4) integration of perception modules (SAM3, VLM, IDM) for accurate state extraction. These innovations collectively improve training stability, efficiency, and interpretability over prior pixel-level or global reward-based methods, enabling effective high-dimensional video optimization.
Novelty
This is the first work to embed compositional LTL constraints directly into embodied video RL, enabling logical task specification and localized diagnostics. Unlike prior methods relying solely on visual similarity or low-level motion metrics, CreFlow combines symbolic logic with high-dimensional visual feedback, providing a new paradigm for sparse reward optimization in complex manipulation tasks. Its explicit correction strategy and reward localization represent significant advances over existing pixel-based or trajectory-based approaches.
Limitations
- The method depends heavily on the accuracy of perception modules; visual or state estimation errors can mislead constraint evaluation, affecting reward fidelity.
- LTL expressiveness is limited to finite-trace formulas, which may not capture highly complex or long-horizon tasks, requiring future extensions to richer logical frameworks.
- Computational costs are high due to multi-modal perception and online RL, limiting scalability to very large datasets or real-time applications.
Future Work
Future directions include integrating more expressive temporal logic for complex tasks, improving perception robustness, and reducing computational overhead. Extending the framework to real-world robotic systems and multi-agent scenarios will test its generalization and practical utility. Additionally, combining this approach with learning-based perception modules could further enhance robustness and scalability in diverse environments.
AI Executive Summary
Robotic manipulation tasks demand videos that are not only visually realistic but also physically plausible and task-compliant. Existing video generation models trained on heterogeneous datasets often produce outputs that, while perceptually convincing, violate physical constraints critical for downstream robotic control. Reinforcement learning offers a promising pathway to adapt these models, but current reward signals tend to be overly simplistic, focusing on low-level visual similarity rather than the logical correctness of robot-object interactions.
This paper introduces CreFlow, a novel online reinforcement learning framework that leverages a compositional reward model based on Linear Temporal Logic (LTL). The core idea is to automatically formulate task requirements as a set of logical constraints, which are then evaluated over generated videos to produce a binary success signal and localized violation traces. These traces enable the training process to focus updates on task-critical regions, avoiding unnecessary perturbations in irrelevant areas. The framework employs a credit-aware NFT loss that confines gradient updates to reward-relevant regions, and a corrective reflow loss that uses successful samples within the same group to explicitly guide failed rollouts toward correct behavior.
Extensive experiments on eight RoboTwin manipulation tasks demonstrate that CreFlow significantly outperforms existing methods, achieving a 23.8% increase in downstream task success rate. The reward judgments align more closely with human and simulator labels, validating the approach’s fidelity. The localized training strategy not only improves success rates but also enhances training stability, reducing gradient noise and accelerating convergence. These advances suggest that integrating logical task specifications with localized reinforcement learning can substantially improve the quality and reliability of embodied video generation, paving the way for more capable autonomous robots.
Looking ahead, future work will focus on expanding logical expressiveness, improving perception robustness, and deploying the framework in real-world robotic systems. The combination of symbolic reasoning and high-dimensional visual feedback offers a powerful paradigm for scalable, interpretable, and physically consistent robot learning, with broad implications across robotics, computer vision, and AI safety.
Deep Dive
Abstract
Video generation models trained on heterogeneous data with likelihood-surrogate objectives can produce visually plausible rollouts that violate physical constraints in embodied manipulation. Although reinforcement-learning post-training offers a natural route to adapting VGMs, existing video-RL rewards often reduce each rollout to a low-level visual metric, whereas manipulation video evaluation requires logic-based verification of whether the rollout satisfies a compositional task specification. To fill this gap, we introduce a compositional constraint-based reward model for post-training embodied video generation models, which automatically formulates task requirements as a composition of Linear Temporal Logic constraints, providing faithful rewards and localized error information in generated videos. To achieve effective improvement in high-dimensional video generation using these reward signals, we further propose CreFlow, a novel online RL framework with two key designs: i) a credit-aware NFT loss that confines the RL update to reward-relevant regions, preventing perturbations to unrelated regions during post-training; and ii) a corrective reflow loss that leverages within-group positive samples as an explicit estimate of the correction direction, stabilizing and accelerating training. Experiments show that CreFlow yields reward judgments better aligned with human and simulator success labels than existing methods and improves downstream execution success by 23.8 percentage points across eight bimanual manipulation tasks.