ChronoFlow-Policy: Unifying Past-Current-Future Interaction Flow in Visuomotor Policy Learning
ChronoFlow-Policy unifies past-current-future interaction flow via sparse 3D keypoints, boosting robot manipulation success by 20%.
Key Findings
Methodology
This work introduces ChronoFlow, a unified spatiotemporal representation capturing past, current, and future interaction dynamics through sparse 3D keypoints of objects and gripper. Based on this, a diffusion-based ChronoFlow-Policy is developed, jointly learning interaction flow and action sequences via a co-training objective. The architecture includes an observation encoder, ChronoFlow encoder, diffusion backbone, and decoders for interaction and actions. The training employs a multi-loss approach, supervising the interaction flow prediction and action generation, enhancing robustness in long-horizon and non-Markovian tasks.
Key Results
- Experiments across 14 simulated and 5 real-world tasks show consistent success rate improvements over baselines, with an average increase of over 20%. In MetaWorld, success rates rose from 47% to 72%, and in RoboTwin from 56% to 66%. The model maintains performance in long-horizon tasks like Prepare Breakfast, demonstrating effective interaction flow prediction. Ablation studies confirm the importance of historical and future interaction streams, especially in deformable and staged tasks.
- In real-world tests, the model effectively memorizes early states, improving decision continuity in non-Markovian scenarios. It outperforms existing methods like DP3, 3D-FDP, and MBA, particularly in tasks requiring precise contact and multi-step coordination, validating the practical benefits of interaction-centric modeling.
- Overall, the approach advances the state-of-the-art in visuomotor control by explicitly modeling the entire interaction timeline, enabling more robust, generalizable, and interpretable robot policies.
Significance
This research addresses a fundamental gap in robot learning—integrating past, present, and future interaction dynamics into a unified framework. By doing so, it significantly enhances robots' ability to perform complex, staged, and deformable object manipulations in unstructured environments. The method bridges the gap between static scene understanding and dynamic interaction modeling, paving the way for more autonomous, adaptable robots capable of long-horizon planning. Its success demonstrates the importance of temporal interaction modeling, inspiring future work in multi-modal, multi-task, and real-world robotic intelligence.
Technical Contribution
The core innovation is the development of a sparse 3D keypoint flow representation that encodes interaction dynamics across time. Coupled with a diffusion-based prediction framework, it enables joint modeling of past, current, and future trajectories. The architecture integrates an attention-based encoder, a diffusion denoising backbone, and decoders for interaction flow and action generation, forming an end-to-end trainable system. This approach surpasses prior methods limited to scene flow or pose trajectories, offering a more expressive and task-relevant interaction understanding.
Novelty
This is the first work to unify past, current, and future interaction flows into a single sparse keypoint-based representation for robot manipulation. Unlike previous approaches focusing solely on scene flow or object pose, it explicitly models gripper-object interactions over time. The integration of diffusion models for joint flow prediction and action decoding further distinguishes it from existing methods, providing a new paradigm for temporal interaction understanding in robotics.
Limitations
- The reliance on accurate keypoint tracking makes the method sensitive to occlusion and sensor noise, which can degrade performance in cluttered or dynamic environments.
- The diffusion sampling process is computationally intensive, limiting real-time deployment without further optimization.
- Current experiments are mainly in controlled simulation and limited real-world tasks; scaling to more diverse, complex scenarios remains a challenge.
Future Work
Future directions include improving keypoint robustness via multi-modal perception, reducing computational costs through model compression, and extending to multi-robot systems. Incorporating self-supervised learning could reduce annotation needs, while broader testing in unstructured, real-world environments will validate scalability. Additionally, integrating language or high-level planning signals could further enhance autonomous capabilities.
AI Executive Summary
This paper introduces ChronoFlow-Policy, a novel approach to robot visuomotor control that unifies past, current, and future interaction dynamics through a sparse 3D keypoint flow representation. Traditional methods often model only static scene states or dense scene flow, limiting their ability to handle complex, staged, or deformable object manipulations. Inspired by human reasoning, the authors propose a framework that captures the evolution of object and gripper keypoints over time, providing a comprehensive temporal context for decision-making.
The core technical innovation lies in combining this interaction-centric representation with a diffusion-based prediction model. The diffusion backbone iteratively refines noisy interaction trajectories, enabling the model to jointly learn and predict the entire interaction flow. This flow then guides the generation of robot actions via a lightweight decoder, resulting in a unified, interpretable, and robust policy. The architecture includes an observation encoder, a ChronoFlow encoder, a diffusion denoising network, and decoders for interaction and actions.
Extensive experiments on simulated benchmarks like MetaWorld and RoboTwin, as well as real-world tasks, demonstrate that ChronoFlow-Policy outperforms existing state-of-the-art methods, including scene flow and pose-based approaches. Success rates improved by over 20%, especially in tasks requiring long-horizon reasoning and deformable object handling. Ablation studies confirm the importance of integrating past and future interaction streams, highlighting the method’s ability to model complex, staged manipulations.
The significance of this work extends beyond performance metrics. It provides a new perspective on temporal interaction modeling, emphasizing the importance of unified past-current-future representations for autonomous robots. The approach enhances robustness, generalization, and interpretability, paving the way for more adaptable and intelligent robotic systems. Future work aims to improve keypoint tracking robustness, reduce computational costs, and extend the framework to multi-modal, multi-robot scenarios, further advancing the field of robotic manipulation.
Deep Dive
Abstract
Visual signals play a crucial role in policy learning by enabling models to capture object motion and interaction dynamics. Just as humans reason about actions using both past experience and anticipated outcomes, effective policies should integrate past interactions with future predictions. However, existing visuomotor policies typically model either historical context or future dynamics in isolation, lacking a unified temporal representation of interaction dynamics. In this work, we introduce ChronoFlow, a temporally unified representation that captures past, current, and future interaction dynamics through sparse 3D keypoints of both objects and the gripper. Based on this representation, we propose ChronoFlow-Policy, a diffusion-based visuomotor policy that jointly learns ChronoFlow and action sequences through a co-training objective. Experiments on 14 simulated tasks and 5 real-world manipulation tasks demonstrate that ChronoFlow-Policy consistently outperforms strong diffusion-policy baselines and improves robustness in long-horizon and non-Markovian manipulation scenarios. Our project page is available at https://the-kamisato-sii.github.io/ChronoFlow-Policy-project-page/.