ContextFlow: In-Context Flow Matching for Robot Manipulation
ContextFlow uses conditional flow matching for fine-tuning-free robot adaptation, reaching 73.5% average LIBERO success and surpassing ICRT by 35 points.
Key Findings
Methodology
ContextFlow formulates in-context imitation as conditional flow matching. A Context Expert encodes current images, text, and demonstrations, while Image, State, and Action Perceiver compressors reduce long multimodal trajectories to fixed-size latent tokens. An Action Expert receives proprioception and Gaussian-noise action tokens, then uses cross-attention and 10-step Euler integration to transport noise toward a continuous target action chunk, avoiding discrete action tokens and token-by-token decoding.
Key Results
- On unseen LIBERO configurations, ContextFlow achieves 73.5% average success, compared with 53.5% for ContextAR and 38.5% for ICRT—a 35-point improvement over ICRT. It also slightly exceeds task-specific one-shot fine-tuned π0 at 72.5%, despite requiring no fine-tuning on unseen tasks.
- ContextFlow obtains 86% and 42% on the two LIBERO-Spatial configurations, averaging 64%, and 76% and 90% on LIBERO-Object, averaging 83%. This indicates useful generalization across spatial relations and object substitutions, although performance remains configuration-dependent.
- ContextFlow-Plain reaches 64.0%, whereas the compressed ContextFlow reaches 73.5%, yielding a 9.5-point gain. On ALOHA, the study uses 1,064 training trajectories across 21 configurations and reports 40% success on a new pen-uncapping configuration.
Significance
The paper moves robotic in-context learning from discrete sequence prediction toward continuous distribution modeling. It directly addresses two persistent problems: quantization artifacts and long-horizon compounding errors under task-distribution shift. The results suggest that a robot can adapt to new objects, layouts, and hand assignments from demonstrations without parameter updates. Academically, the work connects vision-language-action modeling, flow matching, and Perceiver compression. Industrially, it could reduce the cost of collecting new data and fine-tuning policies whenever a workspace or manipulated object changes.
Technical Contribution
The contribution combines a conditional Flow Matching objective, a Context Expert–Action Expert architecture, and modality-specific Perceiver compressors. The training path is as=(1−s)ε+sat, with target velocity at−ε and loss E||uθ(as,ot,d,s)−(at−ε)||². At inference, Gaussian noise is integrated for K=10 forward-Euler steps to produce a continuous action chunk. Block-wise causal attention lets action tokens read context while preventing reverse information flow, so context representations can be cached across flow steps.
Novelty
Unlike ICRT, LipVQ-VAE, and RICL, which rely on autoregressive prediction over discretized action tokens, ContextFlow predicts continuous action chunks in parallel. Its central novelty is the integration of multimodal demonstration compression with conditional flow matching for low-level manipulation over trajectories of hundreds of steps. The paper also extends evaluation beyond simulation to real single-arm and bimanual ALOHA tasks, a setting less explored by prior in-context imitation work.
Limitations
- Evaluation mainly concerns unseen configurations within related task primitives, not entirely novel tasks. Therefore, the results do not establish open-world task learning or broad compositional generalization.
- The real-world study uses one ALOHA platform and 1,064 training demonstrations. Compression improves efficiency but may discard brief contact events or fine temporal coordination signals.
- The 40% success rate on new pen uncapping shows remaining fragility under occlusion, contact dynamics, bimanual synchronization, and recovery from early execution errors.
Future Work
Future work should scale real-robot data and platforms, test cross-embodiment and cross-primitive transfer, and study adaptation with fewer demonstrations. Promising directions include adaptive temporal compression, higher-order ODE solvers, uncertainty estimation, tactile feedback, online replanning, and explicit failure recovery. Systematic studies of action horizon H, flow steps K, demonstration count, and latency would clarify the accuracy–efficiency trade-off.
AI Executive Summary
A robot that can watch a few demonstrations and immediately handle a new object arrangement would dramatically reduce the cost of programming and data collection. Yet methods such as ICRT typically imitate the language-model recipe: continuous actions are discretized into tokens and generated autoregressively. A small early error changes the robot’s later state, causing errors to compound over long rollouts. Distribution shifts to unseen objects, layouts, or hand assignments make the problem worse. Flow matching offers continuous control, but its combination with demonstration-based in-context learning has remained underexplored.
ContextFlow addresses this gap with a conditional flow-matching policy. A Context Expert combines the current observation, task text, and demonstrations. Three Perceiver-style compressors summarize image, proprioceptive, and action streams into compact latent tokens. An Action Expert begins with Gaussian-noise action tokens and iteratively refines them through a learned velocity field, generating a continuous action chunk in parallel. The training path is as=(1−s)ε+sat, and inference uses 10 forward-Euler steps. Block-wise causal attention allows context caching across these steps.
On unseen LIBERO configurations, ContextFlow achieves 73.5% average success, versus 38.5% for ICRT, 53.5% for ContextAR, and 64.0% for ContextFlow-Plain. It also matches or slightly exceeds one-shot fine-tuned π0 at 72.5% without fine-tuning on unseen tasks. On a real ALOHA platform, the method covers single-arm and bimanual tasks and reaches 40% success on a new pen-uncapping configuration. The evidence is encouraging, but the benchmark remains within related primitives; open-ended tasks, cross-platform transfer, and reliable recovery from contact failures remain unresolved.
Deep Analysis
Background
VLA systems such as RT-1, RT-2, OpenVLA, and π0 have advanced vision-to-action learning, while ICRT and RICL explore demonstrations as task context. Most retain autoregressive discrete action modeling. Flow matching and action chunking address continuous control, but efficient encoding of long multimodal demonstrations had not been adequately integrated with in-context flow policies.
Core Problem
The target is π(at|ot,d), where at=at:t+H−1 is a future action chunk, ot contains image and proprioception, and d is a demonstration. The bottlenecks are action quantization, autoregressive error accumulation, quadratic attention over trajectories that may span hundreds of steps, and distribution shift caused by unseen objects, spatial relations, or left–right hand configurations.
Innovation
- �� Conditional Flow Matching directly models continuous action distributions.
- �� Context and Action Experts separate task-context encoding from iterative action generation.
- �� Three Perceiver compressors summarize visual, proprioceptive, and action sequences into fixed-size representations.
- �� Block-wise causal attention enables context caching and avoids recomputation during each flow step.
Methodology
- �� Inputs: RGB sequences, proprioception, actions, and text; demonstrations are subsampled to N1=8 image frames, N2=128 states, and N3=128 actions.
- �� Compression: each modality uses 32 learnable queries; the image compressor has 4 layers, while state and action compressors have 2 layers each.
- �� Flow objective: sample ε~N(0,I), construct as=(1−s)ε+sat, and train uθ toward at−ε with s~Beta(1.5,1.4).
- �� Inference: initialize a0~N(0,I), apply 10 Euler updates with δ=1/10, and output an H-step continuous action chunk.
- �� Training: SigLIP at 224×224; π0 initialization for vision and action components, LoRA rank 32, 20K iterations, batch size 32.
Experiments
LIBERO-Spatial and LIBERO-Object each contain 10 tasks and 500 expert demonstrations; eight tasks are seen and two configurations are unseen, with 50 trials per task. LIBERO-Goal and LIBERO-10 augment training. Baselines include ICRT, ContextAR, π0, OpenVLA-OFT, and one-shot fine-tuned π0. Real evaluation uses ALOHA: 21 configurations, 1,064 training trajectories, plus 254 training-only bimanual demonstrations.
Results
Average LIBERO success is 38.5% for ICRT, 53.5% for ContextAR, 64.0% for ContextFlow-Plain, and 73.5% for ContextFlow. π0 reaches 44.5%, while one-shot fine-tuned π0 reaches 72.5%. ContextFlow averages 64% on Spatial and 83% on Object. The 9.5-point gain over ContextFlow-Plain supports the value of multimodal compression. ALOHA achieves 40% on unseen pen uncapping.
Applications
The method targets warehouse picking, tabletop organization, object placement, tool use, and bimanual cooperation. Deployment requires RGB cameras, proprioception, a demonstration library covering relevant primitives, and a compatible action interface. New configurations can be introduced through demonstrations rather than retraining, potentially reducing operational costs when objects, positions, or hand assignments change.
Limitations & Outlook
The evaluation does not demonstrate completely novel task learning. Real-world evidence is limited to one ALOHA setup, so cross-embodiment, camera, and control-frequency robustness remain unknown. Perceiver compression may lose transient contact information; 10-step Euler integration introduces an accuracy–latency trade-off. The 40% pen-uncapping result also exposes weaknesses in occlusion handling, contact dynamics, synchronization, and recovery.
Plain Language Accessible to non-experts
Imagine a robot as a new employee learning to organize a kitchen. You show it several recordings of someone placing different objects in different locations. An old-fashioned learner memorizes one tiny instruction at a time: move the hand here, then there, then close the gripper. If the first move is slightly wrong, the object and the hand end up in the wrong places, so every later instruction becomes less useful. It also rounds smooth movements into a limited set of codes.
ContextFlow works more like an observant apprentice. It watches the pictures, the robot’s body position, and the teacher’s movements, then keeps a short summary: which object matters, where it is, which hand should act, and what the general sequence looks like. When acting, it first makes a rough whole plan and repeatedly cleans it up until the movement becomes smooth and executable. It plans a short stretch of motion together instead of guessing one tiny move at a time.
In the LIBERO simulation, this approach succeeds 73.5% of the time, compared with 38.5% for ICRT. On a real two-armed ALOHA robot, it succeeds 40% on a new way of uncapping a pen. It is not yet reliable for completely unfamiliar jobs or difficult hidden contacts, but it demonstrates a promising idea: showing a robot how something is done can be more useful than rewriting its entire brain.
ELI14 Explained like you're 14
Think of a robot game where the robot must pick up objects and put them in the right places. You do not want to retrain it every time the object moves or changes color, so you show it a few “pro gamer” clips. Older systems act like players who must guess one button press at a time. If they miss the first move, the robot’s position changes, and the next guesses become worse—like a combo breaking at the first button!
ContextFlow acts more like a teammate who watches the whole replay. It studies the scene, the robot’s joint positions, and the expert’s movements, then makes a compact note about the task. For example: “use the left hand, grab the cup, move around the box.” It does not turn every smooth movement into a tiny code. Instead, it starts with a messy action plan and improves that plan ten times until it becomes a useful sequence.
The results are impressive: in LIBERO, ContextFlow gets 73.5% average success, while ICRT gets 38.5%. On a real ALOHA robot, it tries one unseen pen-uncapping setup and succeeds 40% of the time. That means examples can help the robot adapt without changing its learned parameters.
But it is not a super robot yet! Completely new tasks, blocked camera views, slippery objects, and two hands needing perfect timing can still cause failure. Future versions could use touch sensors, notice when something goes wrong, and try again. Wouldn’t that be much closer to learning by watching a friend?
Glossary
Flow Matching
A generative method that continuously transforms a simple noise distribution into a target distribution. Technically, it learns a time-dependent velocity field rather than discrete action labels.
ContextFlow uses it to generate continuous action chunks from Gaussian noise.
In-Context Learning
Adapting behavior from examples supplied at inference time without updating model parameters. The examples function as a task-specific prompt.
The paper uses multimodal robot demonstrations as context.
Action Chunk
A sequence of future continuous actions predicted together. Chunking can reduce decision frequency and limit sequential error propagation.
ContextFlow predicts an H-step action chunk in parallel.
Perceiver Compressor
An attention architecture that uses a small set of learned queries to extract fixed-size latent representations from long inputs. It reduces the cost of long-sequence processing.
Separate compressors summarize images, states, and actions.
Compounding Error
A chain reaction in which an early control mistake changes later states and makes subsequent predictions increasingly wrong. It becomes especially severe under distribution shift.
The paper uses it to explain autoregressive failures on unseen configurations.
Open Questions Unanswered questions from this research
- 1 The method works within related task primitives, but whether a few demonstrations can support entirely novel tasks remains unknown. This requires broader tests of compositional skills, language goals, and manipulation behaviors.
- 2 Compression may remove brief contact or coordination signals. The field needs temporal-fidelity evaluations and adaptive choices of sampling rate and latent-token count.
- 3 The 40% new pen-uncapping result reveals weak recovery. Combining flow policies with uncertainty, tactile sensing, and online replanning remains an open challenge.
Applications
Immediate Applications
Warehouse and tabletop picking
Operators could provide a few demonstrations involving new objects or placements, allowing ContextFlow to execute related configurations. RGB cameras, proprioception, a compatible controller, and demonstrations covering the task primitive are required; retraining costs could be reduced when SKUs or layouts change.
Bimanual tool manipulation
Pen uncapping, box closing, handover, and wiping require role assignment and closed-loop coordination between hands. ALOHA results indicate feasibility, but deployment should add collision constraints, failure detection, and human takeover.
Long-term Vision
Demonstration-driven general-purpose robots
Robots could learn household or industrial variations from short human demonstrations, building transferable skills across objects, layouts, and hand assignments. Major obstacles include cross-embodiment transfer, open-ended task understanding, safety, and reliable recovery; progress is likely incremental rather than immediate.
Abstract
Although highly effective in vision and language domains, applying in-context learning to robotics remains challenging. Existing autoregressive in-context imitation methods discretize continuous actions and exacerbate the accumulation of early prediction errors through next-token prediction, limiting their generalization on unseen task configurations. Meanwhile, flow-matching policies have been explored for continuous robot control and can help mitigate compounding errors; however, in-context imitation learning within a flow-matching framework remains underexplored. To address these limitations, we introduce ContextFlow, a conditional flow-matching model that learns continuous action distributions for in-context imitation learning. ContextFlow conditions flow-based action prediction on demonstrations and observations, enabling robust generation from noisy action distributions. To better encode multimodal in-context demonstrations, we adapt perceiver-style multimodal context compressors that distill visual, proprioceptive, and action sequences into compact, task-relevant latent representations. On LIBERO, ContextFlow outperforms ICRT by 35 percentage points in average success rate on unseen task configurations, while matching the performance of the task-specific fine-tuned VLA model $π_0$ without any fine-tuning on unseen tasks. On real robots, it generalizes to unseen configurations of both single-arm and bimanual tasks, achieving 40% success on a new pen-uncapping configuration. Project Page: https://dingjiansw101.github.io/contextflow-page/.