Test-Time Perturbation Learning with Delayed Feedback for Vision-Language-Action Models
PDF combines uncertainty-aware augmentation with delayed-feedback logit correction, raising LIBERO success to 0.77 and Atari HNS to 1.07.
Key Findings
Methodology
PDF is a verifier-free test-time adaptation framework that freezes the base VLA. It estimates normalized action entropy, allocates augmented views proportionally to uncertainty, and applies dimension-wise action voting. A lightweight Perturbation Head then modifies action logits using episode-level delayed rewards, combining a REINFORCE-style objective with a positive-feedback-gated KL regularizer. Only this head is updated online.
Key Results
- Across LIBERO Spatial, Object, Goal, and Long, PDF reaches 0.77 average success rate and mean rank 2.5. It obtains 0.90, 0.72, 0.86, and 0.59 respectively, with the strongest relative advantage on long-horizon tasks.
- On Atari-57, PDF achieves 1.07 human-normalized score versus 0.97 for Jat, a +0.10 gain. Forty-seven of 57 games improve; BOXING gains about 60.25 percentage points, while BATTLE ZONE declines by 10.72.
- Ablations show that delayed feedback is essential: removing DF reduces Object and Goal performance to 0.50 and 0.77. A maximum augmentation budget of 3 is generally best; larger budgets accumulate noisy views.
Significance
The work reframes VLA brittleness as a decision-grounding problem, not merely a perception problem. Models may exploit accidental correlations among objects, backgrounds, gripper appearance, and memorized action traces. PDF uses environment outcomes without a pretrained verifier or base-model fine-tuning, offering a practical route to online robustness in robotics and multimodal control. Its frozen-backbone design also lowers deployment barriers for existing VLA systems.
Technical Contribution
PDF unifies uncertainty-driven multi-view augmentation, dimension-wise action voting, and delayed-feedback logit perturbation. Its uncertainty is Ut=−Σkp(ak|sk)logp(ak|sk)/logK, with Nt=NmaxUt. Final logits are z~=hϕ(f)+λhθ(f). The loss combines −(r−b)logπϕ with λKLI[r>b]KL(πϕ||π~), updating only a roughly 9M-parameter P head. This separates general multimodal representation learning from lightweight online policy correction.
Novelty
Relative to entropy-minimization TTA, which can amplify confidently wrong actions, and verifier-based best-of-N methods, PDF jointly uses uncertainty-targeted perturbations and outcome-guided logit learning. Within the paper’s setting, its fundamental novelty is adapting a frozen VLA through a small perturbation head while exploiting delayed environmental feedback rather than an external action verifier.
Limitations
- Evidence is restricted to simulated LIBERO and Atari-57. The paper does not establish safety, stability, or transfer under real-robot occlusion, sensor noise, contact dynamics, or severe visual shifts.
- Episode-level feedback creates a difficult credit-assignment problem, especially for long horizons or sparse rewards. Augmentation also increases inference cost, and excessive views harm performance through noise accumulation.
Future Work
Important next steps include real-robot validation, risk-aware updates, better temporal credit assignment, automatic benchmark-specific budget selection, and calibration under sparse or noisy rewards. Broader studies should test different augmentation families, larger VLA backbones, continual adaptation stability, and compute–performance trade-offs.
AI Executive Summary
Vision-language-action models have brought perception, language following, and control into one policy, but their apparent versatility can collapse after tiny environmental changes. A gripper may repeat a memorized route when a bowl moves slightly—or even when the target is masked. The paper calls this trajectory overfitting: the model relies on accidental correlations between entities, backgrounds, and actions. Existing test-time adaptation is also imperfect: entropy minimization may sharpen an already-wrong prediction, whereas verifier-based sampling requires extra models and repeated rollouts.
The authors propose Perturbation learning with Delayed Feedback, or PDF. At each step, normalized Shannon entropy measures action uncertainty and determines an adaptive augmentation budget. Original and transformed observations are processed jointly, and candidate actions are selected through dimension-wise voting. A lightweight Perturbation Head adjusts the action logits. After an episode ends, its success, failure, or reward updates only this head through a REINFORCE-style term and a KL penalty activated when feedback exceeds a baseline; the visual encoder, transformer, and LM head remain frozen.
Experiments use OpenVLA on four LIBERO suites and Jat on all 57 Atari 2600 games. PDF reaches 0.77 average LIBERO success, including 0.59 on Long, and 1.07 Atari HNS versus 0.97 for Jat. Forty-seven Atari games improve, with BOXING gaining 60.25 points. Removing delayed feedback reduces Object performance to 0.50, while budgets above three generally degrade results. PDF therefore demonstrates an efficient path to online robustness, though real-robot safety, sparse-reward credit assignment, and augmentation cost remain open challenges.
Deep Analysis
Background
VLA research evolved from multimodal models such as Gato, RT-1/RT-2, and OpenVLA, which connect visual observations and language instructions to executable control. OpenVLA provides a strong open platform for robotics, while Jat applies Gato-like generalist modeling to Atari. Yet pretrained policies often bind actions to incidental scene statistics—object pose, backgrounds, or gripper appearance. Test-time adaptation is consequently important for deployment beyond the training distribution.
Core Problem
The central question is how to correct overconfident sequential decisions without labels, a pretrained verifier, or gradients through the base VLA. Entropy minimization assumes confidence correlates with correctness, which fails under trajectory overfitting. Multi-view sampling can improve robustness but costs computation and may introduce noisy evidence. Delayed episode rewards provide useful supervision, yet assigning them to individual actions is difficult.
Innovation
- ��Adaptive uncertainty budget: normalized action entropy determines the number of augmented views. •Dimension-wise voting: each action component is voted across views, enabling flexible deviations from memorized trajectories. •Delayed-feedback adaptation: a small P head modifies logits while the backbone stays fixed. •Gated regularization: REINFORCE aligns perturbations with positive outcomes, while KL is applied only when reward exceeds baseline. Together, DA attacks spurious input correlations and DF corrects policy overconfidence.
Methodology
- ��Input: state st=(ot,ct), with pixels and text encoded into multimodal features ft. •Uncertainty: LM-head logits zt yield Ut=−Σkp(ak|sk)logp(ak|sk)/logK. •Augmentation: sample Nt=NmaxUt transformed views. •Perturbation: compute z~=hϕ(ft)+λhθ(ft), then decode candidate actions. •Voting: select each action dimension by majority vote and store features, logits, and outcomes in a rollout buffer. •Learning: after an episode, optimize LPDF=−(r−b)logπϕ+λKLI[r>b]KL(πϕ||π~). Only hθ is updated; all base VLA parameters are frozen.
Experiments
LIBERO uses Franka Panda RGB views, robot state, task text, and delta end-effector actions across Spatial, Object, Goal, and Long, each containing 10 tasks and 50 evaluation rollouts. OpenVLA is trained for 50 episodes. Atari evaluates all 57 Atari 2600 games with 50 episodes per game; episodes terminate on life loss, level completion, or 108,000 frames. Metrics are LIBERO success rate and Atari HNS. Baselines include OpenVLA, OCTO, TraceVLA, SFT-4LIBERO, MG-Select, and Jat. Ablations remove DA, DF, KL, or REINFORCE and vary budgets from 0 to 4.
Results
PDF obtains 0.77 average LIBERO success and rank 2.5, with Spatial/Object/Goal/Long scores of 0.90/0.72/0.86/0.59. Atari HNS rises from Jat’s 0.97 to 1.07; 47/57 games improve, including BOXING +0.60 and TIME PILOT +0.53. Removing DF gives only 0.50 on Object and 0.77 on Goal. Removing DA still often beats OpenVLA, but the complete system is more stable. Budget 3 is the best general trade-off.
Applications
PDF can be attached to OpenVLA-like systems for grasping, placement, rearrangement, and instruction-following under modest visual shifts. It also suits game agents with episode-level rewards. Requirements are image-text observations, an available success or reward signal, and modest extra inference for augmented views. Because the backbone is frozen and only about 9M parameters are trainable, deployment can reuse existing models without full retraining.
Limitations & Outlook
The evaluation is limited to simulation, so collision safety, camera noise, contact dynamics, and real-time hardware constraints remain untested. Adaptive budgets still require a ceiling and can become expensive; excessive augmentation lowers performance through noisy-view accumulation. Delayed rewards provide weak temporal credit assignment, especially in long or sparse-reward episodes. Future work should combine PDF with safety constraints, learned credit assignment, automatic budget control, and systematic tests across backbones and real environments.
Plain Language Accessible to non-experts
Imagine a cook who has learned thousands of kitchen videos. Whenever he sees a bowl and a plate, he automatically repeats the hand movement from a familiar video. If the bowl is moved a little, or covered by a cloth, he may still follow the old route instead of checking where the real target is. That is the paper’s central failure.
PDF gives the cook several helpers. First, it asks how unsure he is. If he is unsure, helpers look at slightly changed versions of the scene and independently suggest what to do. Their agreement decides each part of the movement. After the whole dish is finished, the kitchen gives one delayed verdict: success or failure. A small correction dial then strengthens movements associated with success and avoids blindly repeating failed ones. The main recipe book is left untouched.
In experiments, this raised average LIBERO task success to 0.77 and Atari’s normalized score to 1.07 from 0.97. But more helpers are not always better: a maximum of about three altered views usually works best because too many suggestions add noise.
ELI14 Explained like you're 14
Think of a game-playing robot that studies lots of walkthrough videos. It sees a familiar screen and copies the old button sequence, even if the box moved slightly. Sometimes the target is hidden and it still performs the same moves—like memorizing an answer without understanding the question. The paper calls this trajectory overfitting.
PDF gives the robot a mini team. First it checks how uncertain its next move is. If it is confused, the team looks at slightly edited versions of the image. Each member votes on parts of the move, so one weird image cannot control everything. Then, after the game ends, the robot learns from the result: winning strengthens useful moves, while losing stops it from becoming even more confident in a bad habit.
The clever part is that PDF changes only a small adjustment module, not the robot’s whole brain. On LIBERO it gets 0.77 average success; on Atari it improves from 0.97 to 1.07, and 47 of 57 games get better. More image versions can actually hurt, so three is often enough! The method still needs testing on real robots, where mistakes can break things.
Glossary
Vision-Language-Action Model
A model that maps images and language instructions to executable actions. Technically, it jointly processes multimodal context and predicts control tokens or action values.
OpenVLA and Jat are the base models evaluated by PDF.
Test-Time Adaptation
Adapting a deployed model during inference rather than retraining it offline. It uses test observations, self-supervision, or environmental feedback.
PDF updates only its Perturbation Head during deployment.
Trajectory Overfitting
Repeating action traces associated with familiar visual contexts instead of grounding decisions in the true goal. Small environmental shifts can therefore trigger failure.
The paper demonstrates it with masked targets and misleading attention.
Uncertainty-Based Action Voting
Using predictive entropy to allocate augmented views, then selecting actions through cross-view majority agreement. It concentrates computation on ambiguous states.
PDF uses Nt=NmaxUt and dimension-wise voting.
Delayed Feedback
A reward or success signal returned after a sequence rather than immediately after each action. It summarizes episode-level performance but complicates credit assignment.
The signal trains the perturbation module after each episode.
KL Divergence
A measure of how different two probability distributions are. As a regularizer, it limits unnecessary policy drift.
PDF gates KL(πϕ||π~) on positive feedback.
Open Questions Unanswered questions from this research
- 1 Real-world safety remains unresolved: it is unknown whether PDF can adapt reliably under camera noise, collisions, occlusion, and changing contact dynamics without unsafe exploration.
- 2 Sparse delayed rewards make action-level credit assignment difficult. Better temporal feedback models are needed to distinguish which perturbations actually caused success.
Applications
Immediate Applications
Robotic manipulation
Teams with OpenVLA-like systems can freeze the backbone and add the roughly 9M-parameter P head. Camera-view perturbations, dimension-wise voting, and episode success feedback can reduce failures caused by object-pose shifts.
Visual game agents
Jat-style agents can use PDF in Atari-like environments with episode rewards. No gradients through the base model are required, making online correction practical when inference can afford a few extra views.
Long-term Vision
Reliable general-purpose embodied agents
With real-robot safety constraints, sparse-reward handling, and automatic budget control, PDF-like adaptation could let general VLA policies calibrate themselves in new environments without repeated laboratory fine-tuning.
Abstract
Vision-Language-Action models (VLAs) achieve remarkable performance in sequential decision-making but remain fragile to subtle environmental shifts, such as small changes in object pose. We attribute this brittleness to trajectory overfitting, where VLAs over-attend to the spurious correlation between actions and entities, then reproduce memorized action patterns. We propose Perturbation learning with Delayed Feedback (PDF), a verifier-free test-time adaptation framework that improves decision performance without fine-tuning the base model. PDF mitigates the spurious correlation through uncertainty-based data augmentation and action voting, while an adaptive scheduler allocates augmentation budgets to balance performance and efficiency. To further improve stability, PDF learns a lightweight perturbation module that retrospectively adjusts action logits guided by delayed feedback, correcting overconfidence issue. Experiments on LIBERO (+7.4\% success rate) and Atari (+10.3 human normalized score) demonstrate consistent gains of PDF in task success over vanilla VLA and VLA with test-time adaptation, establishing a practical path toward reliable test-time adaptation in multimodal decision-making agents. The code is available at \href{https://github.com/zhoujiahuan1991/CVPR2026-PDF}{https://github.com/zhoujiahuan1991/CVPR2026-PDF}.