Elastic Queries Reinforcement Learning: Self-Aware Policy Execution for VLA Models
Proposes Elastic Queries Reinforcement Learning (EQRL) for self-aware resource scheduling in VLA models, reducing inference costs while maintaining success.
Key Findings
Methodology
EQRL employs a lightweight latent-schedule adaptor that jointly selects latent input, denoising budget, and action chunk length without fine-tuning the base VLA model. A critic ensemble estimates state difficulty via disagreement, guiding resource allocation. The approach formulates variable chunk execution as macro-action RL with chunk-dependent discounting and an amortized NFE budget. Residual learning refines scheduling decisions. Experiments in simulation and real robots demonstrate significant inference cost reduction while preserving or improving task success, with the critic effectively distinguishing easy and hard states for adaptive resource distribution.
Key Results
- On LIBERO and ALOHA benchmarks, EQRL improves success rate by 1-6 percentage points and reduces average NFE by 20-24%. In real robot tasks like pouring and placement, EQRL cuts inference time by over 30% while maintaining high success rates, outperforming fixed-schedule baselines.
- Ablation studies confirm that difficulty-aware joint scheduling is essential; simply reducing budget or adjusting one parameter yields less benefit. The method adapts dynamically to state complexity, optimizing resource use.
- In real-world experiments, EQRL decreases trajectory inference time and computational load, enabling faster, more efficient robot behaviors without sacrificing accuracy.
Significance
This work advances the field by integrating state-dependent resource allocation into VLA policy execution, addressing inefficiencies of fixed schedules. It demonstrates that adaptive, difficulty-aware scheduling significantly enhances robot autonomy and efficiency in complex, dynamic environments. The approach bridges the gap between generative modeling and real-time decision-making, paving the way for more intelligent, resource-efficient robotic systems. Its ability to reduce inference costs while maintaining high success rates has profound implications for deploying robots in industrial, service, and autonomous applications, where computational resources and response times are critical.
Technical Contribution
The key technical innovation is the integration of a critic-based difficulty signal into a macro-action RL framework for elastic scheduling. The method jointly optimizes latent input, denoising effort, and action chunk length, guided by critic disagreement, without fine-tuning the base VLA model. The use of chunk-dependent discounting and an amortized NFE budget enables flexible, state-sensitive resource allocation. This approach extends prior work on generative policy RL and adaptive inference by providing a unified, learnable scheduling mechanism that balances computational efficiency with task success, offering new theoretical insights and practical tools for resource-aware robot control.
Novelty
This is the first work to embed a critic-derived difficulty signal into a macro-action RL framework for elastic inference scheduling in VLA models. Unlike prior methods that fix inference parameters or adjust only one aspect, this approach dynamically allocates resources based on real-time state assessment, achieving superior success-efficiency trade-offs. It innovatively combines latent-space steering, macro-actions, and critic ensemble disagreement to enable adaptive, state-aware inference, filling a critical gap in generative robot policy execution.
Limitations
- The approach relies on offline critic training, which may not generalize well to unseen or highly dynamic environments. Real-time adaptation remains challenging under rapid environmental changes.
- Parameter tuning for the schedule priors and residual authority still requires manual effort; fully autonomous tuning is an open problem.
- Computational overhead of critic ensemble evaluation and online scheduling could limit real-time performance in highly complex scenarios. Future work should focus on optimizing computational efficiency and online learning capabilities.
Future Work
Future directions include integrating online learning for continuous adaptation, extending the framework to multi-agent systems for coordinated resource management, and exploring multi-modal signals for more precise state difficulty estimation. Additionally, developing self-tuning mechanisms for schedule parameters and deploying in more diverse, real-world environments will further validate and enhance the robustness of the approach.
AI Executive Summary
Robotic systems leveraging vision-language-action (VLA) models have achieved remarkable progress, yet their execution strategies often rely on rigid, fixed inference schedules. This rigidity hampers efficiency, especially in complex tasks involving variable difficulty states, such as contact-rich manipulation or uncertain environments. Existing fixed-schedule approaches either waste computational resources on easy states or underperform during challenging phases. To address this, the paper introduces Elastic Queries Reinforcement Learning (EQRL), a novel framework that enables self-aware, dynamic resource allocation during policy execution.
EQRL employs a lightweight latent-schedule adaptor that jointly selects latent inputs, denoising efforts, and action chunk lengths at each query. Critic networks, trained via ensemble disagreement, estimate the difficulty of current states, guiding the scheduler to allocate more computation to hard states and less to easy ones. This is achieved through a macro-action RL formulation, where variable-length action chunks are treated as high-level decisions, with chunk-dependent discounting and an amortized NFE budget controlling overall inference costs.
The core technical innovation lies in integrating a critic-derived difficulty signal into the scheduling process, enabling the policy to adaptively balance inference cost and task success. Extensive experiments in simulation on LIBERO and ALOHA benchmarks, as well as real robot tasks like pouring and placement, demonstrate that EQRL reduces inference costs by over 20% while maintaining or improving success rates. Ablation studies confirm that the joint, difficulty-aware scheduling mechanism is crucial for these gains.
This work significantly advances autonomous robot control by providing a flexible, resource-efficient execution paradigm. It opens pathways for deploying more capable, adaptive robots in real-world scenarios where computational resources and response times are critical. Despite some limitations in generalization and real-time overhead, the approach sets a new standard for intelligent, self-regulating policy execution, promising broader impacts in robotics and AI.
Deep Analysis
Background
机器人视觉-语言-动作（VLA）模型近年来快速发展，结合深度学习和大规模数据，已能实现复杂任务的自主操控。代表性工作如Anthony Brohan等的RT-1、Kevin Black的π0模型，以及Diffusion Policy等，推动了多模态感知与生成的融合。这些模型在任务泛化和复杂环境适应方面取得突破，但大多采用固定推理策略，难以应对任务中状态复杂度的变化。传统方法多依赖离线训练和模仿学习，缺乏动态调度机制，导致在实际应用中资源浪费或失败率升高。近年来，强化学习被引入以实现模型的后训练适应，但多采用静态调度或单一资源调节，未充分利用状态信息进行动态调度。随着机器人任务复杂度增加，研究者开始关注推理效率与任务成功的平衡，提出多种优化策略，但仍缺乏一种能根据状态难度自适应调度的统一框架。本文在此背景下，提出了弹性调度机制，结合actor-critic架构，实现基于状态难度的动态资源分配，推动机器人自主调度技术的创新。
Core Problem
现有VLA模型多采用固定推理和重规划策略，导致在状态复杂或不确定时资源不足，影响任务成功率。尤其在接触丰富或多变环境中，模型难以灵活调整推理强度，造成计算资源浪费或决策失误。如何根据状态难度动态调节推理预算和动作长度，成为提升机器人效率和鲁棒性的关键。传统调度方法多为手工调参或静态策略，缺乏对状态信息的敏感性，难以实现真正的自适应。引入强化学习进行调度优化，面临样本效率低、调度策略不稳定等挑战。本文旨在设计一种能在保持模型行为先验的基础上，动态调节推理资源的调度框架，解决资源浪费与任务失败的矛盾，提升机器人在复杂环境中的自主性。
Innovation
本研究的核心创新在于提出结合actor-critic的状态难度感知调度机制，将变长动作块作为宏动作进行强化学习，突破了传统静态调度的限制。具体包括：
- �� 引入轻量级潜在调度适配器，联合选择潜在输入、去噪预算和动作块长度，实现调度的端到端优化；
- �� 利用critic网络集成差异作为状态难度指标，引导调度资源在难控状态下增加去噪步骤、缩短动作块长度；
- �� 采用宏动作RL策略，将调度作为决策单元，结合块依赖折扣和NFE预算，提升调度的连续性和鲁棒性；
- �� 在不微调基础VLA模型的前提下，通过潜在空间调度实现行为优化，增强模型的适应性和可扩展性。这一机制显著提升了任务成功率和推理效率，为机器人自主调度提供了新思路。
Methodology
- �� 设计潜在调度适配器，输入当前状态和任务提示，输出潜在变量w、去噪预算K和动作块长度C；
- �� critic网络由多个Q函数组成，评估潜在调度联合动作的值，通过集成差异估算状态难度信号d(o)；
- �� 利用d(o)调整调度先验，状态越难，增加去噪步骤、缩短动作块；
- �� 结合残差机制，修正调度偏差，优化调度策略；
- �� 采用宏动作RL，定义带块依赖折扣的贝尔曼备份，训练调度策略和critic网络；
- �� 在训练中引入NFE预算约束，平衡推理成本与任务成功。
- �� 在模拟和真实机器人环境中，验证调度策略的有效性和泛化能力。
Experiments
在LIBERO和ALOHA模拟任务中，采用标准的成功率和成功AUC指标，比较EQRL与固定调度和其他RL方法的性能。利用离线和在线机器人任务，评估推理成本和任务成功率。超参数包括最大最小去噪步骤、动作块范围和NFE预算。通过消融实验验证调度中的难度感知和联合调度的重要性。实验结果显示，EQRL在保持或提升成功率的同时，显著降低平均NFE，表现出优越的资源调度能力。
Results
EQRL在LIBERO任务中成功率提升1-6个百分点，平均NFE降低20-24%，在ALOHA任务中成功率提升6个百分点，NFE降低20%。在真实机器人倒水、放置任务中，EQRL减少了30%以上的推理时间，且成功率与基线相当或更优。消融分析表明，难度感知和联合调度是性能提升的关键因素。
Applications
该方法适用于需要高效推理和动态调度的机器人自主操作场景，如仓储、制造和服务机器人。其无需微调基础模型，便于在不同任务中快速部署。未来，结合多模态信息和在线调度学习，可实现更复杂环境下的自主调度优化，推动机器人智能化水平提升。
Limitations & Outlook
目前调度策略依赖离线critic评估，泛化能力有限，难以应对极端或未见状态。调度参数仍需手动调节，缺乏完全自适应机制。实时环境中，调度延迟可能影响系统稳定性。未来需引入在线学习和多模态融合，增强适应性和鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房做饭，有时候需要准备很多调料，有时候只需要少量。厨师根据菜的难易程度，决定用多少调料，花多长时间。这里的调料就像机器人在执行任务时的计算资源，简单的步骤可以快速完成，不用太多调料；复杂的步骤需要花更多时间，准备更多调料。EQRL就像一个聪明的厨师助手，能根据菜的难度，自动调整用料和时间，既保证菜做得好，又不浪费调料。它学会了什么时候多花点时间，什么时候可以快点完成，节省了很多资源，也让厨房工作更高效。
ELI14 Explained like you're 14
你知道在玩游戏时，有时候遇到简单的关卡可以快速过关，但遇到难的关卡就需要花更多时间思考和尝试。这个机器人就像一个聪明的玩家，它能判断哪个关卡难，自己决定用多少时间和努力去解决。比如在简单的任务中，它会快速行动，不用太多计算；在复杂的任务中，它会多花点时间，仔细思考。这个方法让机器人变得更聪明，不会在简单任务上浪费时间，也能在困难任务中表现得更好。就像你在学习时知道什么时候要用功，什么时候可以轻松点一样。这样，机器人可以更快、更聪明地完成任务，还能节省很多计算资源。
Glossary
VLA (Vision-Language-Action) Model
A unified robot control model integrating visual perception, language-conditioned task specification, and generative action decoding.
Describes the foundational model used for action generation in the paper.
Latent-schedule adaptor
A lightweight module that jointly selects latent inputs, denoising effort, and action chunk length for adaptive inference scheduling.
Controls the dynamic resource allocation during policy execution.
Critic network
An ensemble of value functions estimating state-action values, used to derive state difficulty signals.
Guides the scheduling based on estimated task complexity.
Macro-action
A high-level decision unit comprising multiple low-level actions, used to model variable-length execution chunks.
Represents the variable-length action chunks in RL formulation.
NFE (Number of Function Evaluations)
A measure of computational effort in neural network inference, indicating the number of forward passes.
Used to quantify inference cost and optimize resource usage.
Open Questions Unanswered questions from this research
- 1 未来研究应关注调度策略在极端复杂环境中的泛化能力，尤其是在未见状态和快速变化环境下的适应性，仍需探索自适应参数调节和在线学习机制。
Applications
Immediate Applications
工业自动化调度
在制造线上，根据任务难度动态调节机器人推理资源，提高效率，降低能耗。
自主仓储机器人
实现仓库机器人在不同任务中自适应调度，节省计算资源，提升作业速度。
Long-term Vision
智能自主系统
未来机器人能自主判断任务复杂度，实时调节推理强度，适应多变环境，实现高效自主操作。
Abstract
Vision-language-action (VLA) models are powerful action generators for robot manipulation, but they are typically executed with fixed inference and replanning schedules. This rigidity ignores the uneven difficulty of robot control: contact-rich or uncertain states may need more computation and fresher feedback, while easier states can often be handled with fewer inference steps and longer open-loop execution. We propose Elastic Queries Reinforcement Learning (EQRL), a framework that makes each VLA policy query elastic. A lightweight latent-schedule adaptor jointly selects the latent input, denoising budget, and action chunk length, without fine-tuning the underlying VLA model. To make scheduling difficulty-aware, EQRL trains a critic over the joint latent-schedule action and derives a state difficulty signal from critic ensemble disagreement. This signal guides compute toward difficult states, while a learned residual allows task-driven correction. We formulate variable chunk execution as query-level macro-action RL with chunk-dependent discounting and an amortized number-of-function-evaluations (NFE) budget. Across simulation and real-robot manipulation, EQRL reduces amortized inference cost while preserving or improving task success.