Act with Intent: Distilling Behavior Intent for Vision-Language-Action Models
Proposes INDI, a method that distills behavior-level intent into vision-language-action models, boosting success rates by over 20% on benchmarks.
Key Findings
Methodology
The INDI framework leverages a frozen teacher VLM to interpret demonstrated behaviors, extracting multimodal intent representations. During training, the model recovers this intent at an intermediate decoder layer via learnable intent queries, integrating it into action and grounding predictions. This process combines visual, linguistic, and behavioral cues to form a semantic hierarchy guiding action generation. The approach involves: 1) extracting multimodal intent targets from demonstrations; 2) embedding intent queries within the decoder; 3) supervising and aligning intent representations; 4) organizing downstream predictions around recovered intent, which is removed at inference. This design enhances goal consistency and generalization.
Key Results
- On SimperEnv-Bridge, INDI raised GR00T-N1.7 success from 64.3% to 84.7%, a 20.4 percentage point increase, outperforming future supervision variants. On RoboCasa, success improved from 64.1% to 70.3%. In real-world tasks, success rose from 62.0% to 68.7%, with up to 12 percentage points gain on long-horizon tasks. Ablation studies confirmed that recovered intent captures behavior objectives and progression, organizing downstream predictions effectively.
- The method demonstrated consistent improvements across different benchmarks and backbones, including the Octo and NVIDIA GEAR models. The recovered intent representation was shown to be used by the decoder, providing a semantic scaffold that improved task success and out-of-distribution generalization. The results validate the hypothesis that explicit behavior-level intent modeling enhances policy performance and robustness.
- Further analysis revealed that the internal intent embedding encodes the behavior's goal and progress, serving as an organizing principle for downstream predictions. This internal semantic structure enables more interpretable and adaptable robot policies, paving the way for more autonomous and goal-aware systems in complex environments.
Significance
This work advances the field of robot learning by explicitly modeling behavior-level semantic objectives, moving beyond pure imitation. By integrating multimodal intent representations, the approach addresses core challenges in generalization and long-horizon planning. It bridges the gap between low-level action reproduction and high-level goal understanding, offering a scalable pathway toward more intelligent autonomous robots. The method's ability to improve success rates across simulated and real-world tasks demonstrates its practical relevance and potential for industrial, service, and household applications. It also opens avenues for combining intent-based supervision with reinforcement learning, fostering more flexible and goal-driven behaviors.
Technical Contribution
The paper's key technical innovations include: 1) the design of a multimodal intent extraction mechanism grounded in a frozen teacher VLM; 2) the integration of learnable intent queries within the decoder to recover and utilize behavior goals; 3) the formulation of an intent-aware decoding process that organizes subsequent predictions around the recovered intent. These contributions enable the model to explicitly encode and leverage behavior objectives, improving interpretability, robustness, and generalization. The approach also provides theoretical insights into the role of semantic intent in policy learning, establishing a new paradigm for goal-oriented robot control.
Novelty
This research is the first to explicitly incorporate behavior-level intent as a supervision signal in vision-language-action models. Unlike prior works focusing on future state prediction or action imitation, it emphasizes extracting and embedding semantic goals directly from demonstrated behaviors. The use of a frozen multimodal teacher model for intent supervision, combined with an intermediate intent query mechanism, represents a novel integration of multimodal understanding and goal organization. This approach significantly advances the state-of-the-art in semantic behavior modeling for robotic control.
Limitations
- The effectiveness relies heavily on the pre-trained teacher VLM's ability to interpret behaviors accurately; in scenarios where the teacher model's understanding is limited, intent extraction may be suboptimal.
- In highly dynamic or unpredictable environments, the static intent representations may struggle to adapt, requiring further mechanisms for online updating.
- The added complexity of intent queries and supervision increases training and inference costs, which may hinder deployment in resource-constrained settings.
Future Work
Future directions include developing self-supervised or weakly supervised methods for behavior intent learning, reducing reliance on large pre-trained teacher models. Incorporating reinforcement learning to dynamically update and refine intent representations during deployment could further improve adaptability. Extending the framework to multi-agent systems and complex hierarchical tasks will also be explored, aiming for more autonomous, goal-aware robotic systems capable of long-term planning and collaboration.
AI Executive Summary
The quest for truly autonomous robots hinges on their ability to understand and pursue complex goals in dynamic environments. Traditional imitation learning approaches, while effective at replicating demonstrated actions, fall short in capturing the underlying semantic objectives that drive behavior. This limitation hampers their ability to generalize to new tasks or adapt to unforeseen circumstances. Recognizing this gap, the present study introduces the concept of behavior-level intent and proposes a novel framework called Intention Distillation (INDI). INDI leverages a frozen teacher vision-language model to interpret demonstrated behaviors, extracting multimodal intent representations that encode the local objectives guiding actions.
During training, the model learns to recover these intent representations at an intermediate decoder layer through learnable intent queries. This recovered intent then guides subsequent action and grounding predictions, effectively organizing the behavior around a shared semantic goal. Crucially, at inference, all teacher modules are removed, and the model operates solely based on standard multimodal inputs, but with an internalized understanding of the behavior's purpose.
Experimental results across multiple benchmarks—including SimperEnv-Bridge, RoboCasa Kitchen, and real-world tabletop tasks—demonstrate significant performance gains. Success rates on simulated tasks improved by over 20 percentage points, reaching up to 84.7%, while real-world success increased by approximately 6.7 percentage points. These improvements highlight the importance of explicit goal modeling, enabling robots to better interpret instructions, adapt to new scenarios, and execute longer-horizon tasks.
The significance of this work lies in its shift from purely action-centric learning to goal-oriented behavior understanding. By explicitly modeling and recovering behavior intent, the approach enhances the interpretability, robustness, and generalization of robotic policies. This advancement opens new avenues for research, including integrating reinforcement learning for dynamic intent updates and extending to multi-agent systems. Despite current limitations related to dependency on pre-trained models and computational costs, the framework sets a promising foundation for future intelligent, goal-driven robotic systems capable of complex, autonomous operations.
Deep Analysis
Background
近年来，机器人自主控制技术快速发展，尤其在多模态学习和深度强化学习的推动下，出现了如OpenAI的GPT-4、DeepMind的Gato等大规模模型。这些模型通过大规模数据训练，显著提升了机器人对复杂场景的理解和操作能力。代表性工作如Zitkovich等的Vision-Language-Action模型、Kim等的多模态机器人系统，已在多项任务中实现了零样本迁移和泛化。然而，现有方法多依赖模仿示范动作，缺乏对行为背后目标的明确建模，导致在新环境或长时序任务中表现不佳。近年来，未来状态预测和结构化世界建模尝试弥补这一不足，但仍未解决行为意图的显式表示问题。
Core Problem
核心问题在于机器人行为的目标导向性不足，现有模仿学习方法只关注动作复制，忽略了行为背后的语义目标。这导致模型在面对新任务或环境变化时，难以保持一致性和泛化能力。行为的多样性和复杂性使得单纯的动作模仿难以捕获行为的本质意图，尤其在长时序、多目标、多模态场景中表现尤为突出。如何在训练中引入行为目标的显式建模，成为提升机器人自主性和理解能力的关键瓶颈。
Innovation
本研究创新点包括：1）提出行为意图蒸馏（INDI）机制，通过冻结的教师VLM从示范行为中提取多模态意图，作为中间层的目标表示；2）引入可学习的意图查询，将行为目标嵌入动作解码器，增强目标导向性；3）设计了意图引导的解码流程，使行为目标在动作预测中起到组织作用。这些创新区别于传统的行为模仿和未来状态预测，首次实现了行为层次的语义组织，为机器人行为理解提供了新路径。
Methodology
- �� 训练阶段：利用冻结的教师VLM解析示范行为，生成多模态意图目标，包括行为目的和场景变化。
- �� 提取目标：在教师模型中，利用中间层隐藏状态提取多模态意图表示，结合文本描述和视觉特征。
- �� 解码器设计：引入可学习的意图查询（Q_I），在中间层恢复意图状态，并在后续预测中引导动作、视觉和语言地面。
- �� 目标对齐：通过表示对齐损失，确保模型内部的意图表示与教师目标一致。
- �� 训练目标：结合动作流匹配、意图对齐和地面预测，优化整体策略。
- �� 测试阶段：移除教师模型和目标生成模块，模型自主恢复意图并进行行为预测。
Experiments
采用SimperEnv-Bridge和RoboCasa Kitchen两个模拟环境，以及真实机器人桌面任务，评估INDI在不同场景下的性能。对比基线包括行为克隆、未来状态预测和其他模态增强方法。指标主要为成功率和泛化能力，超参数设置包括：意图对齐的tap深度为6，流动时间λ=0.5，训练轮次3000，使用特定的视觉和文本编码器。通过消融实验验证意图表示的作用，分析模型内部的行为目标组织机制。
Results
在SimperEnv-Bridge中，INDI将GR00T-N1.7的成功率从64.3%提升至84.7%，在RoboCasa Kitchen中，从64.1%提升至70.3%，均优于未来状态预测等方法。在真实场景中，成功率由62.0%提升至68.7%，特别在长时序任务中提升达12个百分点。消融实验显示，意图表示的引入不仅提升了目标一致性，还增强了模型对环境变化的适应性。模型内部的意图表示能有效组织行为目标，提升策略的泛化能力。
Applications
该方法适用于机器人自主操作、工业自动化、家庭服务等场景，尤其在复杂、多目标、多模态任务中表现优越。依赖预训练的视觉-语言模型，结合行为意图的显式建模，能显著提升机器人对新环境的适应能力和任务完成效率。未来可结合强化学习，动态更新行为目标，实现更智能的自主决策。
Limitations & Outlook
目前方法依赖于预训练教师模型的多模态理解能力，若教师模型在特定任务中表现不足，可能影响意图提取效果。此外，在极端复杂或动态环境中，行为意图的准确性和鲁棒性仍需验证。模型训练复杂度较高，计算资源消耗较大，实际部署时需权衡效率与效果。未来需优化意图提取的鲁棒性和效率。
Plain Language Accessible to non-experts
想象你在厨房做饭，你知道自己要做一道菜，但具体步骤和目标可能不完全清楚。传统方法就像只记住每个动作，比如切菜、炒菜，但不知道为什么要这么做。本文的方法就像请一位经验丰富的厨师（教师模型）帮你分析整道菜的目标——比如做出一道香味四溢的炒饭。然后，你在做饭时，记住这个目标（意图），在每一步都想着“我要做出香喷喷的炒饭”，而不是单纯模仿动作。这样，即使环境变化，比如换了锅或食材，你也能根据目标调整动作，做出满意的菜。这种理解让机器人更聪明，能更好地完成各种任务，而不是死记硬背动作。
ELI14 Explained like you're 14
想象你在学校里参加一个科学比赛，你知道自己要做一个火箭，但不知道具体怎么做。以前，你只会模仿别人做火箭的步骤，但不知道为什么要这么做。现在，有个聪明的老师会告诉你：‘你的目标是让火箭飞得更远’，并帮你理解每个步骤的意义。你在做火箭时，就会记住这个目标——让火箭飞得更远，而不是单纯模仿别人的动作。这样，即使你用不同的材料或工具，也能根据目标调整方法，做出更好的火箭。机器人也是一样，学会理解行为背后的目标，就能更聪明地完成任务，不怕环境变化。
Glossary
Vision-Language Model (视觉-语言模型)
一种结合视觉和文本信息的深度学习模型，用于理解和生成多模态内容。论文中用以解析示范行为的多模态意图。
在本文中，教师VLM用来从示范行为中提取多模态意图目标。
意图蒸馏 (Intention Distillation)
一种将行为级目标从教师模型中提取并嵌入到学生模型中的方法，以增强策略的目标导向性和泛化能力。
核心创新，显式引入行为目标作为中间表示，指导动作预测。
行为层次意图 (Behavior-level Intent)
指行为中所追求的局部目标或目的，区别于单纯的动作或状态预测，是行为的语义核心。
本文强调在动作解码中恢复行为意图，提升策略理解和泛化。
多模态意图目标 (Multimodal Intent Target)
结合视觉、语言和动作信息，描述行为背后目标的多维表示。
由教师模型生成，作为训练目标引导学生模型学习。
Open Questions Unanswered questions from this research
- 1 如何在更复杂或动态环境中保持意图表示的鲁棒性仍未解决，未来需结合强化学习和自主目标生成技术。
- 2 目前模型依赖预训练教师模型，若教师理解不足，可能影响整体性能，需探索无监督或弱监督的意图学习方法。
Applications
Immediate Applications
机器人自主操作
结合行为意图蒸馏技术，提升机器人在复杂环境中的目标导向性和泛化能力，适用于工业自动化、家庭服务等场景。
多模态任务学习
通过显式建模行为目标，增强机器人对多模态信息的理解和协调能力，改善长时序任务表现。
Long-term Vision
自主决策与协作
结合意图蒸馏与强化学习，推动多机器人系统的自主决策、协作与长远规划，实现更智能的自动化生态。
Abstract
Vision-Language-Action (VLA) models can turn multimodal context into robot actions, but their action decoders are still trained largely by behavior cloning. This supervises which motor command was demonstrated while leaving implicit the local objective served by the behavior under the instruction. Future-based supervision enriches action learning with frames, latent observations, trajectories, or motion representations, but these signals capture particular realizations of what may happen rather than the shared semantic objective of the forthcoming behavior. We propose Intention Distillation (INDI), which distills behavior-level intent into the action decoder. During training, a frozen teacher VLM interprets a demonstrated segment from the current observation, instruction, coarse action summary, and corresponding execution video. From its standard inputs, the deployed VLA recovers the resulting multimodal intent representation at an intermediate decoder layer and uses it to organize action prediction together with representations of how the behavior unfolds and what it achieves. On SimplerEnv-Bridge, INDI improves GR00T-N1.7 from 64.3% to 84.7%, and on RoboCasa Kitchen it improves the controlled GR00T-N1.7 baseline from 64.1% to 70.3%, with consistent gains on $π_{0.5}$ across both benchmarks. In real-world tasks, INDI improves average success from 62.0% to 68.7%, with gains of up to 12.0 pp on longer-horizon tasks. Further analyses show that the recovered latent is used by the decoder, captures behavior objective and execution progress, and organizes downstream predictions in an objective-dependent manner. These results show that action decoders benefit from explicitly modeling the semantic objective of the behavior they generate.