Hy-Embodied-VLM-1.0: Efficient Physical-World Agents
Proposes Hy-Embodied-VLM-1.0, integrating action-centric taxonomy, optimized data pipeline, and Mixture-of-Experts architecture, achieving state-of-the-art multi-task embodied agent performance.
Key Findings
Methodology
This approach employs an action-centric capability taxonomy dividing embodied intelligence into three levels: state understanding, transition reasoning, and sequential adaptation. It leverages the Hy3-A3B language backbone and Hy-ViT2 vision encoder, combined with a systematic data pipeline covering pretraining, fine-tuning, and reinforcement learning. The Mixture-of-Experts architecture enhances inference efficiency while maintaining high capacity. The training incorporates multi-stage supervision, including depth reasoning, task grounding, social interaction, and long-horizon planning, validated across 38 benchmarks. Results show top performance in 19 benchmarks, outperforming models like Qwen3.6-A3B and Cosmos 3, with an 8.4% average improvement over Hy-Embodied-0.5.
Key Results
- The model ranks first in 19 of 38 benchmarks, with an average performance boost of 4.4% over comparable models. Despite activating only 3B parameters, it approaches the performance of 32B models, demonstrating high efficiency. It excels in multi-turn interactions and long-horizon reasoning tasks, confirming its robustness in dynamic environments.
- In embodied perception, spatial reasoning, and navigation, the model surpasses existing state-of-the-art, especially in vision-language navigation and multi-modal understanding, showing excellent generalization across tasks and scenarios.
- The systematic data pipeline and capability taxonomy enable stepwise enhancement from perception to complex reasoning, setting a new paradigm for embodied intelligence research.
Significance
This work advances embodied AI by integrating multimodal perception, action reasoning, and long-term decision-making into a unified framework. It addresses longstanding challenges of efficiency and generalization, facilitating deployment in real-world robotics, virtual assistants, and autonomous systems. The model's ability to perform multi-task, multi-scenario reasoning paves the way for more autonomous, adaptable agents, accelerating progress toward human-level embodied intelligence.
Technical Contribution
Key innovations include the action-centric capability taxonomy, which guides data and model design; the integration of Hy3-A3B and Hy-ViT2 architectures; and the adoption of Mixture-of-Experts for scalable, efficient inference. The multi-stage training pipeline, combining supervised fine-tuning and reinforcement learning, systematically enhances the model’s embodied reasoning capabilities, enabling robust performance across diverse tasks.
Novelty
This is the first work to embed a comprehensive action-centric capability taxonomy into a unified embodied foundation model, systematically organizing data and architecture to address perception, reasoning, and long-horizon planning simultaneously. Unlike prior models focusing on static perception or isolated tasks, this approach emphasizes continuous, adaptive embodied intelligence, representing a significant leap forward.
Limitations
- Despite high efficiency, the model's performance drops in highly dynamic or unseen environments due to limited training data coverage. Its reasoning may falter under extreme complexity or ambiguity, especially in real-time scenarios.
- While the Mixture-of-Experts architecture improves inference speed, large-scale deployment still faces computational challenges, especially in resource-constrained settings.
- Dependence on multi-modal data makes the model vulnerable to missing or noisy inputs, which can degrade performance in real-world applications.
Future Work
Future directions include expanding training datasets to cover more diverse and extreme scenarios, further optimizing inference speed for real-time deployment, and integrating more sophisticated reinforcement learning techniques for autonomous long-term adaptation. Additionally, exploring unsupervised or semi-supervised learning could reduce data dependency, pushing embodied intelligence closer to human-level adaptability.
AI Executive Summary
The quest to develop autonomous embodied agents capable of operating effectively in the physical world remains a central challenge in artificial intelligence. Existing models excel in perception or static reasoning but struggle with dynamic interactions, long-horizon planning, and real-time adaptation. Addressing these gaps, this work introduces Hy-Embodied-VLM-1.0, a comprehensive embodied foundation model built upon a novel action-centric capability taxonomy. This taxonomy hierarchically organizes embodied intelligence into three levels: understanding current physical states, reasoning about actions and their local effects, and planning over extended sequences with adaptive feedback.
The model architecture combines the Hy3-A3B language backbone with Hy-ViT2 vision encoder, enhanced by a Mixture-of-Experts design that balances capacity and inference efficiency. A systematic data pipeline supports multi-stage training, including large-scale pretraining, targeted fine-tuning, and reinforcement learning with task-specific rewards. These strategies enable the model to excel across a broad spectrum of embodied tasks, from perception and spatial reasoning to complex navigation and multi-turn interaction.
Experimental results demonstrate that Hy-Embodied-VLM-1.0 outperforms current state-of-the-art models like Qwen3.6-A3B and Cosmos 3 on 19 of 38 benchmarks, with an average performance increase of 8.4%. It achieves near-competitive performance with only 3B active parameters, showcasing high efficiency. Its robustness in long-horizon, multi-turn tasks highlights its potential for real-world applications such as robotics, virtual assistants, and autonomous systems.
This research not only pushes the boundaries of embodied AI but also establishes a scalable, systematic framework for future advancements. Limitations include challenges in highly dynamic environments and computational costs, guiding future work toward dataset expansion, speed optimization, and more autonomous learning strategies. Overall, Hy-Embodied-VLM-1.0 marks a significant step toward truly intelligent, adaptable physical-world agents.
Deep Dive
Glossary
Hy3-A3B (语言骨架)
一种结合多模态信息的语言模型架构，用于增强模型的语言理解和推理能力。
作为模型的核心语言基础，用于支持复杂的多模态交互与推理。
Hy-ViT2 (视觉编码器)
基于Transformer的视觉编码架构，提升视觉感知与空间理解能力。
用于处理多视角、多模态的视觉信息，支持空间推理。
Mixture-of-Experts (专家混合架构)
一种模型架构，通过多个子模型协作，提高推理效率和容量。
在模型中实现高效推理和多任务处理。
动作中心能力分类
将实体智能划分为状态理解、动作推理和长时序规划三级能力体系。
指导模型设计和数据构建，系统化实现实体智能。
强化学习 (Reinforcement Learning)
一种通过奖励机制优化决策策略的学习方法。
用于提升模型在复杂交互和长序列任务中的表现。
Open Questions Unanswered questions from this research
- 1 当前模型在极端动态环境中的表现仍有限，特别是在高速运动或复杂交互场景下，数据覆盖不足导致理解偏差。未来需要扩展多模态、多场景训练数据，并结合自主学习策略提升泛化能力。
Applications
Immediate Applications
智能机器人
支持自主导航、操作和交互，提升工业自动化和服务机器人智能水平。
虚拟助手
增强虚拟助手的多模态理解和长时序交互能力，改善人机交互体验。
Long-term Vision
自主系统普及
推动自主机器人、无人驾驶等领域实现更高水平的自主决策和适应能力，改变未来生活方式。
Abstract
Building capable embodied agents requires not only multimodal perception and understanding, but also agentic capabilities for reasoning about actions, adapting to evolving situations, and interacting with the physical world. In this report, we introduce Hy-Embodied-VLM-1.0, an efficient and powerful embodied foundation model specifically designed for embodied agents operating in the physical world. To cultivate such capabilities from the pre-training stage onward, we define an action-centric capability taxonomy comprising three progressive dimensions: Action-Relevant State Understanding, Action-Transition Reasoning, and Sequential and Adaptive Reasoning. Guided by this taxonomy, we develop a systematic data pipeline and curate data mixtures spanning both pre-training and post-training. To deliver strong physical-world understanding and interaction capabilities while supporting latency-sensitive deployment, we build our model on the Hy3-A3B language backbone and the Hy-ViT2 vision encoder. Its efficient Mixture-of-Experts architecture combines strong model capacity with high inference efficiency. We evaluate Hy-Embodied-VLM-1.0 on a comprehensive suite of 38 benchmarks covering embodied perception, physical-world understanding, and embodied reasoning. The model achieves the best performance among similarly sized models on 19 of the 38 benchmarks and substantially outperforms strong competitors, including Qwen3.6-A3B and Cosmos 3. Compared with the previous-generation Hy-Embodied-0.5 MoT-2B, Hy-Embodied-VLM-1.0 improves average performance by 8.4%. Despite activating only 3B parameters, it achieves performance close to that of the previous-generation model with 32B activated parameters. Beyond static benchmark evaluation, Hy-Embodied-VLM-1.0 also demonstrates strong performance on embodied agentic tasks requiring multi-turn interaction and long-horizon reasoning.