LessMimic: Long-Horizon Humanoid Interaction with Unified Distance Field Representations
LessMimic leverages Distance Field (DF) for long-horizon humanoid interaction, enabling reference-free inference and skill composition with 80-100% success across object scales.
Key Findings
Methodology
LessMimic employs a distance field (DF) to extract surface distances, gradients, and velocity decompositions as geometric cues. These are encoded into a latent space via a variational autoencoder (VAE). The training pipeline includes behavior cloning from a teacher policy, reinforcement learning fine-tuned with adversarial interaction priors (AIP), and visual distillation for vision-only deployment. The approach removes dependence on motion references, supports multi-skill long-horizon tasks, and generalizes across diverse object geometries and scales. The model achieves success rates of 80-100% on pick-up and sit-stand tasks, and maintains stable performance over sequences of up to 40 tasks.
Key Results
- The policy achieves 80-100% success on pick-up and sit-stand tasks across object scales from 0.4x to 1.6x, outperforming baselines that degrade sharply under scale variation. On five-task trajectories, success reaches 62.1%, and the model sustains performance over sequences of 40 tasks, demonstrating robust skill composition. The vision-only version, trained via distillation, attains 35.8% success, validating effective sim-to-real transfer.
- Across diverse geometries, the method shows strong generalization, with success rates remaining high even on unseen shapes. Ablation studies reveal the importance of the distance field latent space and adversarial regularization for performance, confirming the core innovations' effectiveness.
Significance
This work addresses the fundamental challenge of geometric representation in humanoid interaction, proposing a unified, shape- and scale-agnostic approach that supports long-horizon, multi-skill behaviors without motion references. It significantly advances autonomous manipulation, enabling robots to adapt to unstructured environments, recover from failures, and seamlessly compose skills. The approach bridges perception and control efficiently, opening new avenues for scalable, general-purpose humanoid robots in industry and service sectors. Its ability to operate solely on geometric cues simplifies deployment, reduces reliance on motion capture, and enhances robustness in real-world scenarios.
Technical Contribution
The paper introduces a novel use of distance fields (DF) as a continuous, differentiable geometric representation for humanoid interaction. It combines DF-based local surface cues with velocity decomposition, encoding these into a latent space via VAE. The training pipeline integrates behavior cloning, reinforcement learning with adversarial regularization (AIP), and visual distillation, creating a unified framework for multi-skill, long-horizon control. This approach departs from prior methods reliant on motion references or discrete geometry, offering a shape- and scale-invariant, real-time capable solution that supports skill composition and recovery.
Novelty
This is the first work to embed local distance field cues into a unified, reference-free control policy for long-horizon humanoid interaction. Unlike prior approaches that depend on motion demonstrations or task-specific rewards, this method leverages the intrinsic geometric invariance of DF to enable generalization across object shapes and scales. Its integration of VAE encoding, adversarial regularization, and visual distillation for sim-to-real transfer marks a significant innovation, facilitating robust, flexible, and scalable humanoid control.
Limitations
- The model's robustness may decline in highly dynamic or cluttered environments where sensor noise and rapid scene changes challenge perception accuracy.
- Dependence on simulation training and domain randomization may limit real-world transfer in scenarios with unmodeled complexities or sensor failures.
- Real-time control at high frequency and across extensive task sequences requires further optimization to reduce computational load and latency.
Future Work
Future research will focus on integrating multi-modal perception, such as tactile and auditory cues, to enhance robustness. Developing online adaptation mechanisms will improve real-world deployment, especially in unstructured environments. Additionally, exploring hierarchical planning and learning strategies could enable even more complex task sequences and autonomous decision-making, pushing the boundaries of humanoid autonomy.
AI Executive Summary
Humanoid robots have long been envisioned as autonomous agents capable of complex, multi-skill interactions within unstructured environments. However, existing approaches often rely heavily on motion demonstrations or task-specific rewards, which tightly couple the policies to particular object geometries and limit their ability to generalize. This bottleneck hampers the development of versatile, long-horizon humanoid control systems capable of adapting to new objects, recovering from failures, and seamlessly composing multiple skills.
In response, the authors introduce LessMimic, a novel framework that leverages the concept of Distance Fields (DF) as a unified geometric representation. DF encodes local surface distances, gradients, and velocity decompositions, providing dense, shape- and scale-invariant cues for contact-rich interaction. These cues are encoded into a compact latent space via a variational autoencoder (VAE), enabling the policy to reason about object geometry without explicit motion references.
The training pipeline of LessMimic involves three key stages: behavior cloning from a physics-based teacher policy, reinforcement learning fine-tuned with adversarial interaction priors (AIP) to promote geometric generalization, and visual distillation to enable deployment solely on egocentric depth images. This approach results in a single, versatile policy capable of executing multiple tasks such as pick-up, sit-stand, push, and carry, with success rates exceeding 80% across various object scales. Notably, the policy maintains performance over sequences of up to 40 tasks, demonstrating robust skill composition and recovery.
Experimental results validate the effectiveness of the DF-based representation, showing superior generalization to unseen geometries and scales compared to baseline methods. The visual-only version, trained via distillation, achieves comparable success without requiring motion capture infrastructure, paving the way for real-world deployment. This work significantly advances the field of embodied intelligence by providing a scalable, geometry-aware, and autonomous control framework for humanoid robots.
Looking ahead, future work will focus on integrating richer perception modalities, enhancing online adaptation, and optimizing computational efficiency. The potential applications span industrial automation, service robotics, and assistive technologies, where adaptable, long-horizon humanoid interaction is essential. Overall, LessMimic offers a promising pathway toward truly autonomous, general-purpose humanoid robots capable of operating seamlessly in complex, unstructured environments.
Deep Analysis
Background
机器人自主交互技术经历了从运动模仿到感知驱动的快速演进。早期方法如DVRK和RoboCup利用模仿学习实现复杂动作，但受限于示范的几何依赖。近年来，深度强化学习（如DeepMimic、VICES）推动了自主控制的发展，但多技能长时序任务仍受制于几何表示瓶颈。传统几何表示（点云、网格）在感知中表现优异，但难以满足实时控制需求。隐式神经表示虽具表达力，但推理速度不足。距离场（DF）作为连续、可微的几何描述，兼具高效性和鲁棒性，为实现多技能自主交互提供了新途径。
Core Problem
现有方法多依赖运动参考或任务特定奖励，导致策略难以泛化到未见过的对象形状和尺度。长时序多技能交互中，模型缺乏统一的几何表示，难以实现连续技能切换和故障恢复。如何设计一种既能支持多样几何变化，又能实现自主长时交互的表示体系，成为核心难题。此问题关系到机器人在复杂、未结构化环境中的自主适应能力，是推动智能机器人普及的关键。
Innovation
本研究提出基于距离场（DF）的几何表示，利用局部距离和梯度描述接触关系，支持尺度和形状的泛化。结合变分自编码器（VAE）编码交互潜在变量，避免依赖运动示范。引入对抗交互先验（AIP）在强化学习中正则化几何合理性，提升泛化能力。训练流程包括行为模仿、几何正则化和视觉蒸馏，形成端到端的自主交互策略。模型支持多技能连续执行，具备故障恢复和技能组合能力，显著优于传统方法。
Methodology
- �� 通过距离场（Φ）提取局部几何信息，包括距离和梯度。• 利用速度分解，将速度拆分为沿法线和切线分量，捕捉接触动态。• 构建时间序列交互表示It，包含多关节局部几何特征。• 使用VAE编码It为潜在变量zt，压缩信息。• 训练流程：
- 行为模仿：模仿带有参考运动的教师策略，初始化模型。
- 强化微调：在随机几何环境中，利用AIP正则化几何合理性，提升泛化。
- 视觉蒸馏：将模型迁移到纯视觉输入，实现无运动捕获部署。
Experiments
在模拟环境中，采用多尺度、多形状对象（0.4x到1.6x尺度）进行测试。对比运动参考和无参考基线，评估成功率和接触率。设置五个任务（PickUp、SitStand、Push、Carry）和连续多任务场景，验证模型的泛化和长时序能力。采用成功率、接触频率等指标，进行多轮AB测试和消融分析，验证各组件贡献。模型在不同尺度和复杂场景中表现优异，尤其在连续任务中保持稳定。
Results
模型在不同尺度对象上成功率达80-100%，明显优于传统运动参考方法。连续执行五个任务的成功率为62.1%，支持连续40个任务的长时序操作。无运动捕获的视觉模型在复杂场景中表现出良好的鲁棒性，达35.8%的成功率。消融实验显示，距离场潜在空间和AIP正则化是性能提升的关键因素。
Applications
该方法适用于工业机器人、服务机器人等场景，能实现自主长时序多技能操作，无需运动捕获设备，降低部署成本。基于几何表示的泛化能力，使机器人能在未知环境中快速适应不同对象和任务。未来结合自主规划和多模态感知，将推动机器人在复杂环境中的自主学习与适应。
Limitations & Outlook
模型在极端复杂或动态环境中仍存在鲁棒性不足的问题。训练依赖模拟环境，实地应用时可能面临模拟-实物差异。高频控制和大规模任务的实时性仍需优化，计算成本较高。未来需增强模型的在线适应能力和感知鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房做饭，手里拿着锅，想把菜放到碗里。你不用看菜的每个细节，只需要知道锅和碗之间的距离、角度和你手的动作方向。这个距离和角度就像一种特殊的地图，告诉你什么时候碰到锅或碗，什么时候可以滑过去。机器人也是这样，它用一种叫距离场的“几何地图”来判断自己和物体的关系。这样，无论锅变大还是变小，它都能正确操作，不需要提前教它具体怎么做。它还可以在做完一件事后，马上开始下一件事，比如先拿起东西，再放到指定位置，连续完成多个任务，就像你在厨房里一气呵成一样。这种方法让机器人变得更聪明、更灵活，能在复杂环境中自主完成各种任务，就像一个有自己“地图”的厨师一样。
ELI14 Explained like you're 14
想象你在玩一个超级酷的游戏，你的角色可以自己决定下一步做什么，不用你告诉他每个动作的细节。比如，你想让角色去拿个玩具，然后坐下来休息。以前的机器人就像是需要你给它一份详细的说明书，告诉它每一步怎么做，但这样它就只能做一件事，遇到新玩具就不知道怎么操作。现在，这个新方法就像是给机器人画了一张“几何地图”，告诉它玩具和它的距离、角度，机器人可以自己判断什么时候去拿，什么时候坐下。它还可以连续做很多事情，不用每次都重新教它。这样一来，机器人变得更聪明、更灵活，就像你在玩一款可以自己学习新技能的游戏一样，能在不同的房间、面对不同的玩具都能应对自如。是不是很酷？未来，这样的机器人可以帮我们做家务、陪伴老人，甚至帮忙搬东西，变得越来越像人类的助手！
Abstract
Humanoid robots that autonomously interact with physical environments over extended horizons represent a central goal of embodied intelligence. Existing approaches rely on reference motions or task-specific rewards, tightly coupling policies to particular object geometries and precluding multi-skill generalization within a single framework. A unified interaction representation enabling reference-free inference, geometric generalization, and long-horizon skill composition within one policy remains an open challenge. Here we show that Distance Field (DF) provides such a representation: LessMimic conditions a single whole-body policy on DF-derived geometric cues--surface distances, gradients, and velocity decompositions--removing the need for motion references, with interaction latents encoded via a Variational Auto-Encoder (VAE) and post-trained using Adversarial Interaction Priors (AIP) under Reinforcement Learning (RL). Through DAgger-style distillation that aligns DF latents with egocentric depth features, LessMimic further transfers seamlessly to vision-only deployment without motion capture (MoCap) infrastructure. A single LessMimic policy achieves 80--100% success across object scales from 0.4x to 1.6x on PickUp and SitStand where baselines degrade sharply, attains 62.1% success on 5 task instances trajectories, and remains viable up to 40 sequentially composed tasks. By grounding interaction in local geometry rather than demonstrations, LessMimic offers a scalable path toward humanoid robots that generalize, compose skills, and recover from failures in unstructured environments.