Perception-and-action system for humanoid robot task execution in construction
Proposes Humanoid-PoseNet and Humanoid-ActionNet for learning construction tasks from worker demonstrations, achieving 82.45mm MPJPE accuracy.
Key Findings
Methodology
The system integrates two deep networks: Humanoid-PoseNet, which estimates 3D human poses from 2D videos and retargets them to humanoid configurations using bone angular error optimization; and Humanoid-ActionNet, which employs reinforcement learning with physics-aware rewards to learn stable, executable actions. PoseNet uses MLP encoders for human and humanoid poses, trained with triplet loss based on bone angular error to ensure accurate pose transfer. ActionNet learns control policies that maintain balance, contact stability, and coordination during task execution. The training involves simulation and real-world transfer, enabling the robot to perform construction-related actions with high precision.
Key Results
- The humanoid robot successfully performed 8 construction tasks with an average MPJPE of 82.45mm, demonstrating high accuracy and stability across simulated and physical environments.
- Action success rates exceeded 85% in diverse scenarios, confirming robustness and generalization of the control policies.
- Compared with baseline pose estimation methods like OpenPose and SMPL-X, the proposed approach achieved superior pose accuracy and smoother motion transfer, especially in contact-rich tasks.
Significance
This work advances the field of construction robotics by enabling humanoid robots to learn complex tasks directly from human demonstrations, addressing the challenge of task generalization and physical stability. It bridges the gap between perception and control, paving the way for autonomous, adaptive robots capable of working safely alongside humans in hazardous environments. The methodology enhances the practicality of deploying humanoid robots in real construction sites, reducing labor costs and improving safety standards.
Technical Contribution
The paper introduces a novel combination of pose estimation, cross-domain retargeting, and reinforcement learning with physics-aware rewards, enabling stable, physically feasible motion generation. The use of bone angular error for pose similarity and the teacher-student RL framework for sim-to-real transfer are key innovations that improve robustness and accuracy over existing methods. The integrated architecture offers a scalable framework for multi-task learning in humanoid robotics.
Novelty
This is the first study to enable humanoid robots to learn construction tasks directly from human demonstrations using a deep perception-action pipeline. It uniquely combines 3D pose estimation, cross-domain retargeting, and physics-informed reinforcement learning, filling a critical gap in autonomous construction robotics and setting a new benchmark for imitation learning in complex, contact-rich environments.
Limitations
- The system's performance in highly unstructured or cluttered construction environments remains to be tested, as current experiments are limited to controlled settings.
- Real-time processing and computational efficiency need further optimization for on-site deployment.
- The approach depends on quality and diversity of demonstration data; rare or complex motions may still pose challenges.
Future Work
Future efforts will focus on integrating multimodal sensors such as force and tactile feedback to enhance physical interaction fidelity, developing end-to-end autonomous learning frameworks to reduce dependence on demonstrations, and expanding the system's adaptability to diverse construction scenarios with minimal retraining.
AI Executive Summary
The construction industry faces increasing pressure to improve safety and efficiency amid labor shortages and complex environments. Traditional robots are limited to specific tasks, lacking adaptability for varied construction activities. This study introduces a novel perception-and-action system that empowers humanoid robots to learn from human workers directly. Central to this approach are two deep neural networks: Humanoid-PoseNet, which estimates 3D human poses from video and retargets them to humanoid configurations using bone angular error optimization, and Humanoid-ActionNet, which employs reinforcement learning guided by physics-aware rewards to develop stable, executable control policies.
Humanoid-PoseNet combines 2D-to-3D pose estimation models like VideoPose3D with a cross-domain retargeting network built on multi-layer perceptrons. The pose estimation module reconstructs accurate 3D skeletons from worker videos, while the retargeting module maps these to humanoid-compatible poses, addressing morphological differences via a triplet loss based on bone angular error. This ensures the transferred poses are physically feasible for the robot.
Humanoid-ActionNet leverages reinforcement learning with physics-informed reward functions, including joint tracking, balance, and contact stability, to learn control policies that execute construction actions such as carrying pipes, stacking bricks, and lifting boxes. Training occurs in simulation, with successful transfer to real hardware validated through experiments demonstrating high precision (average MPJPE of 82.45mm) and task success rates over 85%. The system's robustness across different scenarios highlights its potential for real-world deployment.
This integrated approach marks a significant step toward autonomous humanoid construction robots capable of learning complex tasks from human demonstrations, reducing reliance on predefined scripts or manual programming. It offers a scalable framework that combines perception, retargeting, and physics-based control, promising safer, more flexible, and cost-effective construction automation. Future work will focus on multimodal sensing, real-time optimization, and expanding task diversity, aiming to transform the construction industry with intelligent robotic collaborators.
Deep Analysis
Background
建筑行业面临劳动力短缺、安全风险和高成本，自动化技术逐步成为解决方案。早期研究集中在特定任务机器人，如砖铺机器人和材料搬运机器人，但缺乏能自主学习多任务的 humanoid 机器人。近年来，姿势估计和全身控制技术取得突破，为 humanoid 机器人赋能提供了基础，但在复杂施工环境中的应用仍受限。现有方法多依赖预定义动作或繁琐调控，难以实现自主学习和环境适应。随着深度学习的发展，利用视觉感知实现动作迁移成为研究热点，为机器人自主学习提供了新路径。
Core Problem
核心问题在于如何让 humanoid 机器人从工人示范中学习建筑任务，解决姿势迁移的精度和动作执行的稳定性难题。具体挑战包括：人体姿势的跨域重定向、复杂环境中的动态平衡控制、以及在施工现场的实时响应能力。传统方法在姿势估计和动作控制上存在精度不足、适应性差等问题，限制了机器人在实际施工中的应用潜力。实现高效、鲁棒的自主学习体系成为亟待解决的关键。
Innovation
本研究提出两大创新：一是Humanoid-PoseNet，结合2D到3D姿势估计与骨向角误差（BAE）优化，实现工人动作的高精度迁移，解决人体形态差异问题；二是Humanoid-ActionNet，采用强化学习结合物理感知奖励，学习稳定的全身控制策略，确保动作的平衡和接触稳定。这一架构突破了传统单一任务限制，支持多任务学习和环境适应，首次实现工人示范到机器人自主执行的完整流程，极大提升了机器人在施工场景中的实用性。
Methodology
- �� 采集工人示范视频，利用PoseNet模型进行2D到3D姿势估计，获得工人动作的三维骨架。
- �� 设计人类到 humanoid 的姿势重定向网络，使用多层感知机（MLP）和骨向角误差（BAE）优化迁移效果。
- �� 训练过程中，利用三元组损失（triplet loss）拉近相似姿势，推远不相似姿势，确保迁移的稳定性。
- �� 构建强化学习控制策略，结合物理感知奖励（如平衡、接触一致性），实现动作的动态稳定。
- �� 在模拟环境中训练控制策略，验证其在实际硬件上的迁移能力，完成多项建筑任务。
Experiments
采用Unitree humanoid机器人在模拟和实验平台上进行测试，任务包括搬运管道、堆砖、抬箱等。评估指标为MPJPE误差和动作成功率，模型参数通过交叉验证调优。对比不同姿势估计模型，验证迁移效果和控制策略的鲁棒性。实验还包括不同环境干扰和动作复杂度的测试，确保系统在实际施工场景中的适应性。
Results
机器人在8个建筑任务中平均运动误差为82.45mm MPJPE，动作成功率超过85%。姿势迁移的骨向角误差显著优于传统方法，验证了模型的高精度。强化学习控制策略在复杂环境中表现出良好的稳定性和适应性，支持自主执行多样化任务。系统在模拟和硬件环境中的表现一致，证明了迁移能力和实用性。
Applications
该技术可应用于施工现场的自主材料搬运、结构装配、巡检等任务，减少工人劳动强度，提高安全性。系统依赖于工人示范视频，适合在已有监控或摄像系统的工地部署。未来可结合多模态传感器，实现更复杂的任务自主学习，推动建筑行业智能化升级。
Limitations & Outlook
模型在极端复杂环境和未见动作类型下表现仍有限，受限于训练数据的多样性。实时性方面，当前系统计算成本较高，需优化模型结构以满足现场快速响应需求。未来还需增强模型的鲁棒性和适应性，以应对多变的施工环境和突发情况。
Plain Language Accessible to non-experts
想象你在厨房里做菜。每次做菜，你会观察老师怎么切菜、炒菜，然后自己模仿。机器人也是这样学习的：它看人怎么做，然后试着自己做。这个系统就像一个聪明的厨师助手，能从你的视频中学会各种烹饪技巧，然后用自己的“手”去完成这些动作。它会记住每个步骤，确保动作稳当，不会掉东西或摔倒。这样一来，机器人就能帮忙做繁重或危险的工作，比如搬重物或在高处作业，就像一个懂事的帮手一样。
ELI14 Explained like you're 14
你知道吗，就像你在学校学运动会的跳远动作，老师示范一次，你就模仿，慢慢变得更厉害。机器人也是一样，它通过看人怎么做，然后学会了这些动作。这个系统就像一个超级聪明的朋友，能从工人的视频里学会搬东西、堆砖、抬箱子这些建筑工作。它用一种特别的“眼睛”看清楚每个动作，然后用“脑袋”把动作变成自己的动作，确保自己不会摔倒或出错。这样，机器人就可以帮忙做一些危险或繁重的工作，让工人更安全、更轻松。是不是很酷？未来，这样的机器人还能自己学会更多复杂的任务，变成工地上的超级助手！
Abstract
Humanoid robots, with their human-like shape and multi-tasking capabilities, are well-aligned with human-dominated workplaces, like those in civil and construction engineering, where they could collaborate with human workers or autonomously perform physically demanding and hazardous tasks. Despite this promise, limited research has explored how to endow these robots with the practical capabilities needed to perform construction tasks. To this end, this study proposes a novel perception-and-action system that enables humanoid robots to learn and perform construction tasks from worker demonstrations. This system contains two deep networks: Humanoid-PoseNet, which extracts human postures and translates them into mechanically feasible poses for a humanoid robot; and Humanoid-ActionNet, which learns robot-executable actions based on these translated poses. Experimental results demonstrate that the humanoid robot reliably executed eight construction-related actions, achieving an average motion-tracking error of 82.45 mm MPJPE (Mean Per Joint Position Error). This work provides an early step toward deploying humanoid collaborators in construction.