Reconstructing Humans and Objects in Interaction using Large Reconstruction Models
This paper introduces MILO, leveraging Large Reconstruction Models (LRMs) to reconstruct detailed 3D human-object interactions from a single image, outperforming state-of-the-art methods.
Key Findings
Methodology
The core of MILO involves using pre-trained LRMs (e.g., Hunyuan3D-2.0) to generate holistic 3D meshes capturing human-object scenes. The process starts with inputting a single RGB image into the LRM, producing a non-parametric scene mesh. Multi-view rendering and keypoint detection (via ViTPose and HaMeR) estimate 3D human keypoints, which are then fitted with the SMPL-H parametric model. Simultaneously, multi-view segmentation extracts object point clouds, optionally aligned with object templates. This approach circumvents reliance on explicit contact labels or object templates by exploiting the geometric scaffold provided by LRMs, simplifying complex geometric inference.
Key Results
- On datasets like InterCap, HODome, and IMHD, MILO achieves superior Procrustes-aligned Chamfer distances, reducing average errors to 6.85cm, outperforming previous methods such as PICO (13.12cm). This demonstrates significant accuracy gains and robustness across diverse scenarios.
- The method performs well without contact supervision, showing strong generalization to in-the-wild images and complex interactions. Quantitative comparisons confirm its effectiveness over existing approaches, especially in detailed geometry and pose accuracy.
- Ablation studies highlight the importance of multi-view keypoints and template alignment, which substantially improve reconstruction quality, validating the design choices.
Significance
This work advances single-image 3D human-object interaction reconstruction by harnessing the geometric power of large-scale models. It addresses longstanding issues like dependence on contact labels and object templates, enabling more scalable and general solutions. Its high accuracy and robustness open new possibilities for applications in AR/VR, robotics, and embodied AI, fostering more realistic virtual environments and intelligent agents. The approach demonstrates that pre-trained large models can serve as powerful geometric priors, transforming traditional optimization-based pipelines into end-to-end learning-free frameworks that leverage existing scene understanding capabilities.
Technical Contribution
The main technical innovation is repurposing LRMs as geometric scaffolds, transforming the problem from explicit geometric optimization to mesh interpretation. The framework integrates multi-view rendering, keypoint triangulation, parametric model fitting, and semantic segmentation, with optional template alignment via semantic correspondences. This combination yields high-fidelity 3D reconstructions without relying on contact annotations or object templates, representing a paradigm shift in monocular HOI reconstruction. The method also introduces a robust multi-view segmentation and alignment pipeline that can adapt to various object categories and interaction types.
Novelty
This is the first work to systematically leverage large-scale reconstruction models as geometric priors for single-image 3D human-object interaction reconstruction. Unlike prior methods that depend heavily on contact cues, templates, or multi-stage optimization, MILO directly interprets the LRM-generated mesh, simplifying the pipeline and improving accuracy. Its ability to produce detailed, scene-consistent 3D reconstructions without explicit contact supervision marks a significant innovation in the field.
Limitations
- The approach depends heavily on the quality and generalization of the pre-trained LRM; hallucinations or inaccuracies in the mesh, especially under severe occlusion, can affect results.
- Current models struggle with highly complex multi-person scenes or dynamic interactions, as they are primarily designed for static single images.
- Computational costs are relatively high due to multi-view rendering and segmentation, limiting real-time applications. Future work should focus on efficiency and robustness improvements.
Future Work
Future directions include integrating temporal information from videos to improve dynamic scene understanding, extending the framework to multi-person scenarios, and reducing computational overhead. Exploring self-supervised learning to reduce reliance on large pre-trained models and enhancing robustness under occlusion and clutter are also promising avenues. Additionally, combining this approach with multi-view setups could further enhance accuracy and scene completeness.
AI Executive Summary
Reconstructing detailed 3D human-object interactions from a single image remains a significant challenge in computer vision, hindered by depth ambiguities, occlusions, and shape variability. Traditional methods often rely on parametric human models like SMPL and object templates, fitting these to 2D cues, but struggle in complex, real-world scenarios. This paper introduces MILO, a novel framework that leverages the geometric capabilities of Large Reconstruction Models (LRMs) such as Hunyuan3D-2.0, to directly generate holistic 3D meshes capturing human-object scenes. The key insight is that LRMs provide a powerful geometric scaffold that preserves the relative arrangement and proximity cues between humans and objects, simplifying the reconstruction process. The approach involves passing a single RGB image through an LRM to obtain a combined scene mesh, then performing multi-view rendering and keypoint triangulation to estimate human pose via SMPL-H fitting, and segmenting the mesh to extract object point clouds. Optional template alignment further refines the object geometry. Extensive experiments on datasets like InterCap, HODome, and IMHD demonstrate that MILO outperforms existing methods, achieving an average Procrustes distance of 6.85cm, a substantial improvement over prior approaches. Importantly, MILO does not require explicit contact annotations or object templates, making it more scalable and adaptable to in-the-wild scenes. This work signifies a paradigm shift, showing that large-scale pre-trained models can serve as geometric priors, enabling more accurate and robust 3D scene understanding from minimal input. The potential applications span immersive AR/VR, robotics, and embodied AI, with future work focusing on dynamic scenes, multi-person interactions, and efficiency enhancements.
Deep Analysis
Background
近年来，单幅图像的3D人体重建取得了显著进展，代表性方法如SMPL系列模型（SMPL、SMPL-X）在人体结构建模中表现优异。然而，物体重建因缺乏统一参数化模型，面临形状多样性和场景复杂性挑战。深度学习驱动的单视角物体重建（如NeRF、Gaussian Splatting）虽取得突破，但在多类别、多场景交互中仍受限。大规模重建模型（LRMs）如Hunyuan3D-2.0，经过海量数据训练，具备强大场景理解和几何推断能力，为复杂场景的3D重建提供了新工具。尽管如此，如何将LRM的几何信息融入人-物交互重建，仍是研究热点。
Core Problem
单幅图像中实现高精度的人-物交互3D重建，面临深度模糊、遮挡和形状多样性等难题。传统方法依赖接触标签或对象模板，限制泛化能力。复杂场景中的几何推断困难，缺乏有效的端到端方案。如何在无需大量标注的情况下，准确捕获空间关系，成为核心难题。
Innovation
本研究创新在于利用LRM作为几何支架，将人-物交互重建问题转变为mesh解释任务。具体包括：1）利用预训练的LRM生成场景mesh，捕获复杂关系；2）多视角渲染与关键点检测，估算人体关键点并拟合SMPL-H模型；3）多视图分割提取物体点云，支持无模板或模板对齐。该方案突破了依赖接触和模板的限制，显著提升重建精度和泛化能力。
Methodology
- �� 输入单幅RGB图像，传入LRM（如Hunyuan3D-2.0）获得场景mesh。• 多视角渲染mesh，利用ViTPose和HaMeR检测2D关键点，三角化获得3D关键点。• 初始用HMR2.0和HaMeR参数，进行两阶段SMPL-H拟合：根部位置调整和姿态优化，确保模型与mesh一致。• 多视图分割模型提取人和物体点云，支持模板对齐（可选）。• 利用语义对应关系进行模板与mesh匹配，细化几何。• 最终输出高精度人体模型和物体几何，支持多场景应用。
Experiments
在InterCap、HODome和IMHD等公开数据集上，采用Procrustes距离评估，比较MILO与现有方法。模型参数通过交叉验证优化，利用多视角渲染和关键点检测验证鲁棒性。消融实验分析多视角信息和模板对齐的贡献，结果显示MILO在所有指标上均优于对比方法，平均误差降至6.85cm，验证其优越几何重建能力。
Results
在多个数据集上，MILO实现了平均Procrustes距离的显著降低，优于PICO等方法。无需接触标签，模型在复杂交互场景中表现出更强鲁棒性。消融实验验证多视角关键点和模板对齐的重要性，显示出模型在细节和整体结构上的优势。
Applications
该方法适用于虚拟现实、机器人交互、虚拟试衣和增强现实等场景，尤其在需要高精度几何理解的应用中。无需大量标注，降低部署门槛，适应多样场景。未来结合多视角和动态信息，将推动实时交互和场景理解的发展。
Limitations & Outlook
模型在极端遮挡或复杂场景中仍存在hallucination风险，LRM生成mesh可能偏离真实几何。对预训练模型依赖较大，泛化能力有限。多人体和动态场景仍需改进，计算成本较高，未来需优化效率和鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房准备一顿大餐。你有各种食材（人和物体），每样都要放在合适的位置。以前，你用眼睛和手一一测量，确保每样都放得恰当。现在，有个神奇的机器人，它能一眼看出厨房里的所有食材，知道它们的具体位置和形状，还能帮你把它们摆得整齐。这台机器人就像论文里的大模型（LRM），它能用一张照片，帮你画出厨房里所有食材的3D模型。你不用担心遮挡或复杂场景，只要让它看一眼，它就能告诉你每个食材在哪里，形状如何，甚至还能帮你把厨房里的物品按比例摆放。这样，你可以更快、更准地准备饭菜，甚至可以在虚拟现实中提前预览厨房。这种技术未来可以用在虚拟试衣、机器人操作，甚至游戏设计，让虚拟世界变得更真实、更智能。
ELI14 Explained like you're 14
想象你在玩一个超级酷的游戏，你的任务是用相机拍一张房间的照片，然后让游戏里的机器人帮你画出房间里所有东西的3D模型，比如桌子、椅子、书架等等。以前，这很难，因为每个物品的形状都不同，还可能被遮挡或看不清楚。现在，这篇论文介绍了一种新方法，就像给机器人装上了“超级大脑”。这个大脑叫LRM，它经过很多数据训练，能从一张照片里快速“猜出”所有物品的3D形状和位置。它还能帮你把人体的姿势也画出来，就像你在虚拟世界里站着一样。这个方法不用事先知道物品的样子，也不用标记接触点，只需要一张照片，就能得到完整的3D场景。未来，这项技术可以让虚拟现实、机器人和游戏变得更真实、更智能，就像你用魔法一样！
Abstract
Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction procedure, reframing the problem as interpreting the LRM mesh: we segment it into human and object components, fit a parametric body model to the human part, and optionally align an object template to the object part (if such a template is available). MILO achieves strong reconstruction accuracy and outperforms existing baselines across multiple benchmarks and interaction scenarios. Our code is available at https://ac5113.github.io/MILO.