CoGS: Compositional Dynamic Human-Object Scenes Gaussian Splatting from Monocular Video
CoGS introduces a compositional Gaussian splatting framework with a six-stage optimization for monocular human-object scene reconstruction, achieving state-of-the-art fidelity.
Key Findings
Methodology
CoGS employs a three-branch structure: a SMPL-based articulated human branch, a rigid object trajectory branch, and a static scene branch. The six-stage optimization stabilizes each component independently before fusion, using visibility-aware human anchoring, silhouette and motion constraints, and delayed scene regularization. Each branch models its geometry and motion separately, with a differentiable Gaussian point cloud renderer for scene synthesis. Preprocessing includes COLMAP camera calibration, monocular SMPL pose estimation, SAM masks, object trajectory initialization, and scene plane primitives, ensuring multi-modal data alignment.
Key Results
- On HOSNeRF and NeuMan datasets, CoGS outperforms baselines with PSNR reaching 25.34 (2.3% improvement) and LPIPS dropping to 0.03 (44% better). It excels in occlusion, fast motion, and weakly observed regions, demonstrating robustness and detailed reconstruction.
- In human-object interaction and in-the-wild scene reconstruction, CoGS achieves higher fidelity, preserving fine details and realistic motion, validated through quantitative metrics and visual comparisons.
- Ablation studies confirm that the six-stage curriculum and visibility-guided human anchoring are critical for avoiding component interference and enhancing overall quality.
Significance
This work advances monocular dynamic scene reconstruction by effectively disentangling multiple motion models, addressing occlusion and weak observation issues. It significantly improves the realism and detail of reconstructed scenes, impacting virtual reality, AR, and film industries by enabling more accurate and immersive digital twins from minimal input data.
Technical Contribution
The paper introduces a multi-branch Gaussian point cloud representation combined with a staged optimization process, incorporating visibility-aware anchoring and delayed densification. These innovations enable stable, high-fidelity reconstruction of complex human-object scenes from monocular videos, overcoming traditional limitations of entanglement and underconstrained supervision.
Novelty
This is the first approach to explicitly decompose monocular scene reconstruction into separate human, object, and static scene branches with dedicated motion models, integrated through a staged optimization curriculum. It departs from prior monolithic models by emphasizing component independence and targeted regularization, leading to superior detail and robustness.
Limitations
- The method struggles with extreme occlusion and rapid motion due to limited depth cues from monocular input, affecting reconstruction accuracy.
- Computational complexity remains high, limiting real-time applications; further optimization is needed.
- Heavy reliance on preprocessing and initializations may introduce errors, especially in noisy or ambiguous scenes, requiring future robustness improvements.
Future Work
Future directions include integrating multi-view or depth sensors to improve occlusion handling, optimizing algorithms for real-time deployment, and extending the framework to larger, more complex environments. Enhancing robustness against noisy inputs and exploring unsupervised or weakly supervised training are also promising avenues.
AI Executive Summary
Reconstructing dynamic human-object scenes from monocular videos remains a formidable challenge in computer vision, primarily due to the entanglement of different motion models and limited observation. Existing neural rendering methods often struggle to disentangle articulated human motion, rigid object trajectories, and static backgrounds, leading to blurred details and inconsistent motion. To address these issues, this paper introduces CoGS, a novel framework that decomposes the scene into three dedicated Gaussian branches: an articulated human, a rigid object, and a static scene. The core innovation lies in a six-stage optimization schedule that stabilizes each component independently, then gradually fuses them into a coherent scene. Visibility-aware human anchoring ensures that weakly observed regions are completed plausibly, while delayed scene regularization prevents premature overfitting. Extensive experiments on HOSNeRF and NeuMan datasets demonstrate that CoGS surpasses state-of-the-art methods in both quantitative metrics and visual fidelity, achieving PSNR of 25.34 and LPIPS of 0.03, with particular strength in occlusion-heavy and fast-moving scenarios. This approach significantly advances the field of monocular scene reconstruction, opening new possibilities for immersive virtual environments, film production, and AR applications. Despite its success, challenges remain in handling extreme occlusion and reducing computational costs for real-time use. Future work will focus on multi-modal data integration, algorithm acceleration, and broader scene scalability, aiming to bring high-fidelity dynamic scene understanding closer to practical deployment.
Deep Analysis
Background
随着NeRF等神经辐射场技术的发展，场景重建已取得巨大进步，但多视角场景的高质量渲染在单目视频中仍面临挑战。早期方法如Neural Body、HumanNeRF利用人体模型引导重建，但在动态交互和遮挡复杂场景中表现不足。Gaussian Splatting引入高斯点云实现高效渲染，但在单目动态场景中，运动模型混叠和观察不足限制了其应用。近年来，场景分解和多支路策略逐渐成为主流，旨在解耦不同运动模型，提升重建细节。本文在此基础上，结合多阶段优化和可视性引导，提出更鲁棒、细节丰富的重建框架，解决单目视频中多运动组件的解耦难题。
Core Problem
单目视频中的动态人-物场景重建面临多重难题：不同运动模型的组件（人体、物体、背景）共享像素，导致运动信息混叠；观察不足和遮挡限制几何推断；缺乏有效的组件解耦机制，影响细节还原。现有方法难以在复杂交互场景中实现高保真还原，亟需一种能区分运动模型、稳定推断未观察区域的解决方案。这些问题限制了单目场景重建的真实性和细节表现。
Innovation
创新点包括：1）多支路高斯点云模型，分别建模人体、物体和静态场景，避免运动模型混叠；2）六阶段优化策略，逐步稳定组件，减少干扰；3）引入可视性引导的人体锚定机制，增强未观察区域几何推断；4）延迟密度正则化，防止早期过拟合，提升模型稳定性。这些创新共同实现了高质量、鲁棒的单目动态场景重建。
Methodology
- �� 预处理：COLMAP相机校准、SMPL运动估计、SAM掩码、物体轨迹初始化、场景平面原语。• 三支路模型：人体支路基于SMPL模型的高斯集，利用可视性引导进行几何修正；物体支路采用关键帧插值保持几何紧凑；场景支路从COLMAP重建，结合平面原语增强低纹理区域。• 六阶段优化：包括基础拟合、轮廓细化、物体姿态和外观优化、全局融合、细节修正和调优。• 损失函数：结合RGB、轮廓、运动、可视性和正则化，逐步引导模型收敛。• 组件融合：在稳定后，将人体、物体和场景合成为完整场景，支持高质量渲染。
Experiments
采用HOSNeRF和NeuMan两个数据集，前者涵盖多场景人-物交互，后者为野外场景中的人体重建。训练采用六阶段策略，指标包括PSNR、LPIPS等。对比多种基线，验证了CoGS在细节、运动和遮挡鲁棒性方面的优越性。消融研究显示，六阶段优化和可视性引导是性能提升的关键。
Results
在HOSNeRF上，PSNR最高达25.34，LPIPS最低至0.03，优于对比方法2.3%的PSNR提升和44%的LPIPS降低。在NeuMan数据集上，整体性能显著提升，尤其在遮挡和动态交互场景中表现出更强鲁棒性。消融实验验证了多支路设计和分阶段优化的有效性，显示模型在复杂场景中的优越表现。
Applications
该方法适用于虚拟现实、影视特效、虚拟试衣等场景，能实现高质量动态场景重建，特别在单目设备条件下。未来结合多模态信息，拓展到实时交互和大规模场景，推动行业应用落地。
Limitations & Outlook
模型在极端遮挡和高速运动场景下仍有限制，主要因单目深度信息不足。优化过程复杂，计算成本高，实时性待提升。对预处理依赖强，噪声和误差影响效果。未来需增强鲁棒性和效率，拓展应用场景。
Plain Language Accessible to non-experts
想象你在看一场舞台剧，舞台上有演员、道具和背景。传统的方法就像用一张照片试图捕捉全部细节，但演员的动作、道具的变化和背景的静止都混在一起，难以区分。本文的方法像是给每个元素都配备了不同的“标签”和“追踪器”，让每个部分都能独立运动和变化。即使演员遮挡了部分场景，也能准确还原每个角色的动作和道具的变化。通过逐步调整和校准，最终可以在一段录像中清晰地看到每个元素的细节，甚至在复杂的互动中也能还原得栩栩如生。这就像给舞台上的每个角色都装上了“隐形眼镜”，让整体效果变得更真实、更细腻。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，有很多不同的拼图片：有人、物品和背景。每个拼图片都在动，有的人在跳舞，物品在移动，背景也在变化。以前的方法就像是把所有拼图片都拼在一起，但这样很难看清每个细节，尤其当拼图片遮挡或模糊时。现在，这个新方法像是给每个拼图片都装上了专门的追踪器，让它们可以自己动，不会被混淆。这样，即使有人遮挡了部分拼图，我们还是能知道每个拼图片原本应该长什么样，怎么动。这个过程一步步进行，先让每个拼图片自己稳定下来，再把它们拼在一起，最后得到一个完整又细腻的拼图。就像用魔法一样，把复杂的场景变得清晰、真实，像电影里的特效一样酷！
Abstract
Reconstructing dynamic human--object interaction scenes from monocular video is difficult because the human, manipulated object, and background obey different motion models while sharing the same pixels. Existing dynamic radiance-field and Gaussian-splatting methods often entangle these components, causing object motion to leak into the human or static scene, and monocular human reconstruction remains underconstrained in regions that are rarely observed. We present CoGS, a compositional Gaussian-splatting framework for monocular human--object scene reconstruction. CoGS decomposes the video into three coordinated branches: an articulated human initialized from a complete canonical prior, a rigid object field driven by an estimated object trajectory, and a static scene field regularized by weak scene-only planar primitives when available. A six-stage optimization schedule first stabilizes the human and object independently, then fuses them with the scene under full-image supervision, visibility-aware human anchoring, object silhouette and motion constraints, and delayed scene regularization. This design keeps each component responsible for its own geometry and motion while allowing photometric evidence to correct the final composite. Experiments on HOSNeRF and NeuMan show that CoGS improves both human--object interaction reconstruction and in-the-wild human--scene rendering, achieving stronger fidelity and perceptual quality across full-frame and human-focused evaluations. Code will be released upon publication.