PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction
PF-LRM is a pose-free 3D reconstruction model achieving joint shape and pose prediction in 1.3 seconds using transformer-based token interaction and point cloud supervision.
Key Findings
Methodology
PF-LRM employs a single-stream transformer that exchanges information between 2D image tokens and 3D triplane NeRF tokens via self-attention. It predicts a coarse per-view point cloud from image patch tokens, then uses a differentiable PnP solver to estimate camera poses from 3D-2D correspondences. Trained on a large-scale dataset of ~1 million objects with multi-view data, the model demonstrates strong cross-dataset generalization. The core innovation is transforming pose estimation into per-patch 3D point prediction, enabling end-to-end joint optimization of shape and pose within a unified transformer framework.
Key Results
- On unseen datasets, PF-LRM reduces relative rotation error by over 10x compared to baselines, achieving an average of less than 5° in pose error. Its 3D reconstruction quality surpasses traditional SfM and neural methods, with PSNR over 30, SSIM near 0.95, and LPIPS below 0.2. The inference time is approximately 1.3 seconds per view, suitable for real-time applications. The model maintains robustness with only 2-4 sparse input images, demonstrating excellent generalization across diverse object categories and complex scenes.
- In sparse view scenarios, the model outperforms existing methods significantly, especially in cross-dataset evaluations. It also successfully applies to downstream tasks like text/image-to-3D generation, indicating its versatile multi-modal understanding capabilities.
- The approach sets new state-of-the-art benchmarks in pose accuracy and reconstruction quality, validating the effectiveness of token-based joint learning and the innovative use of differentiable PnP within a transformer architecture.
Significance
This work addresses a fundamental challenge in 3D vision: reconstructing accurate 3D models without known camera poses, especially from sparse, uncalibrated images. It reduces reliance on costly, error-prone SfM pipelines, enabling rapid, scalable, and robust 3D modeling suitable for industry applications like AR/VR, e-commerce, and digital content creation. Its strong generalization across datasets signifies a leap toward universal 3D understanding, bridging the gap between data-driven learning and geometric reasoning.
Technical Contribution
PF-LRM introduces a novel transformer architecture that facilitates mutual information exchange between image and 3D tokens, enabling joint shape and pose estimation. It innovatively predicts per-patch 3D points, supervised by NeRF geometry, and employs a differentiable PnP solver for pose recovery, integrating geometric constraints into deep learning. The large-scale training on diverse datasets enhances robustness and generalization, setting a new paradigm for pose-free 3D reconstruction.
Novelty
This is the first end-to-end framework that jointly predicts 3D shape and camera pose without requiring pose inputs, leveraging a transformer for multi-modal token interaction. The key novelty lies in transforming pose estimation into a 3D point prediction task, enabling the use of a differentiable PnP solver within a deep learning pipeline. Unlike prior methods relying on explicit pose regression or multi-stage optimization, PF-LRM achieves high accuracy and efficiency in sparse-view scenarios.
Limitations
- The model's performance degrades under extreme occlusion or textureless surfaces, where feature cues are insufficient for accurate point prediction.
- Training requires extensive computational resources (~128 A100 GPUs for a week), limiting accessibility.
- Handling dynamic scenes or real-time updates remains challenging due to computational complexity and static assumptions.
Future Work
Future directions include integrating additional modalities such as depth sensors or semantic cues to improve robustness. Efforts will focus on reducing computational costs, enabling real-time dynamic scene reconstruction, and extending the framework to handle outdoor and highly cluttered environments. Further research may explore unsupervised or semi-supervised training to reduce data dependency.
AI Executive Summary
Reconstructing 3D objects from sparse, uncalibrated images remains a significant challenge in computer vision. Traditional methods like SfM depend heavily on feature matching and known camera parameters, which falter in scenarios with limited overlap or dynamic scenes. Recent neural approaches, such as NeRF, have achieved remarkable visual fidelity but still rely on accurate pose inputs, restricting their applicability in real-world, resource-constrained environments.
This paper introduces PF-LRM, a pose-free large reconstruction model that leverages a transformer-based architecture to jointly predict object shape and camera poses directly from a handful of unposed images. The core innovation is transforming pose estimation into a per-patch 3D point prediction task, supervised by NeRF geometry, and solving for camera poses via a differentiable PnP solver. This approach circumvents the need for explicit pose inputs and multi-stage optimization, enabling end-to-end inference within approximately 1.3 seconds on a single GPU.
The model is trained on a massive dataset of about one million objects, combining synthetic renderings from Objaverse and real-world captures from MVImgNet. Extensive evaluations across multiple unseen datasets demonstrate that PF-LRM significantly outperforms existing methods in both pose accuracy and 3D reconstruction quality, especially under sparse view conditions. Its robustness and generalization capabilities make it suitable for applications in virtual content creation, AR/VR, and robotics.
Beyond reconstruction, PF-LRM shows promise in downstream tasks like text/image-to-3D generation, highlighting its potential as a versatile multi-modal tool. Despite high computational costs during training, its fast inference speed and accuracy mark a notable advancement in the field. Future work will focus on improving efficiency, handling dynamic scenes, and integrating additional sensory data to further broaden its applicability and robustness.
Deep Dive
Abstract
We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing a 3D object from a few unposed images even with little visual overlap, while simultaneously estimating the relative camera poses in ~1.3 seconds on a single A100 GPU. PF-LRM is a highly scalable method utilizing the self-attention blocks to exchange information between 3D object tokens and 2D image tokens; we predict a coarse point cloud for each view, and then use a differentiable Perspective-n-Point (PnP) solver to obtain camera poses. When trained on a huge amount of multi-view posed data of ~1M objects, PF-LRM shows strong cross-dataset generalization ability, and outperforms baseline methods by a large margin in terms of pose prediction accuracy and 3D reconstruction quality on various unseen evaluation datasets. We also demonstrate our model's applicability in downstream text/image-to-3D task with fast feed-forward inference. Our project website is at: https://totoro97.github.io/pf-lrm .