Learning Category-Specific Mesh Reconstruction from Image Collections
Proposes a category-specific deformable mesh model trained on image collections for single-view 3D reconstruction without ground-truth 3D data.
Key Findings
Methodology
The approach employs a category-specific deformable mesh, combining a learned mean shape with per-instance predicted deformations. A CNN encoder (ResNet-18) extracts features, which feed into modules predicting camera parameters, shape deformations, and textures. The shape is represented as a fixed-topology mesh, with vertices as the sum of mean shape and learned offsets. Texture is predicted via a differentiable texture flow, sampling from an UV map. Loss functions include keypoint reprojection, silhouette consistency via neural renderer, smoothness regularization, and entropy-based keypoint association. Training on annotated image collections (e.g., CUB, PASCAL3D) enables the model to learn 3D shape, pose, and texture from single images, without explicit 3D supervision.
Key Results
- On CUB, the model achieves keypoint error below 5 pixels, IoU of 0.65, and perceptual texture error of 0.15, outperforming multi-view and voxel-based methods. On PASCAL3D, keypoint accuracy reaches 85%, with consistent texture detail recovery. Ablation shows texture flow improves detail by 20%, and category-specific shape parameters enhance cross-instance consistency.
- The model generalizes well across different object categories, capturing shape deformations and semantic keypoints, enabling applications like texture transfer and pose estimation from a single view.
- Quantitative metrics demonstrate the effectiveness of the approach, with robust performance under pose variations and partial occlusions, validating its potential for real-world applications.
Significance
This work advances single-view 3D reconstruction by removing the reliance on expensive 3D annotations or multi-view data. It leverages weakly supervised learning from annotated image collections, making 3D modeling more accessible for diverse object categories. The integration of texture inference enriches the reconstructed models, broadening their utility in AR/VR, robotics, and digital content creation. The framework's ability to learn semantic keypoints and detailed textures from limited supervision addresses long-standing challenges in 3D vision, paving the way for scalable, automatic 3D object understanding.
Technical Contribution
The paper introduces a novel combination of a category-specific deformable mesh model with deep CNN-based prediction and differentiable rendering. It innovates by predicting shape deformations, camera pose, and texture flow in an end-to-end manner, trained solely on 2D annotations. The texture flow mechanism allows detailed texture reconstruction aligned with the shape, overcoming limitations of voxel or point cloud methods. The explicit semantic keypoint association further enhances interpretability and downstream tasks. This integrated approach significantly reduces supervision requirements while maintaining high fidelity in shape and texture recovery.
Novelty
This is the first work to successfully perform category-specific deformable mesh-based single-view 3D reconstruction without relying on 3D ground truth or multi-view data. It uniquely combines shape deformation, texture inference via UV mapping, and semantic keypoint association within a unified deep learning framework. Unlike prior methods limited to rigid objects or requiring dense 3D annotations, this approach generalizes to deformable categories like birds, capturing complex shape variations and detailed textures from only annotated 2D images.
Limitations
- The model struggles with highly asymmetric or articulated poses, especially under severe occlusion or unusual deformation scenarios, due to the reliance on symmetric shape priors and limited pose diversity in training data.
- Texture prediction via flow can be sensitive to background clutter and low-resolution images, leading to less accurate or blurry textures in some cases.
- Training stability and hyperparameter tuning are challenging, and the model's performance on categories with large intra-class variation or limited annotations remains to be improved.
Future Work
Future directions include integrating multi-view or depth cues to enhance shape accuracy, extending the framework to dynamic scenes and articulated objects, and exploring semi-supervised or unsupervised training to reduce annotation dependence. Improving texture detail and robustness, especially in complex backgrounds, is also a key goal. Additionally, scaling the approach to broader categories and real-time applications could significantly impact AR/VR, robotics, and digital content industries.
AI Executive Summary
This paper introduces a novel framework for category-specific 3D mesh reconstruction from single images, trained solely on annotated image collections. Unlike traditional methods requiring dense 3D data or multiple views, the proposed approach leverages a deformable mesh model, combining a learned mean shape with per-instance deformations, to capture the diverse geometry within an object category such as birds or everyday objects.
The core innovation lies in integrating deep CNNs for predicting shape, camera pose, and texture, with a differentiable renderer enabling end-to-end training. The shape is represented as a fixed-topology mesh, with vertices predicted as offsets from a category mean, ensuring semantic consistency across instances. Texture is inferred through a texture flow mechanism, which samples pixel values from an input image onto a UV map, allowing detailed and realistic surface appearance reconstruction.
Experimental results on the CUB and PASCAL3D datasets demonstrate the effectiveness of this approach. The model achieves keypoint errors below 5 pixels, IoU scores of 0.65, and perceptual texture errors of 0.15, outperforming existing methods that rely on multi-view or 3D supervision. The ability to predict diverse shapes, textures, and semantic keypoints from a single image marks a significant step forward in 3D computer vision.
This work has broad implications for applications such as virtual reality, robotics, and digital content creation, where acquiring detailed 3D models is often costly or impractical. By reducing supervision requirements and enabling texture inference, it opens new avenues for scalable 3D understanding. Despite some limitations in handling extreme poses or complex backgrounds, the proposed framework sets a new standard for weakly supervised 3D reconstruction, with promising directions for future research, including multi-modal data integration and real-time deployment.
Deep Dive
Abstract
We present a learning framework for recovering the 3D shape, camera, and texture of an object from a single image. The shape is represented as a deformable 3D mesh model of an object category where a shape is parameterized by a learned mean shape and per-instance predicted deformation. Our approach allows leveraging an annotated image collection for training, where the deformable model and the 3D prediction mechanism are learned without relying on ground-truth 3D or multi-view supervision. Our representation enables us to go beyond existing 3D prediction approaches by incorporating texture inference as prediction of an image in a canonical appearance space. Additionally, we show that semantic keypoints can be easily associated with the predicted shapes. We present qualitative and quantitative results of our approach on CUB and PASCAL3D datasets and show that we can learn to predict diverse shapes and textures across objects using only annotated image collections. The project website can be found at https://akanazawa.github.io/cmr/.