Muskie: Multi-view Masked Image Modeling for 3D Vision Pre-training
Muskie learns multi-view geometry through masked completion, reaching 2.38 cm 3D ATE on NAVI.
Key Findings
Methodology
Muskie extends Masked Image Modeling to multi-view scenes. It receives 2–8 images, applies high-ratio random or contiguous rectangular/elliptical masks, and makes single-frame reconstruction difficult. VGGT-style Alternating Attention alternates intra-view and global cross-view attention, while RoPE improves resolution flexibility. A lightweight linear head predicts both masked pixels and confidence.
Key Results
- On zero-shot NAVI correspondence, Muskie-L obtains 2.38 cm 3D ATE, about 36% lower than DINOv3's 3.76 cm. Its 3D Acc@2 cm is 82.74%, compared with 74.89% for DINOv3.
- On 7Scenes, replacing the π3 encoder with Muskie-L raises pose AUC@30 from 8.514 with DINOv2-L to 47.345. Pointmap L1 error falls from 0.074 to 0.035, demonstrating transfer to downstream reconstruction.
- Ablations show that pre-training matters more than architecture alone: average AUC@30 increases from 3.760 without pre-training to 37.413 with it. On NRGBD, the proposed masking reaches 11.83 cm 3D ATE.
Significance
The work addresses a persistent weakness of vision foundation models: strong semantic features do not guarantee multi-view consistency. DINO and MAE are predominantly trained on independent images, so cross-view agreement is not required. Muskie obtains 3D-aware representations from unlabeled multi-view images, offering a scalable backbone for reconstruction, pose estimation, robotics, and spatial computing without explicit 3D annotations.
Technical Contribution
Muskie's contributions are a multi-view completion objective, aggressive masking that removes single-frame shortcuts, and native joint encoding. Its confidence-aware loss is L=1/|M|Σ[(c+0.1)||p_hat-p||²−0.1log c]. Unlike CroCo's pairwise, independently encoded stereo setup, Muskie supports 2–8 symmetric views, permutation equivariance, Alternating Attention, and RoPE. The decoder is discarded after pre-training.
Novelty
The novelty is not merely applying MIM to more images; it makes cross-view correspondence necessary for solving the task. Large contiguous masks suppress texture-based guessing, while global Alternating Attention establishes interactions in one forward pass. This fundamentally differs from frame-wise DINO/MAE and CroCo's restricted dual-view design.
Limitations
- Muskie requires overlapping views of a largely static scene. Newly exposed, non-co-visible surfaces cannot be recovered from evidence; visualizations show blurry predictions and low confidence in such regions.
- Multi-view global attention and high-resolution training are expensive: Muskie-L requires about two weeks on eight A100 GPUs. Dynamic scenes, extreme baselines, long videos, and very large view counts are not systematically evaluated.
Future Work
Future work should address dynamic scenes, non-adjacent views, long video sequences, and online inference. Sparse or hierarchical cross-view attention could reduce cost, while calibrated confidence could improve reliability. Optional depth, pose, or sparse-geometry supervision may help difficult cases, but large-scale evaluation on robotics, autonomous driving, and open-world data remains necessary.
AI Executive Summary
Recovering 3D structure from photographs requires more than recognizing objects: a system must decide whether observations from different viewpoints refer to the same physical point. DINO, MAE, and related vision foundation models usually encode images independently, so their features may be semantically strong yet geometrically inconsistent. Muskie addresses this gap by making multi-view agreement part of pre-training itself.
Muskie takes several images of one scene and heavily masks their content. To reconstruct a hidden region, it must search other views for corresponding evidence. VGGT-style Alternating Attention alternates within-view and cross-view information exchange, while contiguous masks prevent easy texture interpolation. A confidence-aware L2 objective lets the model express uncertainty when a surface is not visible elsewhere.
The results are substantial. Muskie-L reaches 2.38 cm 3D ATE on NAVI, about 36% better than DINOv3's 3.76 cm. In 7Scenes, a π3 model using Muskie-L achieves AUC@30 of 47.345 versus 8.514 with DINOv2-L, and reduces pointmap L1 error from 0.074 to 0.035. The method still depends on overlapping static views and is computationally costly, but it demonstrates that 3D-aware features can emerge without explicit 3D supervision.
Deep Analysis
Background
DINOv2, DINOv3, and MAE learn powerful dense features from large 2D image collections. Pri3D adds RGB-D geometric supervision, while CroCo adapts masked modeling to stereo pairs; these approaches either require 3D signals or remain limited in view configuration. Feed-forward systems such as VGGT and π3 achieve impressive reconstruction, but commonly rely on frame-wise DINO encoders whose pre-training does not enforce cross-view agreement.
Core Problem
Given V images of one scene, the model should reconstruct masked patches in one image using evidence from other views. This is difficult because viewpoint changes, occlusion, non-co-visible surfaces, and textureless regions invalidate simple appearance matching. If masks are sparse, conventional MIM can exploit local context and avoid learning geometry, creating the central shortcut Muskie is designed to eliminate.
Innovation
- �� Multi-view completion extends MIM from one image to jointly observed views.
- �� Aggressive masking uses high ratios and contiguous rectangles or ellipses, making single-view prediction unreliable.
- �� Native joint encoding uses Alternating Attention rather than independent ViTs.
- �� Symmetric processing treats every view equally and preserves permutation equivariance.
- �� RoPE supports mixed resolutions of 224, 384, and 512 pixels.
Methodology
- �� Input: split each image into N non-overlapping patches and mask set M; retain some reference views.
- �� Encoding: stacked Alternating Attention first models intra-view structure, then uses global query-key interactions across views to discover correspondence.
- �� Decoding: a lightweight linear head predicts masked pixels and confidence c; it is removed after pre-training.
- �� Objective: optimize the confidence-aware L2 loss with epsilon=lambda=0.1, reducing the influence of poorly observable regions.
- �� Training: use Co3Dv2, BlendedMVG, ARkitScenes, DL3DV, MegaDepth, ScanNet++, HyperSim, Waymo, and RealEstate10K; train with AdamW at 2×10^-4, cosine decay, warm-up, and 400 epochs.
Experiments
The evaluation covers zero-shot correspondence, pointmap reconstruction, and camera pose estimation. Correspondence uses eight-frame sequences from object-centric NAVI and indoor ScanNet, reporting 2D pixel and 3D centimeter errors. Baselines include MAE, CroCo, DINOv2, and DINOv3. For downstream evaluation, the authors replace π3's encoder and simplify its decoder from 36 to 4 layers, train on ARKitScenes, ScanNet++, and BlendedMVS, and test on 7Scenes and NRGBD. Ablations vary views, masking, and reference-frame count.
Results
Muskie-L achieves 2.38 cm 3D ATE and 82.74% Acc@2 cm on NAVI, versus DINOv3's 3.76 cm and 74.89%. Muskie-B also beats frame-wise baselines on ScanNet. On 7Scenes, Muskie-L produces pose AUC@30 of 47.345 versus 8.514 for DINOv2-L, and pointmap L1 error of 0.035 versus 0.074. Increasing context views raises NAVI 1 cm recall from 40.59% with two views to 47.36% with eight.
Applications
Muskie can serve as an encoder for multi-view reconstruction, pose estimation, robotic navigation, indoor mapping, and augmented reality. Deployment requires overlapping images of a scene and a suitable task head such as π3. Its consistent features are particularly valuable in repetitive or texture-poor environments, where independently extracted frame features often drift.
Limitations & Outlook
Non-co-visible surfaces remain fundamentally underconstrained and are predicted with blur and low confidence. Global cross-view attention and high-resolution inputs impose substantial memory and compute costs; Muskie-L needs roughly two weeks on eight A100 GPUs. Training and evaluation focus mainly on static scenes, leaving dynamic objects, extreme baselines, long videos, online latency, and very large view sets insufficiently tested. Sparse attention and better uncertainty calibration are natural next steps.
Plain Language Accessible to non-experts
Imagine a team of museum visitors. Each person photographs the same room from a different corner, but a large sticker covers part of every photograph. The team must reconstruct the hidden areas. One person alone can only guess from nearby details; together, they can compare photographs and realize that a table edge, painting, or statue appears from different angles.
Muskie trains a computer to work this way. It views several pictures at once, hides large regions, and learns to search the other pictures for matching evidence. It first understands each picture internally, then lets the pictures exchange information. Because the hidden regions are large, guessing from nearby pixels is not enough. If no other photograph shows a surface, the system produces a blurry answer and lowers its confidence instead of pretending to know.
This training makes the computer better at tracking the same physical point across viewpoints. On NAVI, Muskie-L reaches a 3D error of 2.38 centimeters, and it substantially improves pose and pointmap reconstruction on 7Scenes. The key idea is simple: learn not only what an image looks like, but how the same world remains consistent when seen from elsewhere.
ELI14 Explained like you're 14
Picture a treasure-hunt game where you and friends stand in different corners of a room. Everyone takes a photo, but a giant sticker hides part of each one. To uncover the missing piece, you cannot rely only on your own picture. You have to check your friends' photos and decide whether a desk corner, poster, or box is the same object seen from another angle.
That is basically Muskie! It looks at 2–8 photos of one scene and deliberately covers lots of content. Then the photos “talk” to one another through the model. It searches for matching places across views and uses them to rebuild what is hidden. It also gives each answer a confidence score: strong clues mean high confidence; a surface nobody photographed means low confidence.
Why does this matter? A computer should not only say “this is a chair.” It should know that the chair in several pictures is the same chair, with the same 3D position. On NAVI, Muskie-L gets a 3D error of 2.38 cm. On 7Scenes, its pose AUC@30 is 47.345, compared with 8.514 for DINOv2-L. That is a huge jump!
But it is not magic. Missing information cannot be invented, and processing many high-resolution photos needs powerful GPUs. Still, the idea could help robots map rooms, cars understand roads, and phones build 3D spaces. Pretty useful for a model that learns by playing hide-and-seek with images!
Glossary
Multi-view Masked Image Modeling
A self-supervised task that hides content across related images and reconstructs it using other viewpoints. Technically, it extends masked patch prediction so cross-view geometry becomes useful for minimizing reconstruction error.
Muskie's central pre-training objective.
Alternating Attention
A Transformer mechanism that alternates between attention within each image and attention across images. It separately captures local structure and cross-view correspondence.
The main joint-encoding mechanism in Muskie.
Aggressive masking
Masking with unusually high ratios and large contiguous regions. In plain terms, it removes easy clues; technically, it suppresses intra-view shortcuts and encourages cross-view matching.
Used to force geometric reasoning during pre-training.
Pointmap
A per-pixel prediction of 3D coordinates, representing scene geometry in image-aligned form. It can be converted into a reconstructed point cloud.
The main geometric output in the π3 downstream evaluation.
Average Trajectory Error
The mean distance between predicted and ground-truth point tracks, measured in pixels or centimeters. Lower values indicate more accurate correspondence.
Reported for zero-shot NAVI and ScanNet experiments.
Open Questions Unanswered questions from this research
- 1 Dynamic scenes remain unresolved: moving objects violate the static-scene assumption, so future systems need temporal modeling and motion separation.
- 2 Efficiency at larger view counts is unclear because global attention scales with image tokens and views; sparse or hierarchical mechanisms are needed.
- 3 Confidence calibration is not fully established, especially for reflective surfaces, extreme baselines, and severe occlusion.
Applications
Immediate Applications
Indoor 3D mapping
A robot or smartphone can capture overlapping images of a room and use Muskie inside π3-like reconstruction pipelines. The expected benefits are better pointmap completeness and camera poses, especially on repetitive walls, floors, and furniture.
Multi-camera robotic perception
Robots can process images from several cameras or nearby time steps with consistent features for localization, navigation, and scene stitching. Overlapping views are required, and model compression is needed for real-time latency.
Long-term Vision
Open-world 3D understanding
With extensions to dynamic video, street scenes, and cross-device imagery, Muskie could become a general 3D backbone for autonomous driving, AR maps, and embodied agents. Motion, scale variation, and computation remain major obstacles.
Abstract
We present Muskie, a native multi-view vision backbone designed for 3D vision tasks. Unlike existing models, which are frame-wise and exhibit limited multi-view consistency, Muskie is designed to process multiple views simultaneously and introduce multi-view consistency in pre-training stage. Muskie is trained to reconstruct heavily masked content in one view by finding and utilizing geometric correspondences from other views. Through this pretext task and our proposed aggressive masking strategy, the model implicitly to learn view-invariant features and develop strong geometric understanding without any 3D supervision. Compared with state-of-the-art frame-wise backbones such as DINO, Muskie achieves higher multi-view correspondence accuracy. Furthermore, we demonstrate that using Muskie as a backbone consistently enhances performance on downstream 3D tasks, including camera pose estimation and pointmap reconstruction. Codes are publicly available at https://leo-frank.github.io/Muskie/