Camera Relocalization by Computing Pairwise Relative Poses Using Convolutional Neural Network
A Siamese relative-pose CNN achieves 0.21 m and 9.30° mean median error on 7 Scenes while generalizing across scenes.
Key Findings
Methodology
The method uses a ResNet34-based Siamese CNN to estimate pairwise relative pose. Given an image pair, it predicts quaternion rotation and 3D translation using L=||Δtgt−Δt||²+β||Δrgt−Δr||². A shared branch produces 512-dimensional descriptors for dot-product retrieval. The top five database neighbors then provide relative-pose hypotheses. Two translation directions are triangulated, while a RANSAC scheme selects the location with the largest angular consensus. Rotation hypotheses follow ΔRj=RjᵀRq and are fused by a second consensus procedure.
Key Results
- On 7 Scenes, the method obtains mean median errors of 0.21 m and 9.30°, compared with PoseNet's 0.44 m and 10.4° and HourglassPose's 0.23 m and 9.53°. It is particularly competitive on Chess, Office, and Stairs.
- On the common-frame University dataset, the method achieves 1.58 m and 14.02°, improving over the ResNet34-Pose baseline at 2.88 m and 15.33°. Office improves from 2.76 m to 0.57 m, and Kitchen from 1.65 m to 0.70 m.
- For cross-dataset generalization, training only on University and testing directly on 7 Scenes yields 0.36 m and 18.38° average median error, indicating that relative geometric relations transfer better than scene-specific absolute coordinates.
Significance
The work addresses a central scalability problem in learned localization: absolute pose regressors encode a scene's coordinate frame and therefore tend to require scene-wise training. Relative pose is coordinate-frame independent, allowing one model to support multiple disjoint scenes and even scenes absent during training. This creates a useful bridge between deep representation learning, image retrieval, and geometric estimation, with implications for robotic navigation, augmented reality, and SLAM systems that must grow beyond a single mapped room.
Technical Contribution
The principal contribution is a complete retrieval-and-geometry pipeline rather than merely a new CNN. A shared-weight ResNet34 Siamese network learns pairwise rotation and translation. Known database poses convert relative directions into absolute query-location hypotheses: pairwise triangulation generates candidates, and a first RANSAC rejects inconsistent rays. Relative rotations independently generate query-orientation hypotheses, filtered by a second consensus process. The system requires neither training depth maps nor full database images in memory at inference, since compact descriptors and fast retrieval can be used.
Novelty
Compared with PoseNet, LSTM-Pose, and HourglassPose, which directly regress scene-dependent absolute pose, this paper makes pairwise relative camera pose the central learned quantity and reconstructs absolute 6DoF pose from known database cameras. The novelty lies less in the ResNet34 backbone than in combining transferable relational learning, retrieval, triangulation, and robust hypothesis fusion into one scalable localization design.
Limitations
- Although the network predicts translation, experiments mainly use normalized translation directions. Absolute position therefore depends on at least two geometrically useful neighbors; nearly parallel rays or poor viewpoint coverage make triangulation unstable.
- The pipeline fixes N=5 retrieved neighbors, so retrieval errors directly affect pose estimation. Retrieval is intentionally simple and the paper does not establish performance in large outdoor scenes, severe dynamic occlusion, or extreme appearance changes.
- The L2 quaternion/translation loss uses β=1 and a 20° inlier threshold. These empirical choices may require retuning across cameras, scales, and environments.
Future Work
The authors identify deeper analysis of image retrieval as future work and announce release of the code and University dataset. Natural extensions include learned neighbor selection, explicit scale recovery, multi-view graph optimization, uncertainty-aware pose fusion, and rotation-manifold losses. Large-scale outdoor, day-night, dynamic-scene, and approximate-nearest-neighbor evaluations are needed to test whether the method scales beyond challenging indoor benchmarks.
AI Executive Summary
Camera relocalization asks where a camera is and which way it faces. Classical SIFT, ORB, and SfM-based 2D–3D matching can fail under weak texture, illumination change, occlusion, and repetitive structure. CNN regressors such as PoseNet avoid explicit matching, but learn image-to-coordinate mappings tied to one scene's coordinate frame, making multi-scene deployment and transfer difficult.
Laskar and colleagues reverse the design. Their ResNet34 Siamese CNN learns the relative rotation and translation between two images. One branch also produces a 512-dimensional descriptor used to retrieve the five most similar database images. Because each database camera has a known pose, pairs of predicted translation directions are triangulated into query-location hypotheses. RANSAC removes geometrically inconsistent candidates, while a second consensus procedure fuses orientation hypotheses.
On Microsoft 7 Scenes, the method reaches 0.21 m and 9.30° mean median error, compared with PoseNet's 0.44 m and 10.4°. On the new five-scene, common-frame University dataset, it achieves 1.58 m and 14.02°, versus 2.88 m and 15.33° for the baseline. Training only on University and testing on 7 Scenes still gives 0.36 m and 18.38°, demonstrating transfer. The approach remains sensitive to retrieval quality, ray geometry, and dynamic appearance, but establishes relative pose as a scalable alternative to scene-specific absolute regression.
Deep Analysis
Background
Camera relocalization supports navigation, augmented reality, SfM, and SLAM. SIFT/ORB matching with SfM and PnP-RANSAC is geometrically grounded but vulnerable to texture, lighting, and computational cost. Scene Coordinate Regression Forests and differentiable RANSAC predict 3D points but require depth during training. PoseNet, PoseNet2, HourglassPose, LSTM-Pose, and VidLoc use RGB CNNs, yet generally regress absolute scene coordinates and therefore train scene by scene.
Core Problem
The task is to recover a query image's full 6DoF camera pose from RGB and a pose-labelled image database. Sparse databases create location errors when the query lies far from retrieved views. Absolute regression is tied to a coordinate frame and does not transfer naturally. Relative translation also lacks reliable scale, while incorrect retrieval, perceptual aliasing, and degenerate ray geometry can destabilize reconstruction.
Innovation
- ��Replace scene-bound absolute regression with pairwise relative rotation and translation. •Use shared-weight ResNet34 branches to learn transferable 512-dimensional representations. •Recover absolute query location from known database poses and relative directions. •Introduce two robust fusion stages: translation triangulation plus RANSAC, and rotation consensus filtering. •Release University, a five-scene common-coordinate indoor dataset containing Office, Meeting, Kitchen, Conference, and Coffee Room, with 9,694 training and 5,068 test images.
Methodology
- ��Training input: overlapping image pairs with ground-truth relative rotation Δrgt and translation Δtgt; output: quaternion Δr and 3D Δt; objective: L=||Δtgt−Δt||²+β||Δrgt−Δr||², β=1. •Representation: truncated ImageNet-pretrained ResNet34 produces 512-D descriptors. •Retrieval: rank database images by descriptor dot product and retain N=5. •Pairwise estimation: feed query and neighbor representations to the shared regression head. •Translation: triangulate every pair of predicted directions, producing C(5,2) candidates; use a 20° angular RANSAC consensus. •Rotation: compute Rq hypotheses from ΔRj=RjᵀRq, select the strongest consensus, and use robust rotation averaging for ties.
Experiments
Experiments use Microsoft 7 Scenes and University. Evaluation reports per-scene median translation and orientation errors, comparing PoseNet, LSTM-Pose, VidLoc, HourglassPose, PoseNet2, and a scene-wise ResNet34-Pose baseline. Training lasts 200 epochs with SGD, batch size 64, initial learning rate 0.1 decayed tenfold every 50 epochs, weight decay 10^-5, β=1, random 224×224 crops, and central crops at test time. Cross-dataset generalization is also evaluated.
Results
The 7 Scenes average is 0.21 m and 9.30°, versus PoseNet's 0.44 m and 10.4°, LSTM-Pose's 0.31 m and 9.85°, and HourglassPose's 0.23 m and 9.53°. The method outperforms PoseNet2 in four scenes, although PoseNet2 uses a different architecture and reprojection loss. On University, it improves the baseline from 2.88 m, 15.33° to 1.58 m, 14.02°. University-only training transfers to 7 Scenes at 0.36 m and 18.38°.
Applications
Robots can relocalize against a pose-labelled visual map without retraining a separate absolute regressor for every room. AR glasses and mobile devices can retrieve compact descriptors, estimate relative pose, and stabilize overlays or indoor navigation. Deployment requires calibrated database poses, sufficient visual overlap, and efficient nearest-neighbor search. The design is especially attractive when several sub-scenes use different local structures but must share one localization model.
Limitations & Outlook
The method assumes trustworthy database poses and useful viewpoint overlap. Incorrect retrieval contaminates all subsequent relative estimates; nearly parallel translation rays make triangulation ill-conditioned. It still requires an offline image database and uses fixed N=5 and a 20° threshold. Translation scale is not robustly exploited, and evaluation focuses on indoor benchmarks. Future systems should jointly learn retrieval and pose uncertainty, recover scale explicitly, optimize multi-view pose graphs, and test dynamic, outdoor, day-night, and city-scale settings.
Plain Language Accessible to non-experts
Imagine a guide working inside a large building. The building has an album of photographs, and every photograph is labelled with the exact place where it was taken. When you send the guide a new photograph, the guide first finds the five album pictures that look most similar. It does not try to memorize the entire building; it only chooses useful nearby clues.
Next, it compares your photograph with each selected picture and says, “Compared with this picture, the camera moved in this direction and turned by this amount.” Two labelled pictures provide two direction lines that should meet near your position. The guide tries several pairs and asks the remaining pictures to vote. Guesses that disagree with most voters are discarded.
The same voting idea determines which way the camera faces. This is why the system can work in a building it never saw during training: it learns how views change relative to one another, rather than memorizing one building’s absolute map. On 7 Scenes it reached about 0.21 metres and 9.30 degrees of error. Still, if the album lacks similar views, or all clues point almost parallel to one another, the answer becomes uncertain.
ELI14 Explained like you're 14
Picture a phone trying to locate you in a school it has never visited. It owns a photo album where every picture has a map location. First, it searches for pictures that look like your photo—maybe the same lockers, desk, or poster. Then it compares your picture with those photos and guesses how far the camera moved and how much it turned.
One guess might be wrong, so the phone gets several clues. If two old photos point toward your location, their direction lines can cross and reveal where you are. The other photos act like a jury: if one guess disagrees with everyone else, it gets kicked out! The phone also checks which direction the camera is facing by comparing all the turning guesses.
The cool part is that the phone is not memorizing one school’s exact coordinates. It is learning a general trick for comparing two views. That is why the researchers could train on one group of rooms and test on rooms it had not seen. On the 7 Scenes test, the average error was only 0.21 metres for position and 9.30 degrees for direction.
But is it perfect? Nope! If the album has no similar photo, if objects move, or if all direction clues are almost parallel, the estimate can wobble. Better searching, moving-object handling, and stronger scale estimation could make it useful for robots, games, and AR glasses.
Glossary
Siamese CNN
A neural network with two branches that share weights and process two inputs. It is designed to learn similarity or relations between inputs rather than classify one input alone.
The paper uses shared ResNet34 branches to encode image pairs and predict relative pose.
Relative pose
The rotation and translation describing one camera with respect to another. It is a coordinate-frame-independent geometric relationship.
The network regresses quaternion rotation and 3D translation between query and database images.
Triangulation
A geometric procedure that estimates a point from intersecting viewing directions. It becomes ill-conditioned when the directions are nearly parallel.
Two predicted database-to-query translation directions estimate the query camera location.
RANSAC
A robust estimation algorithm that fits a model using consensus among observations and rejects outliers. It is widely used when measurements may be corrupted.
The paper applies separate consensus procedures to translation and rotation hypotheses.
PoseNet
A CNN that directly regresses an image's absolute camera pose. Its learned mapping is usually tied to the coordinate frame of a particular scene.
PoseNet and PoseNet2 are major comparison methods.
7 Scenes and University
7 Scenes is a standard seven-scene indoor benchmark. University is the paper's five-scene indoor dataset registered to one common coordinate frame.
They support accuracy, multi-scene, and cross-dataset generalization experiments.
Open Questions Unanswered questions from this research
- 1 How can localization remain reliable when retrieval fails, viewpoint overlap is tiny, or visual aliasing is severe? RANSAC can reject inconsistent hypotheses but cannot recover missing geometric evidence.
- 2 Why is predicted translation scale unreliable in practice? Depth, calibrated motion priors, or multi-view optimization may be needed for robust metric-scale recovery.
- 3 Performance and cost remain uncertain in outdoor, dynamic, day-night, and city-scale environments with much larger databases.
Applications
Immediate Applications
Indoor robot relocalization
A robot can compare live RGB frames with a pose-labelled visual map, predict pairwise relative poses, and recover its 6DoF state through triangulation and RANSAC. It requires an offline mapped database with adequate viewpoint coverage, but avoids training a separate absolute regressor for every room.
Augmented-reality tracking
Phones and AR glasses can store compact 512-D descriptors, retrieve visually similar reference images, and estimate camera pose for indoor navigation or stable virtual overlays. The expected benefit is lower memory use and easier deployment across multiple rooms, provided the scene has persistent visual structure.
Long-term Vision
A cross-building localization model
A future system could learn relative geometry across buildings, cameras, seasons, and lighting conditions, then combine neural retrieval with pose-graph optimization. Such a model could let robots enter unseen facilities with minimal adaptation; major obstacles include dynamics, scale consistency, database growth, and real-time computation.
Abstract
We propose a new deep learning based approach for camera relocalization. Our approach localizes a given query image by using a convolutional neural network (CNN) for first retrieving similar database images and then predicting the relative pose between the query and the database images, whose poses are known. The camera location for the query image is obtained via triangulation from two relative translation estimates using a RANSAC based approach. Each relative pose estimate provides a hypothesis for the camera orientation and they are fused in a second RANSAC scheme. The neural network is trained for relative pose estimation in an end-to-end manner using training image pairs. In contrast to previous work, our approach does not require scene-specific training of the network, which improves scalability, and it can also be applied to scenes which are not available during the training of the network. As another main contribution, we release a challenging indoor localisation dataset covering 5 different scenes registered to a common coordinate frame. We evaluate our approach using both our own dataset and the standard 7 Scenes benchmark. The results show that the proposed approach generalizes well to previously unseen scenes and compares favourably to other recent CNN-based methods.