Camera Localization from Pixels to Pose

We introduce PixLoc, a neural network that localizes a given image via direct feature alignment with a 3D model of the environment. PixLoc is trained end-to-end and is interpretable, accurate, and generalizes to new scenes and across domains, e.g. from outdoors to indoors. It is described in our paper:

• Back to the Feature: Learning Robust Camera Localization from Pixels to Pose
• to appear at CVPR 2021
• Authors: Paul-Edouard Sarlin*, Ajaykumar Unagar*, Måns Larsson, Hugo Germain, Carl Toft, Victor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, and Torsten Sattler
• website: psarlin.com/pixloc with videos, slides, and visualizations

Installation

PixLoc is built with Python >=3.6 and PyTorch. The package pixloc includes code for both training and evaluation. Installing the package locally also installs the minimal dependencies listed in requirements.txt:

git clone https://github.com/cvg/pixloc/
cd pixloc/
pip install -e .

Generating visualizations and animations requires extra dependencies that can be installed with:

pip install -e .[extra]

Paths to the datasets and to training and evaluation outputs are defined in pixloc/settings.py. The default structure is as follows:

.
├── datasets     # public datasets
└── outputs
    ├── training # checkpoints and training logs
    ├── hloc     # 3D models and retrieval for localization
    └── results  # outputs of the evaluation

Visualizations

Demo

Have a look at the Jupyter notebook demo.ipynb to localize an image and animate the predictions in 2D and 3D. This requires downloading the pre-trained weights and the data for either the Aachen Day-Night or Extended CMU Seasons datasets using:

python -m pixloc.download --select checkpoints Aachen CMU --CMU_slices 2

3D viewer in the demo notebook.

3D animations

You can also check out our cool 3D viewer by launching the webserver with python3 viewer/server.py and visiting http://localhost:8000/viewer/viewer.html

Prediction confidences

The notebook visualize_confidences.ipynb shows how to visualize the confidences of the predictions over image sequences and turn them into videos.

Datasets

The codebase can evaluate PixLoc on the following datasets: 7Scenes, Cambridge Landmarks, Aachen Day-Night, Extended CMU Seasons, and RobotCar Seasons. Running the evaluation requires to download the following assets:

• The given dataset;
• Sparse 3D Structure-from-Motion point clouds and results of the image retrieval, both generated with our toolbox hloc and hosted here;
• Weights of the model trained on CMU or MegaDepth, hosted here.

We provide a convenient script to download all assets for one or multiple datasets using:

python -m pixloc.download --select [7Scenes|Cambridge|Aachen|CMU|RobotCar|checkpoints]

(see --help for additional arguments like --CMU_slices)

Evaluation

To perform the localization on all queries of one of the supported datasets, simply launch the corresponding run script:

python -m pixloc.run_[7Scenes|Cambridge|Aachen|CMU|RobotCar]  # choose one

Optional flags:

• --results path_to_output_poses defaults to outputs/results/pixloc_[dataset_name].txt
• --from_poses to refine the poses estimated by hloc rather than starting from reference poses
• --inlier_ranking to run the oracle baseline using inliers counts of hloc
• --scenes to select a subset of the scenes of the 7Scenes and Cambridge Landmarks datasets
• any configuration entry can be overwritten from the command line thanks to OmegaConf. For example, to increase the number of reference images, add refinement.num_dbs=5.

This displays the evaluation metrics for 7Scenes and Cambridge, while the other datasets require uploading the poses to the evaluation server hosted at visuallocalization.net.

Training

Data preparation

python -m pixloc.download --select CMU MegaDepth --training

Training experiment

python -m pixloc.pixlib.train pixloc_cmu_reproduce \
		--conf pixloc/pixlib/configs/train_pixloc_cmu.yaml

tensorboard --logdir outputs/training/

python -m pixloc.run_CMU.py experiment=experiment_name

Running PixLoc on your own data

Extending PixLoc

Goodies

Viewer

Geometry objects

from pixloc.pixlib.geometry import Pose, Camera
R    # rotation matrix with shape (B,3,3)
t    # translation vector with shape (B,3)
p3D  # 3D points with shape (B,N,3)

T_w2c = Pose.from_Rt(R, t)
T_w2c = T_w2c.cuda()   # behaves like a torch.Tensor
p3D_c = T_w2c * p3D    # transform points
T_A2C = T_B2C @ T_A2B  # chain Pose objects

cam1 = Camera.from_colmap(dict)     # from a COLMAP dict
cam = torch.stack([cam1, cam2])     # batch Camera objects
p2D, mask = cam.world2image(p3D_c)  # project and undistort
J, mask = cam.J_world2image(p3D_c)  # Jacobian of the projection

Implementation of GN-Net

Disclaimer

Since the publication of the paper, we have substantially refactored the codebase, with many usability improvements and updated dependencies. As a consequence, the results of the evaluation and training might slightly deviate from (and often improve over) the original numbers found in our CVPR 2021 paper. If you are writing a paper, for consistency with the literature, please report the original numbers. If you are building on top of this codebase, consider reporting the latest numbers for fairness.

BibTex Citation

Please consider citing our work if you use any of the ideas presented the paper or code from this repo:

@inproceedings{sarlin21pixloc,
  author    = {Paul-Edouard Sarlin and
               Ajaykumar Unagar and
               Måns Larsson and
               Hugo Germain and
               Carl Toft and
               Victor Larsson and
               Marc Pollefeys and
               Vincent Lepetit and
               Lars Hammarstrand and
               Fredrik Kahl and
               Torsten Sattler},
  title     = {{Back to the Feature: Learning Robust Camera Localization from Pixels to Pose}},
  booktitle = {CVPR},
  year      = {2021},
}

Logo font by Cyril Bourreau.