We propose Probabilistic Warp Consistency, a weakly-supervised learning objective for semantic matching. Our approach directly supervises the dense matching scores predicted by the network, encoded as a conditional probability distribution. We first construct an image triplet by applying a known warp to one of the images in a pair depicting different instances of the same object class. Our probabilistic learning objectives are then derived using the constraints arising from the resulting image triplet. We further account for occlusion and background clutter present in real image pairs by extending our probabilistic output space with a learnable unmatched state. To supervise it, we design an objective between image pairs depicting different object classes. We validate our method by applying it to four recent semantic matching architectures. Our weakly-supervised approach sets a new state-of-the-art on four challenging semantic matching benchmarks. Lastly, we demonstrate that our objective also brings substantial improvements in the strongly-supervised regime, when combined with keypoint annotations.

Warp Consistency Graph	Results

The key challenge in learning dense correspondences lies in the lack of ground-truth matches for real image pairs. While photometric consistency losses provide unsupervised alternatives, they struggle with large appearance changes, which are ubiquitous in geometric and semantic matching tasks. Moreover, methods relying on synthetic training pairs often suffer from poor generalisation to real data. We propose Warp Consistency, an unsupervised learning objective for dense correspondence regression. Our objective is effective even in settings with large appearance and view-point changes. Given a pair of real images, we first construct an image triplet by applying a randomly sampled warp to one of the original images. We derive and analyze all flow-consistency constraints arising between the triplet. From our observations and empirical results, we design a general unsupervised objective employing two of the derived constraints. We validate our warp consistency loss by training three recent dense correspondence networks for the geometric and semantic matching tasks. Our approach sets a new state-of-the-art on several challenging benchmarks, including MegaDepth, RobotCar and TSS.

Dense flow estimation is often inaccurate in the case of large displacements or homogeneous regions. For most applications and down-stream tasks, such as pose estimation, image manipulation, or 3D reconstruction, it is crucial to know when and where to trust the estimated matches. In this work, we aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map indicating the reliability and accuracy of the prediction. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. In particular, we parametrize the predictive distribution as a constrained mixture model, ensuring better modelling of both accurate flow predictions and outliers. Moreover, we develop an architecture and training strategy tailored for robust and generalizable uncertainty prediction in the context of self-supervised training.

The feature correlation layer serves as a key neural network module in numerous computer vision problems that involve dense correspondences between image pairs. It predicts a correspondence volume by evaluating dense scalar products between feature vectors extracted from pairs of locations in two images. However, this point-to-point feature comparison is insufficient when disambiguating multiple similar regions in an image, severely affecting the performance of the end task. This work proposes GOCor, a fully differentiable dense matching module, acting as a direct replacement to the feature correlation layer. The correspondence volume generated by our module is the result of an internal optimization procedure that explicitly accounts for similar regions in the scene. Moreover, our approach is capable of effectively learning spatial matching priors to resolve further matching ambiguities.

PDC-Net and PDC-Net+ have multiple inference alternative options. if model is PDC-Net, add options:

• --confidence_map_R, for computation of the confidence map p_r, default is 1.0
• --multi_stage_type in
  • 'D' (or 'direct')
  • 'H' (or 'homography_from_quarter_resolution_uncertainty')
  • 'MS' (or 'multiscale_homo_from_quarter_resolution_uncertainty')
• --ransac_thresh, used for homography and multiscale multi-stages type, default is 1.0
• --mask_type, for thresholding the estimated confidence map and using the confident matches for internal homography estimation, for homography and multiscale multi-stage types, default is proba_interval_1_above_5
• --homography_visibility_mask, default is True
• --scaling_factors', used for multi-scale, default are [0.5, 0.6, 0.88, 1, 1.33, 1.66, 2]

Use direct ('D') when image pairs only show limited view-point changes (for example consecutive images of a video, like in the optical flow task). For larger view-point changes, use homography ('H') or multi-scale ('MS').

For example, to run PDC-Net or PDC-Net+ with homography, add at the end of the command

PDCNet --multi_stage_type H --mask_type proba_interval_1_above_10

PDC-Net and PDC-Net+ has multiple inference alternative options. if model if PDC-Net, add options:

• --confidence_map_R, for computation of the confidence map p_r, default is 1.0
• --multi_stage_type in
  • 'D' (or 'direct')
  • 'H' (or 'homography_from_quarter_resolution_uncertainty')
  • 'MS' (or 'multiscale_homo_from_quarter_resolution_uncertainty')
• --ransac_thresh, used for homography and multiscale multi-stages type, default is 1.0
• --mask_type, for thresholding the estimated confidence map and using the confident matches for internal homography estimation, for homography and multiscale multi-stage types, default is proba_interval_1_above_5
• --homography_visibility_mask, default is True
• --scaling_factors', used for multi-scale, default are [0.5, 0.6, 0.88, 1, 1.33, 1.66, 2]

For example, to run PDC-Net or PDC-Net+ with homography, add at the end of the command

PDCNet --multi_stage_type H --mask_type proba_interval_1_above_10

Results using PDC-Net with multi-stage (homography_from_quarter_resolution_uncertainty, H) or multi-scale (multiscale_homo_from_quarter_resolution_uncertainty, MS) employ RANSAC internally. Therefore results may vary a bit but should remain within 1-2 %. For pose estimation, we also compute the pose with RANSAC, which leads to some variability in the results.

Data preparation: We use the test set provided in RANSAC-Flow. It is composed of 1600 pairs and also includes a csv file ('test1600Pairs.csv') containing the name of image pairs to evaluate and the corresponding ground-truth correspondences. Download everything with

bash assets/download_megadepth_test.sh

The resulting file structure is the following

megadepth_test_set/
└── MegaDepth/
    └── Test/
        └── test1600Pairs/  
        └── test1600Pairs.csv

Evaluation: After updating the path of 'megadepth' and 'megadepth_csv' in admin/local.py, evaluation is run with

python eval_matching.py --dataset megadepth --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type MS

Similar results should be obtained:

Data preparation: Images can be downloaded from the Visual Localization Challenge (at the bottom of the site), or more precisely here. The CSV file with the ground-truth correspondences can be downloaded from here. The file structure should be the following:

RobotCar
├── img/
└── test6511.csv

Evaluation: After updating the path of 'robotcar' and 'robotcar_csv' in admin/local.py, evaluation is run with

python eval_matching.py --dataset robotcar --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type MS

Similar results should be obtained:

Data preparation: execute 'bash assets/download_ETH3D.sh' from our GLU-Net repo. It does the following:

• Create your root directory ETH3D/, create two sub-directories multiview_testing/ and multiview_training/
• Download the "Low rew multi-view, training data, all distorted images" here and unzip them in multiview_training/
• Download the "Low rew multi-view, testing data, all undistorted images" here and unzip them in multiview_testing/
• We directly provide correspondences for pairs of images taken at different intervals. There is one bundle file for each dataset and each rate of interval, for example "lakeside_every_5_rate_of_3". This means that we sampled the source images every 5 images and the target image is taken at a particular rate from each source image. Download all these files here and unzip them.

As illustration, your root ETH3D directory should be organised as follows:

/ETH3D/
       multiview_testing/
                        lakeside/
                        sand_box/
                        storage_room/
                        storage_room_2/
                        tunnel/
       multiview_training/
                        delivery_area/
                        electro/
                        forest/
                        playground/
                        terrains/
        info_ETH3D_files/

The organisation of your directories is important, since the bundle files contain the relative paths to the images, from the ETH3D root folder.

Evaluation: for each interval rate (3,5,7,9,11,13,15), we compute the metrics for each of the sub-datasets (lakeside, delivery area and so on). The final metrics are the average over all datasets for each rate. After updating the path of 'eth3d' in admin/local.py, evaluation is run with

python eval_matching.py --dataset robotcar --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type D

PCK-1 for different rates of intervals between image pairs:

Note that the PCKs are computed per image, and then averaged per sequence. The final metrics is the average over all sequences. It corresponds to the results '_per_image' in the outputted metric file. Note that this is not the metrics used in the PDC-Net paper, where the PCKs are c omputed per sequence instead, using the PDC-Net direct approach (corresponds to results '-per-rate' in outputted metric file).

PCK-5 for different rates of intervals between image pairs:

Data preparation: Download the data with

bash assets/download_hpatches.sh

The corresponding csv files for each viewpoint ID with the path to the images and the homography parameters relating the pairs are listed in assets/.

Evaluation: After updating the path of 'hp' in admin/local.py, evaluation is run with

python eval_matching.py --dataset hp --model GLUNet_GOCor --pre_trained_models static --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: Both KITTI-2012 and 2015 datasets are available here

Evaluation: After updating the path of 'kitti2012' and 'kitti2015' in admin/local.py, evaluation is run with

python eval_matching.py --dataset kitti2015 --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 PDCNet --multi_stage_type direct

Similar results should be obtained:

Data preparation: Download the data with

bash assets/download_sintel.sh

Evaluation: After updating the path of 'sintel' in admin/local.py, evaluation is run with

python eval_matching.py --dataset sintel --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type direct

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_tss.sh

Evaluation: After updating the path of 'tss' in admin/local.py, evaluation is run with

python eval_matching.py --dataset TSS --model GLUNet_GOCor --pre_trained_models static --optim_iter 3 --local_optim_iter 7 --flipping_condition True --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_pf_pascal.sh

Evaluation: After updating the path of 'PFPascal' in admin/local.py, evaluation is run with

python eval_matching.py --dataset PFPascal --model WarpCSemanticGLUNet --pre_trained_models pfpascal --flipping_condition False --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_pf_willow.sh

Evaluation: After updating the path of 'PFWillow' in admin/local.py, evaluation is run with

python eval_matching.py --dataset PFWillow --model WarpCSemanticGLUNet --pre_trained_models pfpascal --flipping_condition False --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_spair.sh

Evaluation: After updating the path of 'spair' in admin/local.py, evaluation is run with

python eval_matching.py --dataset spair --model WarpCSemanticGLUNet --pre_trained_models pfpascal  --flipping_condition False --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_caltech.sh

Evaluation: After updating the path of 'spair' in admin/local.py, evaluation is run with

python eval_matching.py --dataset caltech --model WarpCSemanticGLUNet --pre_trained_models pfpascal  --flipping_condition False --save_dir path_to_save_dir

Data preparation: The groundtruth for YFCC is provided the file assets/yfcc_test_pairs_with_gt_original.txt (from SuperGlue repo). Images can be downloaded from the OANet repo and moved to the desired location

bash assets/download_yfcc.sh

File structure should be

YFCC
└──  images/
       ├── buckingham_palace/
       ├── notre_dame_front_facade/
       ├── reichstag/
       └── sacre_coeur/

Evaluation: After updating the path 'yfcc' in admin/local.py, compute metrics on YFCC100M with PDC-Net homography (H) using the command:

python -u eval_pose_estimation.py --dataset YFCC --model PDCNet --pre_trained_models megadepth --optim_iter 3  --local_optim_iter 7 --estimate_at_quarter_reso True --mask_type_for_pose_estimation proba_interval_1_above_10 --save_dir path_to_save_dir PDCNet --multi_stage_type H --mask_type proba_interval_1_above_10 

You should get similar metrics (not exactly the same because of RANSAC):

Data preparation: The images of the ScanNet test set (100 scenes, scene0707_00 to scene0806_00) are provided here. They were extracted from ScanNet github repo and processed. We use the groundtruth provided by in the SuperGlue repo provided here in the file assets/scannet_test_pairs_with_gt.txt.

Evaluation: After updating the path 'scannet_test' in admin/local.py, compute metrics on ScanNet with PDC-Net homography (H) using the command:

python -u eval_pose_estimation.py --dataset scannet --model PDCNet --pre_trained_models megadepth --optim_iter 3  --local_optim_iter 7 --estimate_at_quarter_reso True --mask_type_for_pose_estimation proba_interval_1_above_10 --save_dir path_to_save_dir PDCNet --multi_stage_type H --mask_type proba_interval_1_above_10 

You should get similar metrics (not exactly the same because of RANSAC):

This is the same image pairs used in GLU-Net repo. For the training, we use a combination of the DPED, CityScapes and ADE-20K datasets. The DPED training dataset is composed of only approximately 5000 sets of images taken by four different cameras. We use the images from two cameras, resulting in around 10,000 images. CityScapes additionally adds about 23,000 images. We complement with a random sample of ADE-20K images with a minimum resolution of 750 x 750. It results in 40.000 original images, used to create pairs of training images by applying geometric transformations to them. The path to the original images as well as the geometric transformation parameters are given in the csv files 'assets/csv_files/homo_aff_tps_train_DPED_CityScape_ADE.csv' and 'assets/csv_files/homo_aff_tps_test_DPED_CityScape_ADE.csv'.

1. Download the original images

• Download the DPED dataset (54 GB) ==> images are created in original_images/

• Download the CityScapes dataset

  • download 'leftImg8bit_trainvaltest.zip' (11GB, left 8-bit images - train, val, and test sets', 5000 images) ==> images are created in CityScape/
  • download leftImg8bit_trainextra.zip (44GB, left 8-bit images - trainextra set, 19998 images) ==> images are created in CityScape_extra/

• Download the ADE-20K dataset (3.8 GB, 20.210 images) ==> images are created in ADE20K_2016_07_26/

Put all the datasets in the same directory. As illustration, your root training directory should be organised as follows:

training_datasets/
    ├── original_images/
    ├── CityScape/
    ├── CityScape_extra/
    └── ADE20K_2016_07_26/

2. Save the synthetic image pairs and flows to disk
   During training, from this set of original images, the pairs of synthetic images could be created on the fly at each epoch. However, this dataset generation takes time and since no augmentation is applied at each epoch, one can also create the dataset in advance and save it to disk. During training, the image pairs composing the training datasets are then just loaded from the disk before passing through the network, which is a lot faster. To generate the training dataset and save it to disk:

python assets/save_training_dataset_to_disk.py --image_data_path /directory/to/original/training_datasets/ 
--csv_path assets/homo_aff_tps_train_DPED_CityScape_ADE.csv --save_dir /path/to/save_dir --plot True

It will create the images pairs and corresponding flow fields in save_dir/images and save_dir/flow respectively.

3. Add the paths in admin/local.py as 'training_cad_520' and 'validation_cad_520'

This is useful for adding moving objects. Download the images along with annotations from here. The root folder should be organized as follows. The add the paths in admin/local.py as 'coco'.

coco_root
    └── annotations
        └── instances_train2014.json
    └──images
        └── train2014

We use the reconstructions provided in the D2-Net repo. You can download the undistorted reconstructions and aggregated scene information folder directly here - Google Drive.

File structure should be the following:

MegaDepth
├── Undistorted_Sfm
└── scene_info

Them add the paths in admin/local.py as 'megadepth_training'.

This is adapted from our recent work Refign, with code. It allows training a probabilistic correspondence network unsupervised (mix of PDCNet and WarpC). It uses a simplifided version of PDCNet, without GOCor and which is predicting a uni-modal Gaussian probability distribution at every pixel instead of a Mixture of Laplacians.

• UAWarpC.train_UAWarpC_PDCNet_stage1: The default settings used for first stage network training without visibility mask. We train on real image pairs of the MegaDepth dataset.

• UAWarpC.train_UAWarpC_PDCNet_stage2: We further finetune the network trained with stage1, by including our visibility mask.

• PWarpC.train_weakly_supervised_PWarpC_SFNet_pfpascal: The default settings used to train the weakly-supervised PWarpC-SF-Net on PF-Pascal.

• PWarpC.train_weakly_supervised_PWarpC_SFNet_spair_from_pfpascal: The default settings used to train the weakly-supervised PWarpC-SF-Net on SPair. More precisely, the network is first trained on PF-Pascal (above) and further finetuned on SPair-71K.

• PWarpC.train_strongly_supervised_PWarpC_SFNet_pfpascal: The default settings used to train the strongly-supervised PWarpC-SF-Net on PF-Pascal.

• PWarpC.train_strongly_supervised_PWarpC_SFNet_spair_from_pfpascal: The default settings used to train the strongly-supervised PWarpC-SF-Net on Spair-71K.

• The rest to come

• WarpC.train_WarpC_GLUNet_stage1: The default settings used for first stage network training without visibility mask. We train on real image pairs of the MegaDepth dataset.

• WarpC.train_WarpC_GLUNet_stage2: We further finetune the network trained with stage1, by including our visibility mask. The network corresponds to our final WarpC-GLU-Net (see WarpC paper).

• WarpC.train_ft_WarpCSemanticGLUNet: The default settings used for training the final WarpC-SemanticGLU-Net (see WarpC paper). We finetune the original SemanticGLUNet (trained on the static/CAD synthetic data) on PF-Pascal using Warp Consistency.

• PDCNet.train_PDCNet_plus_stage1: The default settings used for first stage network training with fixed backbone weights. We train first on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), on which we add MULTIPLE independently moving objects and perturbations. We also train by applying our object reprojection mask.

• PDCNet.train_PDCNet_plus_stage2: The default settings used for training the final PDC-Net+ model (see PDC-Net+ paper). This setting fine-tunes all layers in the model trained using PDCNet_stage1 (including the feature backbone). As training dataset, we use a combination of the same dataset than in stage 1 as well as image pairs from the MegaDepth dataset and their sparse ground-truth correspondence data. We also apply the reprojection mask.

• PDCNet.train_PDCNet_stage1: The default settings used for first stage network training with fixed backbone weights. We initialize the backbone VGG-16 with pre-trained ImageNet weights. We train first on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), on which we add independently moving objects and perturbations.

• PDCNet.train_PDCNet_stage2: The default settings used for training the final PDC-Net model (see PDC-Net paper). This setting fine-tunes all layers in the model trained using PDCNet_stage1 (including the feature backbone). As training dataset, we use a combination of the same dataset than in stage 1 as well as image pairs from the MegaDepth dataset and their sparse ground-truth correspondence data.

• PDCNet.train_GLUNet_GOCor_star_stage1: Same settings than for PDCNet_stage1, with different model (non probabilistic baseline). The loss is changed accordingly to the L1 loss instead of the negative log likelihood loss.

• PDCNet.train_GLUNet_GOCor_star_stage2: The default settings used for training the final GLU-Net-GOCor* (see PDCNet paper).

• GLUNet.train_GLUNet_with_synthetically_generated_data: This is a simple example of how to generate random transformations on the fly, and to apply them to original images, in order to create training image pairs and their corresponding ground-truth flow. Here, the random transformations are applied to MegaDepth images. On the created image pairs and ground-truth flows, we additionally add a randomly moving object.

• GLUNet.train_GLUNet_static: The default settings used training the final GLU-Net (of the paper GLU-Net).
  We fix the backbone weights and initialize the backbone VGG-16 with pre-trained ImageNet weights. We train on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), which is later (GOCor paper) referred to as 'static' dataset.

• GLUNet.train_GLUNet_dynamic: The default settings used training the final GLU-Net trained on the dynamic dataset (of the paper GOCor).
  We fix the backbone weights and initialize the backbone VGG-16 with pre-trained ImageNet weights. We train on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), on which we add one independently moving object. This dataset is referred to as 'dynamic' dataset in GOCor paper.

• GLUNet.train_GLUNet_GOCor_static: The default settings used training the final GLU-Net-GOCor (of the paper GOCor).
  We fix the backbone weights and initialize the backbone VGG-16 with pre-trained ImageNet weights. We train on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), which is later (GOCor paper) referred to as 'static' dataset.

• GLUNet.train_GLUNet_GOCor_dynamic: The default settings used training the final GLU-Net-GOCor trained on the dynamic dataset (of the paper GOCor).
  We fix the backbone weights and initialize the backbone VGG-16 with pre-trained ImageNet weights. We train on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), on which we add one independently moving object. This dataset is referred to as 'dynamic' dataset in GOCor paper.