valeoai/ALSO valeoai/ALSO

  • Stars
    star
    158
  • Rank 235,892 (Top 5 %)
  • Language
    Python
  • License
    Other
  • Created almost 2 years ago
  • Updated about 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
valeoai/ALSO
github

valeoai

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

ALSO: Automotive Lidar Self-supervision by Occupancy estimation

ALSO: Automotive Lidar Self-supervision by Occupancy estimation

Alexandre Boulch1    Corentin Sautier1,2    Björn Michele1,3    Gilles Puy1    Renaud Marlet1,2

1 Valeo.ai, Paris, France 2 LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France

3 CNRS, IRISA, Univ. Bretagne Sud, Vannes, France


Arxiv


ALSO has been accepted at CVPR 2023


Overview


This is the reference PyTorch implementation for training and testing self-supervision for automotive lidar point cloud using the method described in our paper ALSO: Automotive Lidar Self-supervision by Occupancy estimation

Overview

Citation

Please acknowledge our work in your publications:

@InProceedings{ALSO,
author = {Alexandre Boulch and Corentin Sautier and Björn Michele and Gilles Puy and Renaud Marlet},
title = {{ALSO}: Automotive Lidar Self-supervision by Occupancy estimation},
booktitle = {International Conference on Computer Vision and Pattern Recognition (CVPR)},
year = 2023,
}

Dependencies

We give the library version(s), we used for experiments.

  • Hydra (1.2.0)
  • Pytorch-geometric (2.1.0)
  • Pytorch-Lightning (1.7.2)

Semantic segmentation specific libraries

  • torchsparse (1.4.0)

Detection specific libraries

  • spconv (2.1.21)
  • openpcdet (0.6.0)

Installation

At root of the folder run:

pip install -ve .

Then all scripts can be run from also_selfsup folder:

cd also_selfsup

Datasets

ONCE

We follow intruction from ONCE dataset to generate the information files of the raw_small dataset.

Semantic segmentation

Pre-training

python train_selfsupervised.py cfg=nuscenes

for a pre-training on nuScenes.

for other datasets, replace cfg=nuscenes by the desired dataset

Parameters

We use Hydra for easy configuration of trainings. The pretraining parameters are set for pre-training with a NVidia-V100 16Gb. In order to pre-train on other configurations, one can modify parameters directly in the command line, e.g., setting the batch size to 8:

python train_selfsupervised.py ... cfg.training.batch_size=8

or the backbone to SPVCNN:

python train_selfsupervised.py ... cfg.network.backbone=SPVCNN

Downstream

From scratch

# 100%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=30 cfg.val_interval=5 cfg.downstream.skip_ratio=1

# 50%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=50 cfg.val_interval=5 cfg.downstream.skip_ratio=2

# 10%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=100 cfg.val_interval=10 cfg.downstream.skip_ratio=10

# 1%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=500 cfg.val_interval=50 cfg.downstream.skip_ratio=100

# 0.1%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=1000 cfg.val_interval=100 cfg.downstream.skip_ratio=1000

With pretrained models

First, convert the pre-trained model to be loaded

python convert_models.py --ckpt path_to_pretraining_checkpoint

Second, run downstream training

python train_downstream_semseg.py cfg=nuscenes cfg.downstream.checkpoint_dir='path_to_checkpoint_directory' cfg.downstream.checkpoint_name='pretrained_backbone_XXX.ckpt'

Evaluation of downstream models

First, convert the trained model to be loaded

python convert_models.py --downstream --ckpt path_to_downstream_checkpoint

Second, run the evaluation script

python eval.py --split val --config path_to_downstream_model/config.yaml --ckpt path_to_downstream_checkpoint/trained_model_XXX.ckpt

Detection

Pre-training

Similar to semantic segmentation, pre-training for detection can simply be run using:

python train_selfsupervised.py cfg=kitti3d_second
python train_selfsupervised.py cfg=nuscenes_second_kitti

Downstream

First, convert the trained model to be loaded

python convert_models.py --ckpt path_to_pretraining_checkpoint

Second, use the pre-trained model with the official OpenPCDet code:

cd path_to_OpenPCDet/tools/
python train.py --cfg_file cfgs/kitti_models/second.yaml --extra_tag --pretrained_model path_to_pretrained_model/pretrained_backbone_XXX.ckpt

Pre-trained models

Task Pre-training dataset Model Link Notes
Semantic segmentation nuScenes MinkUNet34 download
    SPVCNN download
  SemanticKITTI MinkUNet18SC download
    SPVCNN download
Detection KITTI3D SECOND/PVRCNN download trained for KITTI3D
  nuScenes SECOND/PVRCNN download trained for KITTI3D
  KITTI360 SECOND/PVRCNN download trained for KITTI3D
  ONCE Small SECOND/PVRCNN download trained for ONCE

Here are links to pre-trained models

Acknowledgments

This project would not have been possible without many community resources and repositories. Among them:

Please, consider acknowleding these projects.

More Repositories

1

WoodScape

The repository containing tools and information about the WoodScape dataset.
Python
602
star
2

ADVENT

Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation
Python
379
star
3

LOST

Pytorch implementation of LOST unsupervised object discovery method
Python
234
star
4

xmuda

Cross-Modal Unsupervised Domain Adaptationfor 3D Semantic Segmentation
Python
192
star
5

ZS3

Zero-Shot Semantic Segmentation
Python
187
star
6

POCO

Python
178
star
7

SLidR

Official PyTorch implementation of "Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data"
Python
172
star
8

ConfidNet

Addressing Failure Prediction by Learning Model Confidence
Python
162
star
9

RADIal

Jupyter Notebook
155
star
10

Maskgit-pytorch

Jupyter Notebook
142
star
11

BF3S

Boosting Few-Shot Visual Learning with Self-Supervision
Python
136
star
12

DADA

Depth-aware Domain Adaptation in Semantic Segmentation
Python
114
star
13

FLOT

FLOT: Scene Flow Estimation by Learned Optimal Transport on Point Clouds
Python
95
star
14

obow

Python
95
star
15

carrada_dataset

Jupyter Notebook
84
star
16

rainbow-iqn-apex

Distributed Rainbow-IQN for Atari
Python
75
star
17

rangevit

Python
72
star
18

FOUND

PyTorch code for Unsupervised Object Localization: Observing the Background to Discover Objects
Python
66
star
19

PointBeV

Official implementation of PointBeV: A Sparse Approach to BeV Predictions
Python
63
star
20

LightConvPoint

Python
62
star
21

Awesome-Unsupervised-Object-Localization

Curated list of awesome works on unsupervised object localization in 2D images.
61
star
22

BEVContrast

BEVContrast: Self-Supervision in BEV Space for Automotive Lidar Point Clouds - Official PyTorch implementation
Python
60
star
23

MVRSS

Python
58
star
24

FKAConv

Python
40
star
25

SALUDA

SALUDA: Surface-based Automotive Lidar Unsupervised Domain Adaptation
Python
38
star
26

LaRa

LaRa: Latents and Rays for Multi-Camera Bird’s-Eye-View Semantic Segmentation
Python
38
star
27

WaffleIron

Python
37
star
28

obsnet

Python
32
star
29

BUDA

Boundless Unsupervised Domain Adaptation in Semantic Segmentation
32
star
30

ScaLR

PyTorch code and models for ScaLR image-to-lidar distillation method
Python
32
star
31

3DGenZ

Public repository of the 3DV 2021 paper "Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds"
Python
31
star
32

SemanticPalette

Semantic Palette: Guiding Scene Generation with Class Proportions
Python
29
star
33

xmuda_journal

[TPAMI] Cross-modal Learning for Domain Adaptation in 3D Semantic Segmentation
Python
28
star
34

NeeDrop

NeeDrop: Self-supervised Shape Representation from Sparse Point Clouds using Needle Dropping
Python
27
star
35

PCAM

Python
27
star
36

MTAF

Multi-Target Adversarial Frameworks for Domain Adaptation in Semantic Segmentation
Python
23
star
37

ESL

ESL: Entropy-guided Self-supervised Learning for Domain Adaptation in Semantic Segmentation
Python
19
star
38

STEEX

STEEX: Steering Counterfactual Explanations with Semantics
Python
18
star
39

OCTET

Python
17
star
40

CAB

Python
16
star
41

MuHDi

Official PyTorch implementation of "Multi-Head Distillation for Continual Unsupervised Domain Adaptation in Semantic Segmentation"
Python
15
star
42

diffhpe

Official code of "DiffHPE: Robust, Coherent 3D Human Pose Lifting with Diffusion"
Python
14
star
43

bravo_challenge

BRAVO Challenge Toolkit and Evaluation Code
Python
14
star
44

sfrik

Official code for "Self-supervised learning with rotation-invariant kernels"
Python
12
star
45

BEEF

Python
11
star
46

SP4ASC

Python
7
star
47

bownet

Learning Representations by Predicting Bags of Visual Words
7
star
48

QuEST

Python
5
star
49

MFEval

[ICRA2024] Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive? This is the official implementation of the evaluation protocol proposed in this work for motion forecasting models with real-world perception inputs.
Python
5
star
50

Occfeat

5
star
51

dl_utils

The library used in the Valeo Deep learning training.
Python
3
star
52

tutorial-images

2
star
53

PAFUSE

Official repository of PAFUSE
Python
2
star
54

MOCA

MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments
2
star
55

valeoai.github.io

JavaScript
1
star