autonomousvision/plant autonomousvision/plant

  • Stars
    star
    201
  • Rank 190,106 (Top 4 %)
  • Language
    Python
  • License
    MIT License
  • Created over 1 year ago
  • Updated 6 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
autonomousvision/plant
github

autonomousvision

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[CoRL'22] PlanT: Explainable Planning Transformers via Object-Level Representations

PlanT: Explainable Planning Transformers via Object-Level Representations

News:
19.01.2023: We released the code to generate the attention visualization.
02.12.2022: We released the perception checkpoint and the code for the SENSORS and MAP track agent. Conda environment needs to be updated. Checkpoints of the perception are in the checkpoint folder. Please download again.
11.11.2022: We made some changes in the agent files to ensure compatibility with our perception PlanT. We therefore uploaded new checkpoint files. The old one does not work anymore with the current code.

Project Page | Paper | Supplementary

This repository provides code for the following paper:

demo

Content

Setup

First, you have to install carla and the conda environment.

# 1. Clone this repository
git clone https://github.com/autonomousvision/plant.git
cd plant
# 2. Setup Carla
# if you have carla already installed, skip the next step AND
# adapt the carla path in setup_env.sh before executing step 3.
chmod +x setup_carla.sh
./setup_carla.sh
# 3. Setup conda environment
chmod +x setup_env.sh
./setup_env.sh

conda activate plant
pip install -U openmim
mim install mmcv-full==1.7.0
pip install mmdet

Data and models

You can download our pretrained PlanT models by executing:

chmod +x download.sh
./download.sh

To download our 3x dataset run:

chmod +x download_data.sh
./download_data.sh

Data generation

You can download our dataset or generate your own dataset. In order to generate your own one you first need to start a Carla server:

# with display
./carla/CarlaUE4.sh --world-port=2000 -opengl

# without display
SDL_VIDEODRIVER=offscreen SDL_HINT_CUDA_DEVICE=0 ./carla/CarlaUE4.sh --world-port=2000 -opengl

To generate the data for the route specified in carla_agent_files/config/eval/train.yaml you can run

python leaderboard/scripts/run_evaluation.py user=$USER experiments=datagen eval=train

If you want to also save the sensor data that we used to train the perception module you can add the flag experiments.SAVE_SENSORS=1.

To generate the whole dataset you can use the datagen.sh file.

Training

To run the PlanT training on the 3x dataset, run:

python training/PlanT/lit_train.py user=$USER model=PlanT

To change any hyperparameters have a look at training/config/model/PlanT.yaml. For general training settings (e.g., #GPUs) check training/config/config.yaml.

Evaluation

This evaluates the PlanT model on the specified benchmark (default: longest6). The config is specified in the folder carla_agent_files/config.

Start a Carla server (see Data generation).
When the server is running, start the evaluation with:

python leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTmedium3x eval=longest6

You can find the results of the evaluation in a newly created evaluation folder inside the model folder. If you want to have a (very minimalistic) visualization you can set the viz flag (i.e., python leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTmedium3x eval=longest6 viz=1)

Explainability

The execution of the explainability agent contains two stages: (1) PlanT forwardpass (no execution of actions) to get attention weights. We filter the vehicles so that only the vehicles with the topk attention scores remain as input for the second step. (2) We execute either the expert or PlanT with the filtered input (the agent only sees topk vehicles instead of all).

Start a Carla server (see Data generation).
When the server is running, start the evaluation with:

python leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTExplainability experiments.exec_model=Expert experiments.topk=1

To obtain the attention visualization set experiments.topk=100000 and in addition add the flag save_explainability_viz=True. This saves a video per route in a viz_vid folder. The image resolution can be changed in carla_agent_files/explainability_agent.py.
Attention: saving the videos slows the evaluation down.

Perception PlanT

We release two PlanT agents suitable for the two CARLA Leaderboard tracks. For the SENSORS track we predict the route with our perception module. In the MAP track model we get the route information from the map. The code is taken from the TransFuser (PAMI 2022) repo and adapted for our usecase. The config is specified in the folder carla_agent_files/config. The config for the perception model is in training/Perception/config.py.

SENSORS track

Start a Carla server (see Data generation).

When the server is running, start the evaluation with:

python leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTSubmission track=SENSORS eval=longest6 save_path=SENSORSagent

Visualization can be activated with the viz flag, and the unblocking from the TransFuser repo can be activated with the experiments.unblock flag.

MAP track

Start a Carla server (see Data generation).

When the server is running, start the evaluation with:

python leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTSubmissionMap track=MAP eval=longest6 save_path=MAPagent

Visualization can be activated with the viz flag, and the unblocking from the TransFuser repo can be activated with the experiments.unblock flag.

Citation

If you use this code and data, please cite the following:

@inproceedings{Renz2022CORL,
    author       = {Katrin Renz and Kashyap Chitta and Otniel-Bogdan Mercea and A. Sophia Koepke and Zeynep Akata and Andreas Geiger},
    title        = {PlanT: Explainable Planning Transformers via Object-Level Representations},
    booktitle    = {Conference on Robotic Learning (CoRL)},
    year         = {2022}
}

Also, check out the code for other recent work on CARLA from our group:

More Repositories

1

sdfstudio

A Unified Framework for Surface Reconstruction
Python
1,861
star
2

occupancy_networks

This repository contains the code for the paper "Occupancy Networks - Learning 3D Reconstruction in Function Space"
Python
1,454
star
3

giraffe

This repository contains the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"
Python
1,227
star
4

stylegan-t

[ICML'23] StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis
Python
1,122
star
5

transfuser

[PAMI'23] TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving; [CVPR'21] Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
Python
1,023
star
6

stylegan-xl

[SIGGRAPH'22] StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets
Python
939
star
7

projected-gan

[NeurIPS'21] Projected GANs Converge Faster
Python
876
star
8

unimatch

[TPAMI'23] Unifying Flow, Stereo and Depth Estimation
Python
855
star
9

convolutional_occupancy_networks

[ECCV'20] Convolutional Occupancy Networks
Python
792
star
10

differentiable_volumetric_rendering

This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"
Python
782
star
11

mip-splatting

[CVPR'24 Oral] Mip-Splatting: Alias-free 3D Gaussian Splatting
Python
700
star
12

monosdf

[NeurIPS'22] MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
Python
535
star
13

shape_as_points

[NeurIPS'21] Shape As Points: A Differentiable Poisson Solver
Python
518
star
14

tuplan_garage

[CoRL'23] Parting with Misconceptions about Learning-based Vehicle Motion Planning
Python
418
star
15

unisurf

[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction
Python
410
star
16

graf

Official code release for "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis"
Jupyter Notebook
393
star
17

kitti360Scripts

This repository contains utility scripts for the KITTI-360 dataset.
Python
353
star
18

neat

[ICCV'21] NEAT: Neural Attention Fields for End-to-End Autonomous Driving
Python
301
star
19

gaussian-opacity-fields

Gaussian Opacity Fields for Efficient and Compact Surface Reconstruction in Unbounded Scenes
Python
285
star
20

occupancy_flow

This repository contains the code for the ICCV 2019 paper "Occupancy Flow - 4D Reconstruction by Learning Particle Dynamics"
Python
207
star
21

factor-fields

[SIGGRAPH 2023] We provide a unified formula for neural fields (Factor Fields) and a novel dictionary factorization (Dictionary Fields)
Jupyter Notebook
183
star
22

voxgraf

Official code release for VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids
Python
123
star
23

carla_garage

[ICCV'23] Hidden Biases of End-to-End Driving Models
Python
121
star
24

navsim

NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation
Python
119
star
25

texture_fields

This repository contains code for the paper 'Texture Fields: Learning Texture Representations in Function Space'.
Python
113
star
26

sledge

SLEDGE: Synthesizing Simulation Environments for Driving Agents with Generative Models
105
star
27

kitti360LabelTool

JavaScript
103
star
28

counterfactual_generative_networks

[ICLR'21] Counterfactual Generative Networks
Python
102
star
29

gta

[ICLR'24] GTA: A Geometry-Aware Attention Mechanism for Multi-view Transformers
Python
95
star
30

murf

[CVPR'24] MuRF: Multi-Baseline Radiance Fields
Python
84
star
31

controllable_image_synthesis

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis, CVPR 2020
Python
69
star
32

king

[ECCV'22] KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients
Python
61
star
33

handheld_svbrdf_geometry

On Joint Estimation of Pose, Geometry and svBRDF from a Handheld Scanner, CVPR2020
Python
57
star
34

connecting_the_dots

This repository contains the code for the paper "Connecting the Dots: Learning Representations for Active Monocular Depth Estimation" https://avg.is.tuebingen.mpg.de/publications/riegler2019cvpr
Python
51
star
35

frequency_bias

Official code for "On the Frequency Bias of Generative Models", NeurIPS 2021
Python
39
star
36

good

[ICLR'23] GOOD: Exploring Geometric Cues for Detecting Objects in an Open World
Python
39
star
37

data_aggregation

This repository contains the code for the CVPR 2020 paper "Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving"
Python
38
star
38

campari

[3DV'21] CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields
Python
29
star
39

autonomousvision.github.io

Blog of the Autonomous Vision Group at MPI-IS Tübingen and University of Tübingen.
HTML
19
star
40

visual_abstractions

6
star
41

slides

Slide repository of the Autonomous Vision Group at MPI-IS Tübingen and University of Tübingen.
CSS
2
star
42

similarity_reconstruction

This code is based on the paper Exploiting Object Similarity in 3D Reconstruction.
C++
1
star
43

slow_flow

This code is based on the paper Slow Flow: Exploiting High-Speed Cameras for Accurate and Diverse Optical Flow Reference Data.
C++
1
star