• Stars
    star
    265
  • Rank 154,577 (Top 4 %)
  • Language
    Jupyter Notebook
  • License
    Other
  • Created over 1 year ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Official Code for RVT-2 and RVT

PWC

RVT: Robotic View Transformer for 3D Object Manipulation
Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, Dieter Fox
CoRL 2023 (Oral)

If you find our work useful, please consider citing:

@article{,
  title={RVT: Robotic View Transformer for 3D Object Manipulation},
  author={Goyal, Ankit and Xu, Jie and Guo, Yijie and Blukis, Valts and Chao, Yu-Wei and Fox, Dieter},
  journal={CoRL},
  year={2023}
}

Getting Started

Install RVT

  • Tested (Recommended) Versions: Python 3.8. We used CUDA 11.1.

  • Step 1 (Optional): We recommend using conda and creating a virtual environment.

conda create --name rvt python=3.8
conda activate rvt
  • Step 2: Install PyTorch. Make sure the PyTorch version is compatible with the CUDA version. One recommended version compatible with CUDA 11.1 and PyTorch3D can be installed with the following command. More instructions to install PyTorch can be found here.
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
  • Step 3: Install PyTorch3D. One recommended version that is compatible with the rest of the library can be installed as follows. Note that this might take some time. For more instructions visit here.
curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
tar xzf 1.10.0.tar.gz
export CUB_HOME=$(pwd)/cub-1.10.0
pip install 'git+https://github.com/facebookresearch/pytorch3d.git@stable'

Once you have downloaded CoppeliaSim, add the following to your ~/.bashrc file. (NOTE: the 'EDIT ME' in the first line)

export COPPELIASIM_ROOT=<EDIT ME>/PATH/TO/COPPELIASIM/INSTALL/DIR
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$COPPELIASIM_ROOT
export QT_QPA_PLATFORM_PLUGIN_PATH=$COPPELIASIM_ROOT
export DISLAY=:1.0

Remember to source your .bashrc (source ~/.bashrc) or .zshrc (source ~/.zshrc) after this.

  • Step 5: Clone the repository with the submodules using the following command.
git clone --recurse-submodules [email protected]:NVlabs/RVT.git && cd RVT && git submodule update --init

Now, locally install RVT and other libraries using the following command. Make sure you are in folder RVT.

pip install -e . 
pip install -e rvt/libs/PyRep 
pip install -e rvt/libs/RLBench 
pip install -e rvt/libs/YARR 
pip install -e rvt/libs/peract_colab
  • Step 6: Download dataset.
    • For experiments on RLBench, we use pre-generated dataset provided by PerAct. Please download and place them under RVT/rvt/data/xxx where xxx is either train, test, or val.

    • Additionally, we use the same dataloader as PerAct, which is based on YARR. YARR creates a replay buffer on the fly which can increase the startup time. We provide an option to directly load the replay buffer from the disk. We recommend using the pre-generated replay buffer (98 GB) as it reduces the startup time. You can either download replay.tar.xz which contains the replay buffer for all tasks or replay buffer for indidual tasks. After downloading, uncompress the replay buffer(s) (for example using the command tar -xf replay.tar.xz) and place it under RVT/rvt/replay/replay_xxx where xxx is either train or val. Note that is useful only if you want to train RVT from scratch and not needed if you want to evaluate the pre-trained model.

Using the library:

Training RVT

Default command

To train RVT on all RLBench tasks, use the following command (from folder RVT/rvt):

python train.py --exp_cfg_path configs/all.yaml --device 0,1,2,3,4,5,6,7

We use 8 V100 GPUs. Change the device flag depending on available compute.

More details about train.py
  • default parameters for an experiment are defined here.
  • default parameters for rvt are defined here.
  • the parameters in for experiment and rvt can be overwritten by two ways:
    • specifying the path of a yaml file
    • manually overwriting using a opts string of format <param1> <val1> <param2> <val2> ..
  • Manual overwriting has higher precedence over the yaml file.
python train.py --exp_cfg_opts <> --mvt_cfg_opts <> --exp_cfg_path <> --mvt_cfg_path <>

The following command overwrites the parameters for the experiment with the configs/all.yaml file. It also overwrites the bs parameters through the command line.

python train.py --exp_cfg_opts "bs 4" --exp_cfg_path configs/all.yaml --device 0

Evaluate on RLBench

Evaluate RVT on RLBench

Download the pretrained RVT model. Place the model (model_14.pth trained for 15 epochs or 100K steps) and the config files under the folder runs/rvt/. Run evaluation using (from folder RVT/rvt):

python eval.py --model-folder runs/rvt  --eval-datafolder ./data/test --tasks all --eval-episodes 25 --log-name test/1 --device 0 --headless --model-name model_14.pth
Evaluate the official PerAct model on RLBench

Download the officially released PerAct model. Put the downloaded policy under the runs folder with the recommended folder layout: runs/peract_official/seed0. Run the evaluation using:

python eval.py --eval-episodes 25 --peract_official --peract_model_dir runs/peract_official/seed0/weights/600000 --model-name QAttentionAgent_layer0.pt --headless --task all --eval-datafolder ./data/test --device 0 

Gotchas

  • If you get qt plugin error like qt.qpa.plugin: Could not load the Qt platform plugin "xcb", try uninstalling opencv-python and installing opencv-python-headless
pip uninstall opencv-python                                                                                         
pip install opencv-python-headless
  • If you have CUDA 11.7, an alternate installation strategy could be to use the following command for Step 2 and Step 3. Note that this is not heavily tested.
# Step 2:
pip install pytorch torchvision torchaudio
# Step 3:
pip install 'git+https://github.com/facebookresearch/pytorch3d.git@stable'
  • If you are having issues running evaluation on a headless server, please refer to #2 (comment).

  • If you want to generate visualization videos, please refer to #5.

FAQ's

Q. What is the advantag of RVT over PerAct?

RVT is both faster to train and performs better than PerAct.

Q. What resources are required to train RVT?

For training on 18 RLBench tasks, with 100 demos per task, we use 8 V100 GPUs (16 GB memory each). The model trains in ~1 day.

Note that for fair comparison with PerAct, we used the same dataset, which means duplicate keyframes are loaded into the replay buffer. For other datasets, one could consider not doing so, which might further speed up training.

Q. Why do you use pe_fix=True in the rvt config?

For fair comparison with offical PerAct model, we use this setting. More detials about this can be found in PerAct code. For future, we recommend using pe_fix=False for language input.

Q. Why are the results for PerAct different from the PerAct paper?

In the PerAct paper, for each task, the best checkpoint is chosen based on the validation set performance. Hence, the model weights can be different for different tasks. We evaluate PerAct and RVT only on the final checkpoint, so that all tasks are strictly evaluated on the same model weights. Note that only the final model for PerAct has been released officially.

Q. Why is there a variance in performance on RLBench even when evaluting the same checkpoint?

We hypothesize that it is because of the sampling based planner used in RLBench, which could be the source of the randomization. Hence, we evaluate each checkpoint 5 times and report mean and variance.

Q. Why did you use a cosine decay learning rate scheduler instead of a fixed learning rate schedule as done in PerAct?

We found the cosine learning rate scheduler led to faster convergence for RVT. Training PerAct with our training hyper-parameters (cosine learning rate scheduler and same number of iterations) led to worse performance (in ~4 days of training time). Hence for Fig. 1, we used the official hyper-parameters for PerAct.

Q. For my use case, I want to render images at real camera locations (input camera poses) with PyTorch3D. Is it possible to do so and how can I do that?

Yes, it is possible to do so. A self-sufficient example is present here. Depending on your use case, the code may need be modified. Also note that 3D augmentation cannot be used while rendering images at real camera locations as it would change the pose of the camera with respect to the point cloud.

For questions and comments, please contact Ankit Goyal.

Acknowledgement

We sincerely thank the authors of the following repositories for sharing their code.

License

License Copyright Β© 2023, NVIDIA Corporation & affiliates. All rights reserved.

This work is made available under the Nvidia Source Code License. The pretrained RVT model is released under the CC-BY-NC-SA-4.0 license.

More Repositories

1

instant-ngp

Instant neural graphics primitives: lightning fast NeRF and more
Cuda
15,749
star
2

stylegan

StyleGAN - Official TensorFlow Implementation
Python
13,882
star
3

stylegan2

StyleGAN2 - Official TensorFlow Implementation
Python
10,740
star
4

SPADE

Semantic Image Synthesis with SPADE
Python
7,518
star
5

stylegan3

Official PyTorch implementation of StyleGAN3
Python
6,236
star
6

neuralangelo

Official implementation of "Neuralangelo: High-Fidelity Neural Surface Reconstruction" (CVPR 2023)
Python
4,316
star
7

imaginaire

NVIDIA's Deep Imagination Team's PyTorch Library
Python
3,941
star
8

stylegan2-ada-pytorch

StyleGAN2-ADA - Official PyTorch implementation
Python
3,866
star
9

tiny-cuda-nn

Lightning fast C++/CUDA neural network framework
C++
3,627
star
10

ffhq-dataset

Flickr-Faces-HQ Dataset (FFHQ)
Python
3,483
star
11

eg3d

Python
3,194
star
12

MUNIT

Multimodal Unsupervised Image-to-Image Translation
Python
2,564
star
13

SegFormer

Official PyTorch implementation of SegFormer
Python
2,521
star
14

nvdiffrec

Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".
Python
2,080
star
15

VILA

VILA - a multi-image visual language model with training, inference and evaluation recipe, deployable from cloud to edge (Jetson Orin and laptops)
Python
1,849
star
16

few-shot-vid2vid

Pytorch implementation for few-shot photorealistic video-to-video translation.
Python
1,780
star
17

stylegan2-ada

StyleGAN2 with adaptive discriminator augmentation (ADA) - Official TensorFlow implementation
Python
1,778
star
18

FUNIT

Translate images to unseen domains in the test time with few example images.
Python
1,545
star
19

PWC-Net

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume, CVPR 2018 (Oral)
Python
1,512
star
20

noise2noise

Noise2Noise: Learning Image Restoration without Clean Data - Official TensorFlow implementation of the ICML 2018 paper
Python
1,356
star
21

nvdiffrast

Nvdiffrast - Modular Primitives for High-Performance Differentiable Rendering
C++
1,348
star
22

alias-free-gan

Alias-Free GAN project website and code
1,320
star
23

edm

Elucidating the Design Space of Diffusion-Based Generative Models (EDM)
Python
1,303
star
24

prismer

The implementation of "Prismer: A Vision-Language Model with Multi-Task Experts".
Python
1,297
star
25

FoundationPose

[CVPR 2024 Highlight] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
Python
1,293
star
26

DG-Net

πŸ‘« Joint Discriminative and Generative Learning for Person Re-identification. CVPR'19 (Oral) πŸ‘«
Python
1,274
star
27

VoxFormer

Official PyTorch implementation of VoxFormer [CVPR 2023 Highlight]
Python
1,023
star
28

Deep_Object_Pose

Deep Object Pose Estimation (DOPE) – ROS inference (CoRL 2018)
Python
1,011
star
29

BundleSDF

[CVPR 2023] BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects
Python
989
star
30

NVAE

The Official PyTorch Implementation of "NVAE: A Deep Hierarchical Variational Autoencoder" (NeurIPS 2020 spotlight paper)
Python
889
star
31

ODISE

Official PyTorch implementation of ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models [CVPR 2023 Highlight]
Python
844
star
32

FasterViT

[ICLR 2024] Official PyTorch implementation of FasterViT: Fast Vision Transformers with Hierarchical Attention
Python
775
star
33

MambaVision

Official PyTorch Implementation of MambaVision: A Hybrid Mamba-Transformer Vision Backbone
Python
742
star
34

GroupViT

Official PyTorch implementation of GroupViT: Semantic Segmentation Emerges from Text Supervision, CVPR 2022.
Python
718
star
35

curobo

CUDA Accelerated Robot Library
Python
711
star
36

sionna

Sionna: An Open-Source Library for Next-Generation Physical Layer Research
Python
709
star
37

denoising-diffusion-gan

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs https://arxiv.org/abs/2112.07804
Python
660
star
38

InstantSplat

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds
Python
650
star
39

GA3C

Hybrid CPU/GPU implementation of the A3C algorithm for deep reinforcement learning.
Python
649
star
40

FB-BEV

Official PyTorch implementation of FB-BEV & FB-OCC - Forward-backward view transformation for vision-centric autonomous driving perception
Python
629
star
41

genvs

625
star
42

DoRA

[ICML2024 (Oral)] Official PyTorch implementation of DoRA: Weight-Decomposed Low-Rank Adaptation
Python
574
star
43

RADIO

Official repository for "AM-RADIO: Reduce All Domains Into One"
Python
566
star
44

EmerNeRF

PyTorch Implementation of EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision
Python
554
star
45

CALM

Python
527
star
46

EAGLE

EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
Python
526
star
47

Dancing2Music

Python
513
star
48

FourCastNet

Initial public release of code, data, and model weights for FourCastNet
Python
511
star
49

planercnn

PlaneRCNN detects and reconstructs piece-wise planar surfaces from a single RGB image
Python
502
star
50

pacnet

Pixel-Adaptive Convolutional Neural Networks (CVPR '19)
Python
490
star
51

edm2

Analyzing and Improving the Training Dynamics of Diffusion Models (EDM2)
Python
489
star
52

DeepInversion

Official PyTorch implementation of Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion (CVPR 2020)
Python
485
star
53

FAN

Official PyTorch implementation of Fully Attentional Networks
Python
464
star
54

DiffiT

[ECCV 2024] Official Repository for DiffiT: Diffusion Vision Transformers for Image Generation
443
star
55

GCVit

[ICML 2023] Official PyTorch implementation of Global Context Vision Transformers
Python
423
star
56

intrinsic3d

Intrinsic3D - High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting (ICCV 2017)
C++
411
star
57

nvdiffmodeling

Differentiable rasterization applied to 3D model simplification tasks
Python
404
star
58

flip

A tool for visualizing and communicating the errors in rendered images.
C++
375
star
59

nvdiffrecmc

Official code for the NeurIPS 2022 paper "Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising".
C
362
star
60

wetectron

Weakly-supervised object detection.
Python
355
star
61

GLAMR

[CVPR 2022 Oral] Official PyTorch Implementation of "GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras”.
Python
351
star
62

geomapnet

Geometry-Aware Learning of Maps for Camera Localization (CVPR2018)
Python
338
star
63

LSGM

The Official PyTorch Implementation of "LSGM: Score-based Generative Modeling in Latent Space" (NeurIPS 2021)
Python
338
star
64

timeloop

Timeloop performs modeling, mapping and code-generation for tensor algebra workloads on various accelerator architectures.
C++
325
star
65

ssn_superpixels

Superpixel Sampling Networks (ECCV2018)
Python
323
star
66

FreeSOLO

FreeSOLO for unsupervised instance segmentation, CVPR 2022
Python
313
star
67

long-video-gan

Official PyTorch implementation of LongVideoGAN
Python
308
star
68

trajdata

A unified interface to many trajectory forecasting datasets.
Python
301
star
69

contact_graspnet

Efficient 6-DoF Grasp Generation in Cluttered Scenes
Python
295
star
70

neuralrgbd

Neural RGB→D Sensing: Per-pixel depth and its uncertainty estimation from a monocular RGB video
Python
294
star
71

selfsupervised-denoising

High-Quality Self-Supervised Deep Image Denoising - Official TensorFlow implementation of the NeurIPS 2019 paper
Python
293
star
72

CF-3DGS

Python
286
star
73

sim-web-visualizer

Web Based Visualizer for Simulation Environments
Python
280
star
74

Taylor_pruning

Pruning Neural Networks with Taylor criterion in Pytorch
Python
279
star
75

mimicgen

This code corresponds to simulation environments used as part of the MimicGen project.
Python
275
star
76

metfaces-dataset

Python
272
star
77

few_shot_gaze

Pytorch implementation and demo of FAZE: Few-Shot Adaptive Gaze Estimation (ICCV 2019, oral)
Python
272
star
78

Hydra-MDP

269
star
79

splatnet

SPLATNet: Sparse Lattice Networks for Point Cloud Processing (CVPR2018)
Python
268
star
80

VILA-archive

VILA - A multi-image visual language model with training, inference and evaluation recipe, deployable from cloud to edge (Jetson Orin and laptops)
Python
267
star
81

MinVIS

Python
264
star
82

CenterPose

Single-Stage Keypoint-based Category-level Object Pose Estimation from an RGB Image (ICRA 2022)
Python
262
star
83

matchlib

SystemC/C++ library of commonly-used hardware functions and components for HLS.
C++
255
star
84

Minitron

A family of compressed models obtained via pruning and knowledge distillation
252
star
85

DiffRL

[ICLR 2022] Accelerated Policy Learning with Parallel Differentiable Simulation
Python
249
star
86

DiffPure

A new adversarial purification method that uses the forward and reverse processes of diffusion models to remove adversarial perturbations.
Python
249
star
87

STEP

STEP: Spatio-Temporal Progressive Learning for Video Action Detection. CVPR'19 (Oral)
Python
244
star
88

I2SB

Python
235
star
89

SCOPS

SCOPS: Self-Supervised Co-Part Segmentation (CVPR'19)
Python
221
star
90

UMR

Self-supervised Single-view 3D Reconstruction
Python
221
star
91

cule

CuLE: A CUDA port of the Atari Learning Environment (ALE)
C++
216
star
92

SSV

Pytorch implementation of SSV: Self-Supervised Viewpoint Learning from Image Collections (CVPR 2020)
Python
214
star
93

NVBit

210
star
94

AFNO-transformer

Adaptive FNO transformer - official Pytorch implementation
Python
207
star
95

6dof-graspnet

Implementation of 6-DoF GraspNet with tensorflow and python. This repo has been tested with python 2.7 and tensorflow 1.12.
Python
205
star
96

latentfusion

LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation
Python
197
star
97

nvbio

NVBIO is a library of reusable components designed to accelerate bioinformatics applications using CUDA.
C++
193
star
98

OmniDrive

Python
190
star
99

UnseenObjectClustering

Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation
Python
175
star
100

traffic-behavior-simulation

Python
173
star