NVlabs/CALM NVlabs/CALM

  • Stars
    star
    527
  • Rank 84,091 (Top 2 %)
  • Language
    Python
  • License
    Other
  • Created over 1 year ago
  • Updated over 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
NVlabs/CALM
github

NVlabs

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Conditional Adversarial Latent Models

Code accompanying the paper: "CALM: Conditional Adversarial Latent Models for Directable Virtual Characters"
Skills

CALM builds upon, and borrows code from, Adversarial Skill Embeddings (Peng et. al., 2022, ASE).

Installation

Download Isaac Gym from the website, then follow the installation instructions.

Once Isaac Gym is installed, install the external dependencies for this repo:

pip install -r requirements.txt

CALM

Pre-Training

First, a CALM model can be trained to imitate a dataset of motions clips using the following command:

python calm/run.py --task HumanoidAMPGetup --cfg_env calm/data/cfg/humanoid_calm_sword_shield_getup.yaml --cfg_train calm/data/cfg/train/rlg/calm_humanoid.yaml --motion_file calm/data/motions/reallusion_sword_shield/dataset_reallusion_sword_shield.yaml --headless  --track

--motion_file can be used to specify a dataset of motion clips that the model should imitate. The task HumanoidAMPGetup will train a model to imitate a dataset of motion clips and get up after falling. Over the course of training, the latest checkpoint Humanoid.pth will be regularly saved to output/, along with a Tensorboard log. --headless is used to disable visualizations and --track is used for tracking using weights and biases. If you want to view the simulation, simply remove this flag. To test a trained model, use the following command:

python calm/run.py --test --task HumanoidAMPGetup --num_envs 16 --cfg_env calm/data/cfg/humanoid_calm_sword_shield_getup.yaml --cfg_train calm/data/cfg/train/rlg/calm_humanoid.yaml --motion_file calm/data/motions/reallusion_sword_shield/dataset_reallusion_sword_shield.yaml --checkpoint [path_to_calm_checkpoint]

You can also test the robustness of the model with --task HumanoidPerturb, which will throw projectiles at the character.

 

Precision-Training

After the CALM low-level controller has been trained, it can be used to train style-constrained-locomotion controllers. The following command will use a pre-trained CALM model to perform a target heading task:

python calm/run.py --task HumanoidHeadingConditioned --cfg_env calm/data/cfg/humanoid_sword_shield_heading_conditioned.yaml --cfg_train calm/data/cfg/train/rlg/hrl_humanoid_style_control.yaml --motion_file calm/data/motions/reallusion_sword_shield/dataset_reallusion_sword_shield_fsm_movements.yaml --llc_checkpoint [path_to_llc_checkpoint] --headless --track

--llc_checkpoint specifies the checkpoint to use for the low-level controller. A pre-trained CALM low-level controller is available in calm/data/models/calm_llc_reallusion_sword_shield.pth.

To test a trained model, use the following command:

python calm/run.py --test --task HumanoidHeadingConditioned --num_envs 16 --cfg_env calm/data/cfg/humanoid_sword_shield_heading_conditioned.yaml --cfg_train calm/data/cfg/train/rlg/hrl_humanoid.yaml --motion_file calm/data/motions/reallusion_sword_shield/dataset_reallusion_sword_shield_fsm_movements.yaml --llc_checkpoint [path_to_llc_checkpoint] --checkpoint [path_to_hlc_checkpoint]

 

Task-Solving (Inference -- no training!)

The CALM low-level controller and the high-level locomotion controller can be combined to solve tasks without further trianing. This phase is inference only.

python calm/run.py --test --task HumanoidStrikeFSM --num_envs 16 --cfg_env calm/data/cfg/humanoid_sword_shield_strike_fsm.yaml --cfg_train calm/data/cfg/train/rlg/hrl_humanoid_fsm.yaml --motion_file calm/data/motions/reallusion_sword_shield/dataset_reallusion_sword_shield_fsm_movements.yaml --llc_checkpoint [path_to_llc_checkpoint] --checkpoint [path_to_hlc_checkpoint]

--llc_checkpoint specifies the checkpoint to use for the low-level controller. A pre-trained CALM low-level controller is available in calm/data/models/calm_llc_reallusion_sword_shield.pth. --checkpoint specified the checkpoint to use for the precision-trained high-level controller. A pre-trained high-level precision-trained controller is available in calm/data/models/calm_hlc_precision_trained_reallusion_sword_shield.pth.

The built-in tasks and their respective config files are:

HumanoidStrikeFSM: calm/data/cfg/humanoid_sword_shield_strike_fsm.yaml
HumanoidLocationFSM: calm/data/cfg/humanoid_sword_shield_location_fsm.yaml

 

 

Task-Training

In addition to precision training, a high-level controller can also be trained to directly solve tasks. The following command will use a pre-trained CALM model to perform a target heading task:

python calm/run.py --task HumanoidHeading --cfg_env calm/data/cfg/humanoid_sword_shield_heading.yaml --cfg_train calm/data/cfg/train/rlg/hrl_humanoid.yaml --motion_file calm/data/motions/reallusion_sword_shield/RL_Avatar_Idle_Ready_Motion.npy --llc_checkpoint [path_to_llc_checkpoint] --headless --track

--llc_checkpoint specifies the checkpoint to use for the low-level controller. A pre-trained CALM low-level controller is available in calm/data/models/calm_llc_reallusion_sword_shield.ckpt. --task specifies the task that the character should perform, and --cfg_env specifies the environment configurations for that task. The built-in tasks and their respective config files are:

HumanoidReach: calm/data/cfg/humanoid_sword_shield_reach.yaml
HumanoidHeading: calm/data/cfg/humanoid_sword_shield_heading.yaml
HumanoidLocation: calm/data/cfg/humanoid_sword_shield_location.yaml
HumanoidStrike: calm/data/cfg/humanoid_sword_shield_strike.yaml

To test a trained model, use the following command:

python calm/run.py --test --task HumanoidHeading --num_envs 16 --cfg_env calm/data/cfg/humanoid_sword_shield_heading.yaml --cfg_train calm/data/cfg/train/rlg/hrl_humanoid.yaml --motion_file calm/data/motions/reallusion_sword_shield/RL_Avatar_Idle_Ready_Motion.npy --llc_checkpoint [path_to_llc_checkpoint] --checkpoint [path_to_hlc_checkpoint]

 

 

AMP

We also provide an implementation of Adversarial Motion Priors (https://xbpeng.github.io/projects/AMP/index.html). A model can be trained to imitate a given reference motion using the following command:

python calm/run.py --task HumanoidAMP --cfg_env calm/data/cfg/humanoid_sword_shield.yaml --cfg_train calm/data/cfg/train/rlg/amp_humanoid.yaml --motion_file calm/data/motions/reallusion_sword_shield/sword_shield/RL_Avatar_Atk_2xCombo01_Motion.npy --headless  --track

The trained model can then be tested with:

python calm/run.py --test --task HumanoidAMP --num_envs 16 --cfg_env calm/data/cfg/humanoid_sword_shield.yaml --cfg_train calm/data/cfg/train/rlg/amp_humanoid.yaml --motion_file calm/data/motions/reallusion_sword_shield/sword_shield/RL_Avatar_Atk_2xCombo01_Motion.npy --checkpoint [path_to_amp_checkpoint]

 

 

Motion Data

Motion clips are located in calm/data/motions/. Individual motion clips are stored as .npy files. Motion datasets are specified by .yaml files, which contains a list of motion clips to be included in the dataset. Motion clips can be visualized with the following command:

python calm/run.py --test --task HumanoidViewMotion --num_envs 2 --cfg_env calm/data/cfg/humanoid_sword_shield.yaml --cfg_train calm/data/cfg/train/rlg/amp_humanoid.yaml --motion_file calm/data/motions/reallusion_sword_shield/sword_shield/RL_Avatar_Atk_2xCombo01_Motion.npy

--motion_file can be used to visualize a single motion clip .npy or a motion dataset .yaml.

If you want to retarget new motion clips to the character, you can take a look at an example retargeting script in calm/poselib/retarget_motion.py.

More Repositories

1

instant-ngp

Instant neural graphics primitives: lightning fast NeRF and more
Cuda
15,749
star
2

stylegan

StyleGAN - Official TensorFlow Implementation
Python
13,882
star
3

stylegan2

StyleGAN2 - Official TensorFlow Implementation
Python
10,740
star
4

SPADE

Semantic Image Synthesis with SPADE
Python
7,518
star
5

stylegan3

Official PyTorch implementation of StyleGAN3
Python
6,236
star
6

neuralangelo

Official implementation of "Neuralangelo: High-Fidelity Neural Surface Reconstruction" (CVPR 2023)
Python
4,316
star
7

imaginaire

NVIDIA's Deep Imagination Team's PyTorch Library
Python
3,941
star
8

stylegan2-ada-pytorch

StyleGAN2-ADA - Official PyTorch implementation
Python
3,866
star
9

tiny-cuda-nn

Lightning fast C++/CUDA neural network framework
C++
3,627
star
10

ffhq-dataset

Flickr-Faces-HQ Dataset (FFHQ)
Python
3,483
star
11

eg3d

Python
3,194
star
12

MUNIT

Multimodal Unsupervised Image-to-Image Translation
Python
2,564
star
13

SegFormer

Official PyTorch implementation of SegFormer
Python
2,521
star
14

nvdiffrec

Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".
Python
2,080
star
15

VILA

VILA - a multi-image visual language model with training, inference and evaluation recipe, deployable from cloud to edge (Jetson Orin and laptops)
Python
1,849
star
16

few-shot-vid2vid

Pytorch implementation for few-shot photorealistic video-to-video translation.
Python
1,780
star
17

stylegan2-ada

StyleGAN2 with adaptive discriminator augmentation (ADA) - Official TensorFlow implementation
Python
1,778
star
18

FUNIT

Translate images to unseen domains in the test time with few example images.
Python
1,545
star
19

PWC-Net

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume, CVPR 2018 (Oral)
Python
1,512
star
20

noise2noise

Noise2Noise: Learning Image Restoration without Clean Data - Official TensorFlow implementation of the ICML 2018 paper
Python
1,356
star
21

nvdiffrast

Nvdiffrast - Modular Primitives for High-Performance Differentiable Rendering
C++
1,348
star
22

alias-free-gan

Alias-Free GAN project website and code
1,320
star
23

edm

Elucidating the Design Space of Diffusion-Based Generative Models (EDM)
Python
1,303
star
24

prismer

The implementation of "Prismer: A Vision-Language Model with Multi-Task Experts".
Python
1,297
star
25

FoundationPose

[CVPR 2024 Highlight] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
Python
1,293
star
26

DG-Net

👫 Joint Discriminative and Generative Learning for Person Re-identification. CVPR'19 (Oral) 👫
Python
1,274
star
27

VoxFormer

Official PyTorch implementation of VoxFormer [CVPR 2023 Highlight]
Python
1,023
star
28

Deep_Object_Pose

Deep Object Pose Estimation (DOPE) – ROS inference (CoRL 2018)
Python
1,011
star
29

BundleSDF

[CVPR 2023] BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects
Python
989
star
30

NVAE

The Official PyTorch Implementation of "NVAE: A Deep Hierarchical Variational Autoencoder" (NeurIPS 2020 spotlight paper)
Python
889
star
31

ODISE

Official PyTorch implementation of ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models [CVPR 2023 Highlight]
Python
844
star
32

FasterViT

[ICLR 2024] Official PyTorch implementation of FasterViT: Fast Vision Transformers with Hierarchical Attention
Python
775
star
33

MambaVision

Official PyTorch Implementation of MambaVision: A Hybrid Mamba-Transformer Vision Backbone
Python
742
star
34

GroupViT

Official PyTorch implementation of GroupViT: Semantic Segmentation Emerges from Text Supervision, CVPR 2022.
Python
718
star
35

curobo

CUDA Accelerated Robot Library
Python
711
star
36

sionna

Sionna: An Open-Source Library for Next-Generation Physical Layer Research
Python
709
star
37

denoising-diffusion-gan

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs https://arxiv.org/abs/2112.07804
Python
660
star
38

InstantSplat

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds
Python
650
star
39

GA3C

Hybrid CPU/GPU implementation of the A3C algorithm for deep reinforcement learning.
Python
649
star
40

FB-BEV

Official PyTorch implementation of FB-BEV & FB-OCC - Forward-backward view transformation for vision-centric autonomous driving perception
Python
629
star
41

genvs

625
star
42

DoRA

[ICML2024 (Oral)] Official PyTorch implementation of DoRA: Weight-Decomposed Low-Rank Adaptation
Python
574
star
43

RADIO

Official repository for "AM-RADIO: Reduce All Domains Into One"
Python
566
star
44

EmerNeRF

PyTorch Implementation of EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision
Python
554
star
45

EAGLE

EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
Python
526
star
46

Dancing2Music

Python
513
star
47

FourCastNet

Initial public release of code, data, and model weights for FourCastNet
Python
511
star
48

planercnn

PlaneRCNN detects and reconstructs piece-wise planar surfaces from a single RGB image
Python
502
star
49

pacnet

Pixel-Adaptive Convolutional Neural Networks (CVPR '19)
Python
490
star
50

edm2

Analyzing and Improving the Training Dynamics of Diffusion Models (EDM2)
Python
489
star
51

DeepInversion

Official PyTorch implementation of Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion (CVPR 2020)
Python
485
star
52

FAN

Official PyTorch implementation of Fully Attentional Networks
Python
464
star
53

DiffiT

[ECCV 2024] Official Repository for DiffiT: Diffusion Vision Transformers for Image Generation
443
star
54

GCVit

[ICML 2023] Official PyTorch implementation of Global Context Vision Transformers
Python
423
star
55

intrinsic3d

Intrinsic3D - High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting (ICCV 2017)
C++
411
star
56

nvdiffmodeling

Differentiable rasterization applied to 3D model simplification tasks
Python
404
star
57

flip

A tool for visualizing and communicating the errors in rendered images.
C++
375
star
58

nvdiffrecmc

Official code for the NeurIPS 2022 paper "Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising".
C
362
star
59

wetectron

Weakly-supervised object detection.
Python
355
star
60

GLAMR

[CVPR 2022 Oral] Official PyTorch Implementation of "GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras”.
Python
351
star
61

geomapnet

Geometry-Aware Learning of Maps for Camera Localization (CVPR2018)
Python
338
star
62

LSGM

The Official PyTorch Implementation of "LSGM: Score-based Generative Modeling in Latent Space" (NeurIPS 2021)
Python
338
star
63

timeloop

Timeloop performs modeling, mapping and code-generation for tensor algebra workloads on various accelerator architectures.
C++
325
star
64

ssn_superpixels

Superpixel Sampling Networks (ECCV2018)
Python
323
star
65

FreeSOLO

FreeSOLO for unsupervised instance segmentation, CVPR 2022
Python
313
star
66

long-video-gan

Official PyTorch implementation of LongVideoGAN
Python
308
star
67

trajdata

A unified interface to many trajectory forecasting datasets.
Python
301
star
68

contact_graspnet

Efficient 6-DoF Grasp Generation in Cluttered Scenes
Python
295
star
69

neuralrgbd

Neural RGB→D Sensing: Per-pixel depth and its uncertainty estimation from a monocular RGB video
Python
294
star
70

selfsupervised-denoising

High-Quality Self-Supervised Deep Image Denoising - Official TensorFlow implementation of the NeurIPS 2019 paper
Python
293
star
71

CF-3DGS

Python
286
star
72

sim-web-visualizer

Web Based Visualizer for Simulation Environments
Python
280
star
73

Taylor_pruning

Pruning Neural Networks with Taylor criterion in Pytorch
Python
279
star
74

mimicgen

This code corresponds to simulation environments used as part of the MimicGen project.
Python
275
star
75

metfaces-dataset

Python
272
star
76

few_shot_gaze

Pytorch implementation and demo of FAZE: Few-Shot Adaptive Gaze Estimation (ICCV 2019, oral)
Python
272
star
77

Hydra-MDP

269
star
78

splatnet

SPLATNet: Sparse Lattice Networks for Point Cloud Processing (CVPR2018)
Python
268
star
79

VILA-archive

VILA - A multi-image visual language model with training, inference and evaluation recipe, deployable from cloud to edge (Jetson Orin and laptops)
Python
267
star
80

RVT

Official Code for RVT-2 and RVT
Jupyter Notebook
265
star
81

MinVIS

Python
264
star
82

CenterPose

Single-Stage Keypoint-based Category-level Object Pose Estimation from an RGB Image (ICRA 2022)
Python
262
star
83

matchlib

SystemC/C++ library of commonly-used hardware functions and components for HLS.
C++
255
star
84

Minitron

A family of compressed models obtained via pruning and knowledge distillation
252
star
85

DiffRL

[ICLR 2022] Accelerated Policy Learning with Parallel Differentiable Simulation
Python
249
star
86

DiffPure

A new adversarial purification method that uses the forward and reverse processes of diffusion models to remove adversarial perturbations.
Python
249
star
87

STEP

STEP: Spatio-Temporal Progressive Learning for Video Action Detection. CVPR'19 (Oral)
Python
244
star
88

I2SB

Python
235
star
89

SCOPS

SCOPS: Self-Supervised Co-Part Segmentation (CVPR'19)
Python
221
star
90

UMR

Self-supervised Single-view 3D Reconstruction
Python
221
star
91

cule

CuLE: A CUDA port of the Atari Learning Environment (ALE)
C++
216
star
92

SSV

Pytorch implementation of SSV: Self-Supervised Viewpoint Learning from Image Collections (CVPR 2020)
Python
214
star
93

NVBit

210
star
94

AFNO-transformer

Adaptive FNO transformer - official Pytorch implementation
Python
207
star
95

6dof-graspnet

Implementation of 6-DoF GraspNet with tensorflow and python. This repo has been tested with python 2.7 and tensorflow 1.12.
Python
205
star
96

latentfusion

LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation
Python
197
star
97

nvbio

NVBIO is a library of reusable components designed to accelerate bioinformatics applications using CUDA.
C++
193
star
98

OmniDrive

Python
190
star
99

UnseenObjectClustering

Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation
Python
175
star
100

traffic-behavior-simulation

Python
173
star