Skinned Motion Retargeting with Residual Perception of Motion Semantics & Geometry

This is the code for the CVPR 2023 paper Skinned Motion Retargeting with Residual Perception of Motion Semantics & Geometry by Jiaxu Zhang, et al.

R2ET is a neural motion retargeting model that can preserve the source motion semantics and avoid interpenetration in the target motion.

Quick Start

1. Conda environment

conda create python=3.9 --name r2et
conda activate r2et

2. Install dependencies (Anaconda installation is recommended)

• Install the packages in requirements.txt and install PyTorch 1.10.0

pip install -r requirements.txt

• Install pytorch

conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=10.2 -c pytorch

3. Download and Install Blender

• Download and install from: https://www.blender.org/download/

4. Data preparation

Training data:

• Firstly, create an account on the Mixamo website.

• Next, download the fbx animation files for each character folder in ./datasets/mixamo/train_char/. The animation list can be refered to NKN. we collect 1952 non-overlapping motion sequences for training.

• Once the fbx files have been downloaded, run the following blender script to convert them into BVH files:

blender -b -P ./datasets/fbx2bvh.py

• Finally, preprocess the bvh files into npy files by running the following command:

python ./datasets/preprocess_q.py

• The shape information saved in ./datasets/mixamo/train_shape (already preprocessed) for each character's T-pose is preprocessed by:

blender -b -P ./datasets/extract_shape.py

5. Install the CUDA implementation of SDF function

• This CUDA implementation requires CUDA 10.2+ in the environment

cd ./outside-code/sdf
python setup.py install

Inference

Performing inference using bvh files:

python3 inference_bvh.py --config ./config/inference_bvh_cfg.yaml

Training

Skeleton-aware Network:

python3 train_skeleton_aware.py --config ./config/train_skeleton_aware.yaml

Shape-aware Network:

python3 train_shape_aware.py --config ./config/train_shape_aware.yaml

Visualization

The visualization parameters are in the ./visualization/options.py

cd ./visualization
blender -P visualize.py

Citation

• If you find this work helpful, please consider citing it as follows:

@inproceedings{zhang2023skinned,
  title={Skinned Motion Retargeting with Residual Perception of Motion Semantics \& Geometry},
  author={Zhang, Jiaxu and Weng, Junwu and Kang, Di and Zhao, Fang and Huang, Shaoli and Zhe, Xuefei and Bao, Linchao and Shan, Ying and Wang, Jue and Tu, Zhigang},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={13864--13872},
  year={2023}
}

Acknowledgments

Thanks to PMnet, SAN and NKN, our code is partially borrowing from them.