Zielon/INSTA-pytorch

INSTA - Instant Volumetric Head Avatars (pytorch)

Wojciech Zielonka, Timo Bolkart, Justus Thies

Max Planck Institute for Intelligent Systems, Tübingen, Germany

Video  Paper  Project Website  Dataset  Face Tracker  Email

This repository is based on torch-ngp and implements most of the C++ version. Please note that this is not 1:1 port of C++ version and some features are missing thus the speed of training, rendering, and the quality are not exactly the same like in the case of the paper version of INSTA. Therefore, for any comparisons please use the C++ version.

Installation

Please follow the installation from torch-ngp. This implementation requires to use cuda ray marching, which is enabled by --cuda-ray. Therefore, C++ extension must be built. Moreover, for a fast nearest neighbor search we are using BVH repository.

First create the environment.

conda create -n insta-pytorch python=3.9
conda activate insta-pytorch
pip install -r requirements.txt

After that use the install.sh script to compile all the required libraries and prepare the workbench.

Usage

The dataset structure is compatible with the C++ version. In order to generate a new sequence input follow instructions. The hardware requirement are the same as in the case of C++ version.

The released avatars are compatible with training, however, the checkpoints were generated for the C++ version.

# Offscreen rendering
python main_insta.py data/nerf/wojtek --workspace workspace/wojtek -O --tcnn

# GUI
python main_insta.py data/nerf/wojtek --workspace workspace/wojtek -O --tcnn --gui

Using GUI in the Menu/Options you can control the selected mesh.

Citation

If you use this project in your research please cite INSTA:

@proceedings{INSTA:CVPR2023,
  author = {Zielonka, Wojciech and Bolkart, Timo and Thies, Justus},
  title = {Instant Volumetric Head Avatars},
  journal = {Conference on Computer Vision and Pattern Recognition},
  year = {2023}
}