CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction

ICCV, 2021
Lixin Yang · Xinyu Zhan · Kailin Li · Wenqiang Xu · Jiefeng Li · Cewu Lu

This repo contains model, demo, and test codes.

Installation

Following the Installation Instruction to setup environment, assets, datasets and models.

Demo

Notice: require active screen.

visualize GeO fitting pipeline

We create a FHBExample dataset in hocontact/hodatasets/fhb_example.py that only contains 10 samples to demonstrate our pipeline.

# Only support single GPU !
$ python scripts/run_demo.py \
    --gpu 0 \
    --init_ckpt CPF_checkpoints/picr/fhb/checkpoint_200.pth.tar \
    --honet_mano_fhb_hand

Press q in the "runtime hand" window to start fitting

visualize A-MANO'S anchor position

This demo shows the anochrs positions on MANO hand's surface

$ python scripts/recover_anchor.py --render

Evaluation

We provide shell srcipts for evaluating on FHB, HO3Dv1 and v2.

FHB dataset

dump the results of HoNet and PiCR:

# recommend 2 GPUs
$ export GPU_ID=0,1 && sh ./scripts/dump_HoNetPiCR_FHB.sh

and fit GeO optimizer:

# setting 1: hand-only 
# recommend 4 GPUs
$ export GPU_ID=0,1,2,3 && sh ./scripts/fit_GeO_handonly_FHB.sh

# setting 2: hand-obj
$ export GPU_ID=0,1,2,3 && sh ./scripts/fit_GeO_handobj_FHB.sh

HO3Dv1

dump the results of HoNet and PiCR:

# recommend 2 GPUs
$ export GPU_ID=0,1 && sh ./scripts/dump_HoNetPiCR_HO3Dv1.sh

and fit GeO optimizer:

# hand-only
# recommend 8 GPUs
$ export GPU_ID=0,1,2,3,4,5,6,7 && sh ./scripts/fit_GeO_handonly_HO3Dv1.sh

# hand-obj
# recommend 8 GPUs
$ export GPU_ID=0,1,2,3,4,5,6,7 && sh ./scripts/fit_GeO_handobj_HO3Dv1.sh

HO3Dv2 (version 2)

dump the results of HoNet and PiCR:

# recommend 2 GPUs
$ export GPU_ID=0,1 && sh ./scripts/dump_HoNetPiCR_HO3Dv2.sh

and fit GeO optimizer:

# recommend 8 GPUs
$ export GPU_ID=0,1,2,3,4,5,6,7 && sh ./scripts/fit_GeO_handobj_HO3Dv2.sh

evaluation results

Above scripts may take a while ( ~ 1 day ). We also provide the results in fitting_res.

Anatomical Constrained A-MANO

We provide pytorch implementation of our Anatomical Constrained MANO in lixiny/manopth, which is modified from the original hassony2/manopth.

TODO

• testing code and pretrained models
  • HoNet (FHB, HO3Dv1/v2)
  • PiCR (FHB, HO3Dv1/v2)
• fitting code of GeO, both hand-only and hand-object (FHB, HO3Dv1/v2)
• training code
• contact region visualization

Citation

If you find this work helpful, please consider citing us:

@inproceedings{yang2021cpf,
    title={{CPF}: Learning a Contact Potential Field to Model the Hand-Object Interaction},
    author={Yang, Lixin and Zhan, Xinyu and Li, Kailin and Xu, Wenqiang and Li, Jiefeng and Lu, Cewu},
    booktitle={ICCV},
    year={2021}
}