ldkong1205/TranSVAE

Unsupervised Video Domain Adaptation for Action Recognition:
A Disentanglement Perspective

Pengfei Wei¹  Lingdong Kong^1,2  Xinghua Qu¹  Xiang Yin¹  Zhiqiang Xu³  Jing Jiang⁴  Zejun Ma^1
1ByteDance AI Lab   ²National University of Singapore   ³MBZUAI   ⁴University of Technology Sydney

About

TranSVAE is a disentanglement framework designed for unsupervised video domain adaptation. It aims at disentangling the domain information from the data during the adaptation process. We consider the generation of cross-domain videos from two sets of latent factors: one encoding the static domain-related information and another encoding the temporal and semantic-related information. Objectives are enforced to constrain these latent factors to achieve domain disentanglement and transfer.

Col1: Original sequences ("Human" $\mathcal{D}=\mathbf{P}_1$ and "Alien" $\mathcal{D}=\mathbf{P}_2$); Col2: Sequence reconstructions; Col3: Reconstructed sequences using $z_1^{\mathcal{D}},...,z_T^{\mathcal{D}}$; Col4: Domain transferred sequences with exchanged $z_d^{\mathcal{D}}$.

Visit our project page to explore more details. 🐾

Updates

• [2022.08] - TranSVAE achieves 1st place among the UDA leaderboards of UCF-HMDB, Jester, and Epic-Kitchens, based on Paper-with-Code.
• [2022.08] - Try a Gradio demo for domain disentanglement in TranSVAE at Hugging Face Spaces! 🤗
• [2022.08] - Our paper is available on arXiv, click here to check it out!

Outline

• Highlights
• Installation
• Data Preparation
• Getting Started
• Main Results
• TODO List
• License
• Acknowledgement
• Citation

Highlight

Conceptual Comparison
Graphical Model
Framework Overview

Installation

Please refer to INSTALL.md for the installation details.

Data Preparation

Please refer to DATA_PREPARE.md for the details to prepare the ¹UCF₁₀₁, ²HMDB₅₁, ³Jester, ⁴Epic-Kitchens, and ⁵Sprites datasets.

Getting Started

Please refer to GET_STARTED.md to learn more usage about this codebase.

Main Result

UCF₁₀₁ - HMDB₅₁

Jester

Epic-Kitchens

Ablation Study

UCF₁₀₁ → HMDB₅₁

HMDB₅₁ → UCF₁₀₁

Domain Transfer Example

Source (Original)	Target (Original)		Source (Original)	Target (Original)
Reconstruct ($\mathbf{z}_d^{\mathcal{S}}$ + $\mathbf{z}_t^{\mathcal{S}}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}}$ + $\mathbf{z}_t^{\mathcal{T}}$)		Reconstruct ($\mathbf{z}_d^{\mathcal{S}}$ + $\mathbf{z}_t^{\mathcal{S}}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}}$ + $\mathbf{z}_t^{\mathcal{T}}$)
Reconstruct ($\mathbf{z}_d^{\mathcal{S}} + \mathbf{0}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}} + \mathbf{0}$)		Reconstruct ($\mathbf{z}_d^{\mathcal{S}} + \mathbf{0}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}} + \mathbf{0}$)
Reconstruct ($\mathbf{0} + \mathbf{z}_t^{\mathcal{S}}$)	Reconstruct ($\mathbf{0} + \mathbf{z}_t^{\mathcal{T}}$)		Reconstruct ($\mathbf{0} + \mathbf{z}_t^{\mathcal{S}}$)	Reconstruct ($\mathbf{0} + \mathbf{z}_t^{\mathcal{T}}$)
Reconstruct ($\mathbf{z}_d^{\mathcal{S}} + \mathbf{z}_t^{\mathcal{T}}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}} + \mathbf{z}_t^{\mathcal{S}}$)		Reconstruct ($\mathbf{z}_d^{\mathcal{S}} + \mathbf{z}_t^{\mathcal{T}}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}} + \mathbf{z}_t^{\mathcal{S}}$)

TODO List

• Initial release. 🚀
• Add license. See here for more details.
• Add demo at Hugging Face Spaces.
• Add installation details.
• Add data preparation details.
• Add evaluation details.
• Add training details.

License

This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Acknowledgement

We acknowledge the use of the following public resources during the course of this work: ¹UCF₁₀₁, ²HMDB₅₁, ³Jester, ⁴Epic-Kitchens, ⁵Sprites, ⁶I3D, and ⁷TRN.

Citation

If you find this work helpful, please kindly consider citing our paper:

@ARTICLE{wei2022transvae,
  title={Unsupervised Video Domain Adaptation: A Disentanglement Perspective},
  author={Wei, Pengfei and Kong, Lingdong and Qu, Xinghua and Yin, Xiang and Xu, Zhiqiang and Jiang, Jing and Ma, Zejun},
  journal={arXiv preprint arXiv:2208.07365}, 
  year={2022},
}