thuml/Transfer-Learning-Library thuml/Transfer-Learning-Library

  • Stars
    star
    3,318
  • Rank 13,406 (Top 0.3 %)
  • Language
    Python
  • License
    MIT License
  • Created over 4 years ago
  • Updated 4 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
thuml/Transfer-Learning-Library
github

thuml

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Transfer Learning Library for Domain Adaptation, Task Adaptation, and Domain Generalization

Transfer Learning Library

Introduction

TLlib is an open-source and well-documented library for Transfer Learning. It is based on pure PyTorch with high performance and friendly API. Our code is pythonic, and the design is consistent with torchvision. You can easily develop new algorithms, or readily apply existing algorithms.

Our API is divided by methods, which include:

  • domain alignment methods (tllib.aligment)
  • domain translation methods (tllib.translation)
  • self-training methods (tllib.self_training)
  • regularization methods (tllib.regularization)
  • data reweighting/resampling methods (tllib.reweight)
  • model ranking/selection methods (tllib.ranking)
  • normalization-based methods (tllib.normalization)

We provide many example codes in the directory examples, which is divided by learning setups. Currently, the supported learning setups include:

  • DA (domain adaptation)
  • TA (task adaptation, also known as finetune)
  • OOD (out-of-distribution generalization, also known as DG / domain generalization)
  • SSL (semi-supervised learning)
  • Model Selection

Our supported tasks include: classification, regression, object detection, segmentation, keypoint detection, and so on.

Updates

2022.9

We support installing TLlib via pip, which is experimental currently.

pip install -i https://test.pypi.org/simple/ tllib==0.4

2022.8

We release v0.4 of TLlib. Previous versions of TLlib can be found here. In v0.4, we add implementations of the following methods:

Besides, we maintain a collection of awesome papers in Transfer Learning in another repo A Roadmap for Transfer Learning.

2022.2

We adjusted our API following our survey Transferablity in Deep Learning.

Supported Methods

The currently supported algorithms include:

Domain Adaptation for Classification [Code]
Domain Adaptation for Object Detection [Code]
  • CycleGAN - Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks [ICCV 2017] [Code]
  • D-adapt - Decoupled Adaptation for Cross-Domain Object Detection [ICLR 2022] [Code]
Domain Adaptation for Semantic Segmentation [Code]
  • CycleGAN - Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks [ICCV 2017] [Code]
  • CyCADA - Cycle-Consistent Adversarial Domain Adaptation [ICML 2018] [Code]
  • ADVENT - Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation [CVPR 2019] [Code]
  • FDA - Fourier Domain Adaptation for Semantic Segmentation [CVPR 2020] [Code]
Domain Adaptation for Keypoint Detection [Code]
  • RegDA - Regressive Domain Adaptation for Unsupervised Keypoint Detection [CVPR 2021] [Code]
Domain Adaptation for Person Re-identification [Code]
  • IBN-Net - Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net [ECCV 2018]
  • MMT - Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification [ICLR 2020] [Code]
  • SPGAN - Similarity Preserving Generative Adversarial Network [CVPR 2018] [Code]
Partial Domain Adaptation [Code]
  • IWAN - Importance Weighted Adversarial Nets for Partial Domain Adaptation[CVPR 2018] [Code]
  • AFN - Larger Norm More Transferable: An Adaptive Feature Norm Approach for Unsupervised Domain Adaptation [ICCV 2019] [Code]
Open-set Domain Adaptation [Code]
Domain Generalization for Classification [Code]
  • IBN-Net - Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net [ECCV 2018]
  • MixStyle - Domain Generalization with MixStyle [ICLR 2021] [Code]
  • MLDG - Learning to Generalize: Meta-Learning for Domain Generalization [AAAI 2018] [Code]
  • IRM - Invariant Risk Minimization [ArXiv] [Code]
  • VREx - Out-of-Distribution Generalization via Risk Extrapolation [ICML 2021] [Code]
  • GroupDRO - Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization [ArXiv] [Code]
  • Deep CORAL - Correlation Alignment for Deep Domain Adaptation [ECCV 2016] [Code]
Domain Generalization for Person Re-identification [Code]
  • IBN-Net - Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net [ECCV 2018]
  • MixStyle - Domain Generalization with MixStyle [ICLR 2021] [Code]
Task Adaptation (Fine-Tuning) for Image Classification [Code]
  • L2-SP - Explicit inductive bias for transfer learning with convolutional networks [ICML 2018] [Code]
  • BSS - Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer Learning [NIPS 2019] [Code]
  • DELTA - DEep Learning Transfer using Fea- ture Map with Attention for convolutional networks [ICLR 2019] [Code]
  • Co-Tuning - Co-Tuning for Transfer Learning [NIPS 2020] [Code]
  • StochNorm - Stochastic Normalization [NIPS 2020] [Code]
  • LWF - Learning Without Forgetting [ECCV 2016] [Code]
  • Bi-Tuning - Bi-tuning of Pre-trained Representations [ArXiv] [Code]
Pre-trained Model Selection [Code]
  • H-Score - An Information-theoretic Approach to Transferability in Task Transfer Learning [ICIP 2019] [Code]
  • NCE - Negative Conditional Entropy in `Transferability and Hardness of Supervised Classification Tasks [ICCV 2019] [Code]
  • LEEP - LEEP: A New Measure to Evaluate Transferability of Learned Representations [ICML 2020] [Code]
  • LogME - Log Maximum Evidence in `LogME: Practical Assessment of Pre-trained Models for Transfer Learning [ICML 2021] [Code]
Semi-Supervised Learning for Classification [Code]
  • Pseudo Label - Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks [ICML 2013] [Code]
  • Pi Model - Temporal Ensembling for Semi-Supervised Learning [ICLR 2017] [Code]
  • Mean Teacher - Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results [NIPS 2017] [Code]
  • Noisy Student - Self-Training With Noisy Student Improves ImageNet Classification [CVPR 2020] [Code]
  • UDA - Unsupervised Data Augmentation for Consistency Training [NIPS 2020] [Code]
  • FixMatch - Simplifying Semi-Supervised Learning with Consistency and Confidence [NIPS 2020] [Code]
  • Self-Tuning - Self-Tuning for Data-Efficient Deep Learning [ICML 2021] [Code]
  • FlexMatch - FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling [NIPS 2021] [Code]
  • DebiasMatch - Debiased Learning From Naturally Imbalanced Pseudo-Labels [CVPR 2022] [Code]
  • DST - Debiased Self-Training for Semi-Supervised Learning [NIPS 2022 Oral] [Code]

Installation

Install from Source Code
  • Please git clone the library first. Then, run the following commands to install tllib and all the dependency.
python setup.py install
pip install -r requirements.txt
Install via pip
  • Installing via pip is currently experimental.
pip install -i https://test.pypi.org/simple/ tllib==0.4

Documentation

You can find the API documentation on the website: Documentation.

Usage

You can find examples in the directory examples. A typical usage is

# Train a DANN on Office-31 Amazon -> Webcam task using ResNet 50.
# Assume you have put the datasets under the path `data/office-31`, 
# or you are glad to download the datasets automatically from the Internet to this path
python dann.py data/office31 -d Office31 -s A -t W -a resnet50  --epochs 20

Contributing

We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion. If you plan to contribute new features, utility functions or extensions, please first open an issue and discuss the feature with us.

Disclaimer on Datasets

This is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have licenses to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license.

If you're a dataset owner and wish to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thanks for your contribution to the ML community!

Contact

If you have any problem with our code or have some suggestions, including the future feature, feel free to contact

or describe it in Issues.

For Q&A in Chinese, you can choose to ask questions here before sending an email. 迁移学习算法库答疑专区

Citation

If you use this toolbox or benchmark in your research, please cite this project.

@misc{jiang2022transferability,
      title={Transferability in Deep Learning: A Survey}, 
      author={Junguang Jiang and Yang Shu and Jianmin Wang and Mingsheng Long},
      year={2022},
      eprint={2201.05867},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

@misc{tllib,
    author = {Junguang Jiang, Baixu Chen, Bo Fu, Mingsheng Long},
    title = {Transfer-Learning-library},
    year = {2020},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/thuml/Transfer-Learning-Library}},
}

Acknowledgment

We would like to thank School of Software, Tsinghua University and The National Engineering Laboratory for Big Data Software for providing such an excellent ML research platform.

More Repositories

1

Time-Series-Library

A Library for Advanced Deep Time Series Models.
Python
6,099
star
2

Autoformer

About Code release for "Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting" (NeurIPS 2021), https://arxiv.org/abs/2106.13008
Jupyter Notebook
1,882
star
3

iTransformer

Official implementation for "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting" (ICLR 2024 Spotlight), https://openreview.net/forum?id=JePfAI8fah
Python
1,097
star
4

Anomaly-Transformer

About Code release for "Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy" (ICLR 2022 Spotlight), https://openreview.net/forum?id=LzQQ89U1qm_
Python
708
star
5

TimesNet

About Code release for "TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis" (ICLR 2023), https://openreview.net/pdf?id=ju_Uqw384Oq
673
star
6

awesome-multi-task-learning

2024 up-to-date list of DATASETS, CODEBASES and PAPERS on Multi-Task Learning (MTL), from Machine Learning perspective.
625
star
7

Xlearn

Transfer Learning Library
Jupyter Notebook
459
star
8

Nonstationary_Transformers

Code release for "Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting" (NeurIPS 2022), https://arxiv.org/abs/2205.14415
Python
455
star
9

predrnn-pytorch

Official implementation for NIPS'17 paper: PredRNN: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal LSTMs.
Python
438
star
10

depyf

depyf is a tool to help you understand and adapt to PyTorch compiler torch.compile.
Python
407
star
11

CDAN

Code release for "Conditional Adversarial Domain Adaptation" (NIPS 2018)
Jupyter Notebook
392
star
12

Flowformer

About Code release for "Flowformer: Linearizing Transformers with Conservation Flows" (ICML 2022), https://arxiv.org/pdf/2202.06258.pdf
Python
300
star
13

Universal-Domain-Adaptation

Code release for Universal Domain Adaptation(CVPR 2019)
Python
272
star
14

HashNet

Code release for "HashNet: Deep Learning to Hash by Continuation" (ICCV 2017)
Jupyter Notebook
240
star
15

Large-Time-Series-Model

Official code, datasets and checkpoints for "Timer: Generative Pre-trained Transformers Are Large Time Series Models" (ICML 2024)
Python
214
star
16

LogME

Code release for "LogME: Practical Assessment of Pre-trained Models for Transfer Learning" (ICML 2021) and Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs (JMLR 2022)
Python
200
star
17

Koopa

Code release for "Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors" (NeurIPS 2023), https://arxiv.org/abs/2305.18803
Python
173
star
18

Corrformer

About code release of "Interpretable Weather Forecasting for Worldwide Stations with a Unified Deep Model", Nature Machine Intelligence, 2023. https://www.nature.com/articles/s42256-023-00667-9
Python
155
star
19

A-Roadmap-for-Transfer-Learning

151
star
20

MDD

Code released for ICML 2019 paper "Bridging Theory and Algorithm for Domain Adaptation".
Python
129
star
21

Self-Tuning

Code release for "Self-Tuning for Data-Efficient Deep Learning" (ICML 2021)
Python
109
star
22

SimMTM

About Code release for "SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling" (NeurIPS 2023 Spotlight), https://arxiv.org/abs/2302.00861
Python
103
star
23

PADA

Code release for "Partial Adversarial Domain Adaptation" (ECCV 2018)
Python
100
star
24

Batch-Spectral-Penalization

Code release for Transferability vs. Discriminability: Batch Spectral Penalization for Adversarial Domain Adaptation (ICML 2019)
Python
91
star
25

Transferable-Adversarial-Training

Code release for Transferable Adversarial Training: A General Approach to Adapting Deep Classifiers (ICML2019)
Python
80
star
26

TransNorm

Code release for "Transferable Normalization: Towards Improving Transferability of Deep Neural Networks" (NeurIPS 2019)
Python
78
star
27

MTlearn

Code release for "Learning Multiple Tasks with Multilinear Relationship Networks" (NIPS 2017)
Python
70
star
28

SAN

Code release for "Partial Transfer Learning with Selective Adversarial Networks" (CVPR 2018)
Jupyter Notebook
69
star
29

Domain-Adaptation-Regression

Code release for Representation Subspace Distance for Domain Adaptation Regression (ICML 2021)
Python
69
star
30

HashGAN

HashGAN: Deep Learning to Hash with Pair Conditional Wasserstein GAN
Python
68
star
31

Deep-Embedded-Validation

Code release for Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation (ICML 2019)
Python
61
star
32

Latent-Spectral-Models

About Code Release for "Solving High-Dimensional PDEs with Latent Spectral Models" (ICML 2023), https://arxiv.org/abs/2301.12664
Python
59
star
33

CLIPood

About Code Release for "CLIPood: Generalizing CLIP to Out-of-Distributions" (ICML 2023), https://arxiv.org/abs/2302.00864
Python
58
star
34

iVideoGPT

Official repo for "iVideoGPT: Interactive VideoGPTs are Scalable World Models", https://arxiv.org/abs/2405.15223
Python
58
star
35

Transolver

About code release of "Transolver: A Fast Transformer Solver for PDEs on General Geometries", ICML 2024 Spotlight. https://arxiv.org/abs/2402.02366
Python
57
star
36

MADA

Code release for "Multi-Adversarial Domain Adaptation" (AAAI 2018)
C++
56
star
37

MotionRNN

About Code release for "MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions" (CVPR 2021) https://arxiv.org/abs/2103.02243
Python
50
star
38

ETN

Code released for CVPR 2019 paper "Learning to Transfer Examples for Partial Domain Adaptation"
Python
50
star
39

Debiased-Self-Training

Code release of paper Debiased Self-Training for Semi-Supervised Learning (NeurIPS 2022 Oral)
50
star
40

Versatile-Domain-Adaptation

Code Release for "Minimum Class Confusion for Versatile Domain Adaptation"(ECCV2020)
Python
50
star
41

ContextWM

Code release for "Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning" (NeurIPS 2023), https://arxiv.org/abs/2305.18499
Python
50
star
42

Separate_to_Adapt

Code release for Separate to Adapt: Open Set Domain Adaptation via Progressive Separation (CVPR 2019)
Python
49
star
43

AutoTimes

Official implementation for "AutoTimes: Autoregressive Time Series Forecasters via Large Language Models"
Python
45
star
44

CoTuning

Code release for NeurIPS 2020 paper "Co-Tuning for Transfer Learning"
Python
39
star
45

OpenDG-DAML

Code release for Open Domain Generalization with Domain-Augmented Meta-Learning (CVPR2021)
Python
32
star
46

Calibrated-Multiple-Uncertainties

Code Release for "Learning to Detect Open Classes for Universal Domain Adaptation"(ECCV2020)
Python
30
star
47

TimeSiam

Python
25
star
48

Batch-Spectral-Shrinkage

Code release for Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer Learning (NeurIPS 2019)
Python
24
star
49

StochNorm

Code release for NeurIPS 2020 paper "Stochastic Normalization"
Python
23
star
50

Transferable-Query-Selection

Code Release for "Transferable Query Selection for Active Domain Adaptation"(CVPR2021)
Python
23
star
51

Decoupled-Adaptation-for-Cross-Domain-Object-Detection

Code for ICLR2022 Decoupled Adaptation for Cross-Domain Object Detection (D-adapt) https://arxiv.org/abs/2110.02578
22
star
52

few-shot

A lightweight library that implements state-of-the-art few-shot learning algorithms.
Python
21
star
53

HarmonyDream

Code release for "HarmonyDream: Task Harmonization Inside World Models" (ICML 2024), https://arxiv.org/abs/2310.00344
Python
21
star
54

transferable-memory

Python
20
star
55

VideoDG

Python
20
star
56

TCL

Code release for Transferable Curriculum for Weakly-Supervised Domain Adaptation (AAAI2019)
Python
18
star
57

SPOT

Code release for "Supported Policy Optimization for Offline Reinforcement Learning" (NeurIPS 2022), https://arxiv.org/abs/2202.06239
Python
18
star
58

DPH

Code release for "Deep Priority Hashing" (ACMMM 2018)
C++
18
star
59

MMHH

Python
15
star
60

Metasets

Python
15
star
61

PAN

Python
15
star
62

DCN

Deep Calibration Network
Python
15
star
63

ModeRNN

Python
14
star
64

ForkMerge

Code release of paper "ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning" (NeurIPS 2023)
14
star
65

TAH

Code release for "Transfer Adversarial Hashing for Hamming Space Retrieval" (AAAI 2018)
C++
13
star
66

TransCal

Python
12
star
67

learn_torch.compile

torch.compile artifacts for common deep learning models, can be used as a learning resource for torch.compile
Python
12
star
68

HelmFluid

About code release of "HelmFluid: Learning Helmholtz Dynamics for Interpretable Fluid Prediction", ICML 2024. https://arxiv.org/pdf/2310.10565
Python
11
star
69

Multi-Embedding

About Code Release for "On the Embedding Collapse When Scaling Up Recommendation Models" (ICML 2024)
Python
11
star
70

Zoo-Tuning

Code release for Zoo-Tuning: Adaptive Transfer from A Zoo of Models (ICML2021)
Python
7
star
71

timer

See the official code and checkpoints for "Timer: Generative Pre-trained Transformers Are Large Time Series Models"
HTML
5
star
72

Regressive-Domain-Adaptation-for-Unsupervised-Keypoint-Detection

Code for CVPR 2021 Regressive Domain Adaptation for Unsupervised Keypoint Detection (RegDA) https://arxiv.org/abs/2103.06175
5
star
73

MitNet

About Code Release for "Estimating Heterogeneous Treatment Effects: Mutual Information Bounds and Learning Algorithms" (ICML 2023)
Python
4
star
74

MobileAttention

Official implementation of "Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers in PyTorch". To run the code, you can refer to https://github.com/thuml/Flowformer.
Python
1
star