• Stars
    star
    125
  • Rank 286,335 (Top 6 %)
  • Language
    Python
  • License
    MIT License
  • Created over 3 years ago
  • Updated almost 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[TPAMI 2022] "Bag of Tricks for Training Deeper Graph Neural Networks A Comprehensive Benchmark Study" by Tianlong Chen*, Kaixiong Zhou*, Keyu Duan, Wenqing Zheng, Peihao Wang, Xia Hu, Zhangyang Wang

Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study

License: MIT

Codes for [Preprint] Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study

Tianlong Chen*, Kaixiong Zhou*, Keyu Duan, Wenqing Zheng, Peihao Wang, Xia Hu, Zhangyang Wang

Introduction

This is the first fair and reproducible benchmark dedicated to assessing the "tricks" of training deep GNNs. We categorize existing approaches, investigate their hyperparameter sensitivity, and unify the basic configuration. Comprehensive evaluations are then conducted on tens of representative graph datasets including the recent large-scale Open Graph Benchmark (OGB), with diverse deep GNN backbones. Based on synergistic studies, we discover the transferable combo of superior training tricks, that lead us to attain the new state-of-the-art results for deep GCNs, across multiple representative graph datasets.

Requirements

Installation with Conda

conda create -n deep_gcn_benchmark
conda activate deep_gcn_benchmark
pip install -r requirement.txt

Our Installation Notes for PyTorch Geometric.

What env configs that we tried that have succeeded: Mac/Linux + cuda driver 11.2 + Torch with cuda 11.1 + torch_geometric/torch sparse/etc with cuda 11.1.

What env configs that we tried but didn't work: Linux+Cuda 11.1/11.0/10.2 + whatever version of Torch.

In the above case when it did work, we adopted the following installation commands, and it automatically downloaded built wheels, and the installation completes within seconds.

In the case when it did not work, the installation appears to be very slow (ten minutes level for torch sparse/torch scatter). Then the installation did not produce any error, while when import torch_geometric in python code, it reports errors of different types.

Installation codes that we adopted on Linux cuda 11.2 that did work:

pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cu111.html
pip install torch-sparse -f https://pytorch-geometric.com/whl/torch-1.9.0+cu111.html
pip install torch-geometric

Project Structure

.
β”œβ”€β”€ Dataloader.py
β”œβ”€β”€ main.py
β”œβ”€β”€ trainer.py
β”œβ”€β”€ models
β”‚   β”œβ”€β”€ *.py
β”œβ”€β”€ options
β”‚   β”œβ”€β”€ base_options.py
β”‚   └── configs
β”‚       β”œβ”€β”€ *.yml
β”œβ”€β”€ tricks
β”‚   β”œβ”€β”€ tricks
β”‚   β”‚   β”œβ”€β”€ dropouts.py
β”‚   β”‚   β”œβ”€β”€ norms.py
β”‚   β”‚   β”œβ”€β”€ others.py
β”‚   β”‚   └── skipConnections.py
β”‚   └── tricks_comb.py
└── utils.py

How to Use the Benchmark

Train Deep GCN models as your baselines

To train a deep GCN model <model> on dataset <dataset> as your baseline, run:

python main.py --compare_model=1 --cuda_num=0 --type_model=<model> --dataset=<dataset>
# <model>   in  [APPNP, DAGNN, GAT, GCN, GCNII, GPRGNN, JKNet, SGC]
# <dataset> in  [Cora, Citeseer, Pubmed, ogbn-arixv, CoauthorCS, CoauthorPhysics, AmazonComputers, AmazonPhoto, TEXAS, WISCONSIN, CORNELL, ACTOR]

we comprehensively explored the optimal hyperparameters for all models we implemented and train the models under the well-studied hyperparameter settings. For model-specific hyperparameter configs, please refer to options/configs/*.yml

Explore different trick combinations

To explore different trick combinations, we provide a tricks_comb model, which integrates different types of tricks as follows:

dropouts:        DropEdge, DropNode, FastGCN, LADIES
norms:           BatchNorm, PairNorm, NodeNorm, MeanNorm, GroupNorm, CombNorm
skipConnections: Residual, Initial, Jumping, Dense
others:          IdentityMapping

To train a tricks_comb model with specific tricks, run:

python main.py --compare_model=0 --cuda_num=0 --type_trick=<trick_1>+<trick_2>+...+<trick_n> --dataset=<dataset>

, where you can assign type_trick with any number of tricks. For instance, to train a trick_comb model with Initial, EdgeDrop, BatchNorm and IdentityMapping on Cora, run:

python main.py --compare_model=0 --cuda_num=0 --type_trick=Initial+EdgeDrop+BatchNorm+IdentityMapping --dataset=Cora

We provide two backbones --type_model=GCN and --type_tricks=SGC for trick combinations. Specifically, when --type_model=SGC and --type_trick=IdenityMapping co-occur, IdentityMapping has higher priority.

How to Contribute

You are welcome to make any type of contributions. Here we provide a brief guidance to add your own deep GCN models and tricks.

Add your own model

Several simple steps to add your own deep GCN model <DeepGCN>.

  1. Create a python file named <DeepGCN>.py
  2. Implement your own model as a torch.nn.Module, where the class name is recommended to be consistent with your filename <DeepGCN>
  3. Make sure the commonly-used hyperparameters is consistent with ours (listed as follows). To create any new hyperparameter, add it in options/base_options.py.
 --dim_hidden        # hidden dimension
 --num_layers        # number of GCN layers
 --dropout           # rate of dropout for GCN layers
 --lr:               # learning rate
 --weight_decay      # rate of l2 regularization
  1. Register your model in models/__init__.py by add the following codes:
from <DeepGCN> import <DeepGCN>
__all__.append('<DeepGCN>')
  1. You are recommend to use YAML to store your dataset-specific hyperparameter configuration. Create a YAML file <DeepGCN>.yml in options/configs and add the hyperparameters as the following style:
<dataset_1>
  <hyperparameter_1> : value_1
  <hyperparameter_2> : value_2

Now your own model <DeepGCN> should be added successfully into our benchmark framework. To test the performance of <DeepGCN> on <dataset>, run:

python main.py --compare_model=1 --type_model=<DeepGCN> --dataset=<dataset>

Add your own trick

As all implemented tricks are coupled in tricks_comb.py tightly, we do not recommend integrating your own trick to trick_comb to avoid unexpected errors. However, you can use the interfaces we provided in tricks/tricks/ to combine your own trick with ours.

Main Results and Leaderboard

  • Superior performance of our best combo with 32 layers deep GCNs
Model Ranking on Cora Test Accuracy
Ours 85.48
GCNII 85.29
APPNP 83.68
DAGNN 83.39
GPRGNN 83.13
JKNet 73.23
SGC 68.45
Model Ranking on Citeseer Test Accuracy
Ours 73.35
GCNII 73.24
DAGNN 72.59
APPNP 72.13
GPRGNN 71.01
SGC 61.92
JKNet 50.68
Model Ranking on PubMed Test Accuracy
Ours 80.76
DAGNN 80.58
APPNP 80.24
GCNII 79.91
GPRGNN 78.46
SGC 66.61
JKNet 63.77
Model Ranking on OGBN-ArXiv Test Accuracy
Ours 72.70
GCNII 72.60
DAGNN 71.46
GPRGNN 70.18
APPNP 66.94
JKNet 66.31
SGC 34.22
  • Transferability of our best combo with 32 layers deep GCNs
Models Average Ranking on (CS, Physics, Computers, Photo, Texas, Wisconsin, Cornell, Actor)
Ours 1.500
SGC 6.250
DAGNN 4.375
GCNII 3.875
JKNet 4.875
APPNP 4.000
GPRGNN 3.125
  • Takeaways of the best combo

Citation

if you find this repo is helpful, please cite

@misc{chen2021bag,
      title={Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study}, 
      author={Tianlong Chen and Kaixiong Zhou and Keyu Duan and Wenqing Zheng and Peihao Wang and Xia Hu and Zhangyang Wang},
      year={2021},
      eprint={2108.10521},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

More Repositories

1

TransGAN

[NeurIPSβ€˜2021] "TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up", Yifan Jiang, Shiyu Chang, Zhangyang Wang
Python
1,635
star
2

DeblurGANv2

[ICCV 2019] "DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better" by Orest Kupyn, Tetiana Martyniuk, Junru Wu, Zhangyang Wang
Python
1,002
star
3

EnlightenGAN

[IEEE TIP] "EnlightenGAN: Deep Light Enhancement without Paired Supervision" by Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, Zhangyang Wang
Python
790
star
4

LightGaussian

[NeurIPS 2024 Spotlight]"LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS", Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, Zhangyang Wang
Python
563
star
5

FasterSeg

[ICLR 2020] "FasterSeg: Searching for Faster Real-time Semantic Segmentation" by Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang, Yuan Li, Zhangyang Wang
Python
526
star
6

AutoGAN

[ICCV 2019] "AutoGAN: Neural Architecture Search for Generative Adversarial Networks" by Xinyu Gong, Shiyu Chang, Yifan Jiang and Zhangyang Wang
Python
463
star
7

ShapeMatchingGAN

[ICCV 2019, Oral] Controllable Artistic Text Style Transfer via Shape-Matching GAN
Jupyter Notebook
425
star
8

FSGS

[ECCV 2024]"FSGS: Real-Time Few-Shot View Synthesis using Gaussian Splatting", Zehao Zhu*, Zhiwen Fan*, Yifan Jiang, Zhangyang Wang
Python
368
star
9

GLNet

[CVPR 2019, Oral] "Collaborative Global-Local Networks for Memory-Efficient Segmentation of Ultra-High Resolution Images" by Wuyang Chen*, Ziyu Jiang*, Zhangyang Wang, Kexin Cui, and Xiaoning Qian
Python
344
star
10

GNT

[ICLR 2023] "Is Attention All NeRF Needs?" by Mukund Varma T*, Peihao Wang* , Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang
Python
338
star
11

SinNeRF

[ECCV 2022] "SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image", Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang
Python
327
star
12

NeuralLift-360

[CVPR 2023, Highlight] "NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360Β° Views", Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, Zhangyang Wang
Python
312
star
13

ABD-Net

[ICCV 2019] "ABD-Net: Attentive but Diverse Person Re-Identification" https://arxiv.org/abs/1908.01114
Python
306
star
14

SLaK

[ICLR 2023] "More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity"; [ICML 2023] "Are Large Kernels Better Teachers than Transformers for ConvNets?"
HTML
264
star
15

Open-L2O

Open-L2O: A Comprehensive and Reproducible Benchmark for Learning to Optimize Algorithms
C++
260
star
16

Diffusion4D

"Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models", Hanwen Liang*, Yuyang Yin*, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N. Plataniotis, Yao Zhao, Yunchao Wei
Python
208
star
17

AutoSpeech

[InterSpeech 2020] "AutoSpeech: Neural Architecture Search for Speaker Recognition" by Shaojin Ding*, Tianlong Chen*, Xinyu Gong, Weiwei Zha, Zhangyang Wang
Python
207
star
18

4DGen

"4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency", Yuyang Yin*, Dejia Xu*, Zhangyang Wang, Yao Zhao, Yunchao Wei
Python
201
star
19

TENAS

[ICLR 2021] "Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective" by Wuyang Chen, Xinyu Gong, Zhangyang Wang
Python
164
star
20

Deep-K-Means-pytorch

[ICML 2018] "Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions"
Python
150
star
21

BERT-Tickets

[NeurIPS 2020] "The Lottery Ticket Hypothesis for Pre-trained BERT Networks", Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, Michael Carbin
Python
137
star
22

Q-GaLore

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.
Python
131
star
23

Orthogonality-in-CNNs

[NeurIPS '18] "Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?" Official Implementation.
Python
127
star
24

NeRF-SOS

[ICLR2023] "NeRF-SOS: Any-View Self-supervised Object Segmentation from Complex Real-World Scenes", Zhiwen Fan, Peihao Wang, Xinyu Gong, Yifan Jiang, Dejia Xu, Zhangyang Wang
Python
127
star
25

AugMax

[NeurIPS'21] "AugMax: Adversarial Composition of Random Augmentations for Robust Training" by Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Animashree Anandkumar, and Zhangyang Wang.
Python
125
star
26

Aug-NeRF

[CVPR 2022] "Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations" by Tianlong Chen*, Peihao Wang*, Zhiwen Fan, Zhangyang Wang
Python
124
star
27

Adversarial-Contrastive-Learning

[NeurIPS 2020] β€œ Robust Pre-Training by Adversarial Contrastive Learning”, Ziyu Jiang, Tianlong Chen, Ting Chen, Zhangyang Wang
Python
112
star
28

INS

[ECCV2022]"Unified Implicit Neural Stylization" which proposes a unified stylization framework for SIREN, SDF and NeRF
Python
109
star
29

GAN-Slimming

[ECCV 2020] "All-in-One GAN Compression by Unified Optimization" by Haotao Wang, Shupeng Gui, Haichuan Yang, Ji Liu, and Zhangyang Wang
Python
107
star
30

AGD

[ICML2020] "AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks" by Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, Zhangyang Wang
Python
102
star
31

SSHarmonization

[ICCV'2021] "SSH: A Self-Supervised Framework for Image Harmonization", Yifan Jiang, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi, Sarah Kong, Zhangyang Wang
Jupyter Notebook
98
star
32

M3ViT

[NeurIPS 2022] β€œMΒ³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design”, Hanxue Liang*, Zhiwen Fan*, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang
Python
94
star
33

UAV-NDFT

[ICCV 2019] "Delving into Robust Object Detection from Unmanned Aerial Vehicles: A Deep Nuisance Disentanglement Approach"
Python
91
star
34

SViTE

[NeurIPS'21] "Chasing Sparsity in Vision Transformers: An End-to-End Exploration" by Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, Zhangyang Wang
Python
88
star
35

Adv-SS-Pretraining

[CVPR 2020] Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
Python
84
star
36

Ultra-Data-Efficient-GAN-Training

[NeurIPS'21] "Ultra-Data-Efficient GAN Training: Drawing A Lottery Ticket First, Then Training It Toughly", Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, Zhangyang Wang
Python
84
star
37

MM3DGS-SLAM

[IROS 2024] MM3DGS SLAM: Multi-modal 3D Gaussian Splatting for SLAM Using Vision, Depth, and Inertial Measurements
Python
83
star
38

LiGO

[ICLR 2023] "Learning to Grow Pretrained Models for Efficient Transformer Training" by Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Cox, Zhangyang Wang, Yoon Kim
Python
80
star
39

Nasty-Teacher

[ICLR 2021 Spotlight Oral] "Undistillable: Making A Nasty Teacher That CANNOT teach students", Haoyu Ma, Tianlong Chen, Ting-Kuei Hu, Chenyu You, Xiaohui Xie, Zhangyang Wang
Python
79
star
40

DeepPS

[ECCV 2020] "Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches"
Python
77
star
41

ViT-Anti-Oversmoothing

[ICLR 2022] "Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice" by Peihao Wang, Wenqing Zheng, Tianlong Chen, Zhangyang Wang
Python
76
star
42

AsViT

[ICLR 2022] "As-ViT: Auto-scaling Vision Transformers without Training" by Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, Denny Zhou
Python
76
star
43

All-In-One-Underwater-Image-Enhancement-using-Domain-Adversarial-Learning

[CVPRW 2019] All-In-One Underwater Image Enhancement using Domain-Adversarial Learning
Python
69
star
44

Random_Pruning

[ICLR 2022] The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training by Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, Mykola Pechenizkiy
Python
69
star
45

ALISTA

[ICLR 2019] "ALISTA: Analytic Weights Are As Good As Learned Weights in LISTA", by Jialin Liu*, Xiaohan Chen*, Zhangyang Wang and Wotao Yin.
Python
67
star
46

CV_LTH_Pre-training

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang
Python
67
star
47

Comp4D

"Comp4D: Compositional 4D Scene Generation", Dejia Xu*, Hanwen Liang*, Neel P. Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N. Plataniotis, and Zhangyang Wang
Python
67
star
48

CADTransformer

[CVPR 2022]"CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings", Zhiwen Fan, Tianlong Chen, Peihao Wang, Zhangyang Wang
Python
65
star
49

SDCLR

[ICML 2021] β€œ Self-Damaging Contrastive Learning”, Ziyu Jiang, Tianlong Chen, Bobak Mortazavi, Zhangyang Wang
Python
63
star
50

LLaGA

[ICML2024] "LLaGA: Large Language and Graph Assistant", Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, Zhangyang Wang
Python
62
star
51

Unified-LTH-GNN

[ICML 2021] "A Unified Lottery Tickets Hypothesis for Graph Neural Networks", Tianlong Chen*, Yongduo Sui*, Xuxi Chen, Aston Zhang, Zhangyang Wang
Python
61
star
52

Simple3D-Former

[Preprint 2022] β€œCan We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?” by Yi Wang, Zhiwen Fan, Tianlong Chen, Hehe Fan, Zhangyang Wang
Python
61
star
53

Large_Scale_GCN_Benchmarking

This is an authors' implementation of the NIPS 2022 dataset and Benchmark Track Paper "A Comprehensive Study on Large Scale Graph Training: Benchmarking and Rethinking" in PyTorch.
Python
59
star
54

Self-PU

[ICML2020] "Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training" by Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, Zhangyang Wang
Python
59
star
55

BNN_NoBN

[CVPRW 21] "BNN - BN = ? Training Binary Neural Networks without Batch Normalization", Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, Zhangyang Wang
Python
54
star
56

Sandwich-Batch-Normalization

[WACV 2022] "Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneity" by Xinyu Gong, Wuyang Chen, Tianlong Chen and Zhangyang Wang
Python
49
star
57

INSP

[NeurIPS 2022] "Signal Processing for Implicit Neural Representations" by Dejia Xu*, Peihao Wang*, Yifan Jiang, Zhiwen Fan, Zhangyang Wang
Python
49
star
58

ATMC

[NeurIPS'2019] Shupeng Gui, Haotao Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, Ji Liu, β€œModel Compression with Adversarial Robustness: A Unified Optimization Framework”
Python
48
star
59

LISTA-CPSS

[NeurIPS'18, Spotlight oral] "Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds", by Xiaohan Chen*, Jialin Liu*, Zhangyang Wang and Wotao Yin.
Python
48
star
60

dehaze

[Preprint] "Improved Techniques for Learning to Dehaze and Beyond: A Collective Study"
C++
47
star
61

GNT-MOVE

[ICCV2023] "Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts" by Wenyan Cong, Hanxue Liang, Peihao Wang, Zhiwen Fan, Tianlong Chen, Mukund Varma, Yi Wang, Zhangyang Wang
Python
46
star
62

UVC

[ICLR 2022] "Unified Vision Transformer Compression" by Shixing Yu*, Tianlong Chen*, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, Zhangyang Wang
Python
45
star
63

FAT

[Preprint] "In Defense of the Triplet Loss Again: Learning Robust Person Re-Identification with Fast Approximated Triplet Loss and Label Distillation" by Ye Yuan, Wuyang Chen, Yang Yang, Zhangyang Wang
Python
44
star
64

3D-Mode-Collapse

"Taming Mode Collapse in Score Distillation for Text-to-3D Generation" by Peihao Wang, Dejia Xu, Zhiwen Fan, Dilin Wang, Sreyas Mohan, Forrest Iandola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang, Vikas Chandra
Python
44
star
65

Once-for-All-Adversarial-Training

[NeurIPS 2020] "Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free" by Haotao Wang*, Tianlong Chen*, Shupeng Gui, Ting-Kuei Hu, Ji Liu, and Zhangyang Wang
Python
43
star
66

WeLore

From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, Zhangyang Wang
Python
43
star
67

Random-MoE-as-Dropout

[ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal, Shiwei Liu, Zhangyang Wang
Python
42
star
68

Alleviate-Robust-Overfitting

[ICLR 2021] "Robust Overfitting may be mitigated by properly learned smoothening" by Tianlong Chen*, Zhenyu Zhang*, Sijia Liu, Shiyu Chang, Zhangyang Wang
Python
42
star
69

Focus-Longer-to-See-Better

[CVPRW 2020] Focus Longer to See Better:Recursively Refined Attention for Fine-Grained Image Classification
Python
40
star
70

PA-HMDB51

[TPAMI 2020] "Privacy-Preserving Deep Action Recognition: An Adversarial Learning Framework and A New Dataset" by Zhenyu Wu, Haotao Wang, Zhaowen Wang, Hailin Jin, and Zhangyang Wang
Jupyter Notebook
40
star
71

UMEC

[ICLR 2021] "UMEC: Unified Model and Embedding Compression for Efficient Recommendation Systems" by Jiayi Shen, Haotao Wang*, Shupeng Gui*, Jianchao Tan, Zhangyang Wang, and Ji Liu
Python
39
star
72

Privacy-AdversarialLearning

[ECCV 2018] Towards Privacy-Preserving Visual Recognition via Adversarial Training: A Pilot Study
Jupyter Notebook
38
star
73

Sparsity-Win-Robust-Generalization

[ICLR 2022] "Sparsity Winning Twice: Better Robust Generalization from More Efficient Training" by Tianlong Chen*, Zhenyu Zhang*, Pengjun Wang*, Santosh Balachandra*, Haoyu Ma*, Zehao Wang, Zhangyang Wang
Python
37
star
74

ChainCoder

πŸ“œ [ICML 2023] "Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation", Wenqing Zheng, S P Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, Zhangyang Wang
Python
36
star
75

USAID

[Preprint] "Segmentation-Aware Image Denoising without Knowing True Segmentation"
Python
35
star
76

SteinDreamer

β€œSteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity” by Peihao Wang, Zhiwen Fan, Dejia Xu, Dilin Wang, Sreyas Mohan, Forrest Iandola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang, Vikas Chandra
34
star
77

Neural-Implicit-Dict

[ICML 2022] "Neural Implicit Dictionary via Mixture-of-Expert Training" by Peihao Wang, Zhiwen Fan, Tianlong Chen, Zhangyang Wang
Python
32
star
78

L2-GCN

[CVPR 2020] L2-GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networks
Python
30
star
79

Audio-Lottery

[ICLR 2022] "Audio Lottery: Speech Recognition Made Ultra-Lightweight, Noise-Robust, and Transferable", by Shaojin Ding, Tianlong Chen, Zhangyang Wang
Python
30
star
80

Structure-LTH

[ICML 2022] "Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets" by Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, Zhangyang Wang.
Cuda
30
star
81

SFW-Once-for-All-Pruning

[ICLR 2022] "Learning Pruning-Friendly Networks via Frank-Wolfe: One-Shot, Any-Sparsity, and No Retraining" by Lu Miao*, Xiaolong Luo*, Tianlong Chen, Wuyang Chen, Dong Liu, Zhangyang Wang
Python
29
star
82

DP-OPT

[ICLR'24 Spotlight] DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer
Python
28
star
83

Graph-Mixture-of-Experts

[NeurIPS'23] Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling. Haotao Wang, Ziyu Jiang, Yuning You, Yan Han, Gaowen Liu, Jayanth Srinivasa, Ramana Rao Kompella, Zhangyang Wang
Python
28
star
84

TurbNet

[ECCV 2022] "Single Frame Atmospheric Turbulence Mitigation: A Benchmark Study and A New Physics-Inspired Transformer Model" by Zhiyuan Mao, Ajay Jaiswal, Zhangyang Wang, Stanley Chan.
Python
28
star
85

MAK

[NeurIPS 2021] β€œImproving Contrastive Learning on Imbalanced Data via Open-World Sampling”, Ziyu Jiang, Tianlong Chen, Ting Chen, Zhangyang Wang
Python
27
star
86

LongTailCXR

[DALI 2022] "Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study" by Gregory Holste, Song Wang, Ziyu Jiang, Thomas C. Shen, Ronald M. Summers, Yifan Peng, and Zhangyang Wang
Python
27
star
87

WeakNAS

[NeurIPS 2021] β€œStronger NAS with Weaker Predictorsβ€œ, Junru Wu, Xiyang Dai, Dongdong Chen, Yinpeng Chen, Mengchen Liu, Ye Yu, Zhangyang Wang, Zicheng Liu, Mei Chen and Lu Yuan
Jupyter Notebook
27
star
88

HotProtein

[ICLR 2023] "HotProtein: A Novel Framework for Protein Thermostability Prediction and Editing" by Tianlong Chen*, Chengyue Gong*, Daniel Jesus Diaz, Xuxi Chen, Jordan Tyler Wells, Qiang Liu, Zhangyang Wang, Andrew Ellington, Alex Dimakis, Adam Klivans
Python
27
star
89

ViHGNN

[ICCV2023] "Vision HGNN: An Image is More than a Graph of Nodes" by Yan Han, Peihao Wang, Souvik Kundu, Ying Ding, and Zhangyang Wang
Python
27
star
90

GraNet

[Neurips 2021] Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
Python
26
star
91

GAN-LTH

[ICLR 2021] "GANs Can Play Lottery Too" by Xuxi Chen, Zhenyu Zhang, Yongduo Sui, Tianlong Chen
Python
26
star
92

FreeTickets

[ICLR 2022] "Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity" by Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, Decebal Constantin Mocanu
Python
26
star
93

L2O-Training-Techniques

[NeurIPS 2020 Spotlight Oral] "Training Stronger Baselines for Learning to Optimize", Tianlong Chen*, Weiyi Zhang*, Jingyang Zhou, Shiyu Chang, Sijia Liu, Lisa Amini, Zhangyang Wang
Python
26
star
94

PrAC-LTH

[ICML 2021] "Efficient Lottery Ticket Finding: Less Data is More" by Zhenyu Zhang*, Xuxi Chen*, Tianlong Chen*, Zhangyang Wang
Python
25
star
95

Diverse-ViT

[CVPR 2022] "The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy" by Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, Zhangyang Wang
Python
25
star
96

TEGNAS

"Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics" by Wuyang Chen, Xinyu Gong, Yunchao Wei, Humphrey Shi, Zhicheng Yan, Yi Yang, and Zhangyang Wang
Python
25
star
97

mm-hand

[ACM'MM 2020] "MM-Hand: 3D-Aware Multi-Modal Guided Hand Generative Network for 3D Hand Pose Synthesis" Zhenyu Wu, Duc Hoang, Shih-Yao Lin, Yusheng Xie, Liangjian Chen, Yen-Yu Lin, Zhangyang Wang, Wei Fan
Python
24
star
98

triple-wins

[ICLR 2020] ”Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inferenceβ€œ
Python
24
star
99

SMC-Bench

[ICLR 2023] "Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!" Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen, Tianjin Huang, AJAY KUMAR JAISWAL, Zhangyang Wang
Python
24
star
100

Backdoor-LTH

[CVPR 2022] "Quarantine: Sparsity Can Uncover the Trojan Attack Trigger for Free" by Tianlong Chen*, Zhenyu Zhang*, Yihua Zhang*, Shiyu Chang, Sijia Liu, and Zhangyang Wang
Python
24
star