Awesome-Normalization-Techniques
Papers for normalization techniques, released codes collections.
Any addition or bug feel free to open an issue or pull requests.
2020 - 2019 - 2018 - 2017 - 2016
2020
- A New Look at Ghost Normalization
- Neofytos Dimitriou, Ognjen Arandjelovic
- [Paper]
- TaskNorm: Rethinking Batch Normalization for Meta-Learning (ICML 2020)
- John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, Richard E Turner
- [Paper]
- [Python Reference]
- Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise (AAAI 2020)
- Senwei Liang, Zhongzhan Huang, Mingfu Liang, Haizhao Yang
- [Paper]
- [Python Reference]
- Towards Stabilizing Batch Statistics in BackWard Propagation of Batch Normalization (ICLR 2020)
- Junjie Yan, Ruosi Wan, Xiangyu Zhang, Wei Zhang, Yichen Wei, Jian Sun
- [Paper]
- Rethinking Spatially-Adaptive Normalization
- Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, Nenghai Yu
- [Paper]
- Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks (CVPR 2020)
- Saurabh Singh, Shankar Krishnan
- [Paper]
- [Python Reference]
- Extended Batch Normalization
- Chunjie Luo, Jianfeng Zhan, Lei Wang, and Wanling Gao
- [Paper]
- [C++ Reference]
- Knowledge Distillation via Adaptive Instance Normalization
- Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos
- [Paper]
- Four Things Everyone Should Know to Improve Batch Normalization
- Cecilia Summers, Michael J. Dinneen
- [Paper]
- Region Normalization for Image Inpainting (AAAI 2020)
- Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, Sen Liu
- [Paper]
- [Python Reference]
- Local Context Normalization: Revisiting Local Normalization (CVPR 2020)
- Anthony Ortiz , Caleb Robinson, Mahmudulla Hassan, Dan Morris2, Olac Fuentes1, Christopher Kiekintveld, and Nebojsa Jojic
- [Paper]
- Attentive Normalization for Conditional Image Generation (CVPR 2020)
- Yi Wang, Ying-Cong Chen, Xiangyu Zhang, Jian Sun, Jiaya Jia
- [Paper]
- [Python Reference]
- Cross-Iteration Batch Normalization
- Zhuliang Yao, Yue Cao, Shuxin Zheng, Gao Huang, Stephen Lin
- [Paper]
- [Python Reference]
- SEAN: Image Synthesis with Semantic Region-Adaptive Normalization (CVPR 2020)
- Peihao Zhu, Rameen Abdal, Yipeng Qin, Peter Wonka
- [Paper]
- [Python Reference]
- Evolving Normalization-Activation Layers
- Hanxiao Liu, Andrew Brock, Karen Simonyan, Quoc V. Le
- [Paper]
2019
- Differentiable Dynamic Normalization for Learning Deep Representation (PMLR 2019/ICML 2019)
- Ping Luo, Peng Zhanglin, Shao Wenqi, Zhang Ruimao, Ren Jiamin, Wu Lingyun
- [Paper]
- Mean Spectral Normalization of Deep Neural Networks for Embedded Automation
- Anand Krishnamoorthy Subramanian, Nak Young Chong
- [Paper]
- [Python Reference]
- An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
- Vincent Michalski, Vikram Voleti, Samira Ebrahimi Kahou, Anthony Ortiz, Pascal Vincent, Chris Pal, Doina Precup
- [Paper]
- Generalized Batch Normalization: Towards Accelerating Deep Neural Networks (AAAI 2019)
- Xiaoyong Yuan, Zheng Feng, Matthew Norton, Xiaolin Li
- [Paper]
- Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift
- Michał Zając, Konrad Zolna, Stanisław Jastrzębski
- [Paper]
- Channel Normalization in Convolutional Neural Network avoids Vanishing Gradients (ICML 2019)
- Zhenwei Dai, Reinhard Heckel
- [Paper]
- [Python Reference]
- Unpaired Image Translation via Adaptive Convolution-based Normalization
- Wonwoong Cho, Kangyeol Kim, Eungyeup Kim, Hyunwoo J. Kim, Jaegul Choo
- [Paper]
- EvalNorm: Estimating Batch Normalization Statistics for Evaluation (ICCV 2019)
- Saurabh Singh, Abhinav Shrivastava
- [Paper]
- Online Normalization for Training Neural Networks (NIPS 2019)
- Vitaliy Chiley, Ilya Sharapov, Atli Kosson Urs Koster
- [Paper]
- [Python Reference]
- Transferable Normalization: Towards Improving Transferability of Deep Neural Networks (NIPS 2019)
- Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael I. Jordan
- [Paper]
- [Python Reference]
- Iterative Normalization: Beyond Standardization Towards Efficient Whitening (CVPR 2019)
- Lei Huang, Yi Zhou, Fan Zhu, Li Liu, Ling Shao
- [Paper]
- [Python Reference]
- Domain-Specific Batch Normalization for Unsupervised Domain Adaptation (CVPR 2019)
- Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, Bohyung Han
- [Paper]
- [Python Reference]
- Attentive Normalization
- Xilai Li, Wei Sun and Tianfu Wu
- [Paper]
- [Python Reference]
- Rethinking Normalization and Elimination Singularity in Neural Networks
- Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, Alan Yuille
- [Paper]
- Dynamic Instance Normalization for Arbitrary
- Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, Shilei Wen
- [Paper]
- Semantic Image Synthesis with Spatially-Adaptive Normalization (CVPR 2019)
- Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu
- [Paper]
- [Python Reference]
- Differentiable Learning-to-Normalize via Switchable Normalization (ICLR 2019)
- Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, Jingyu Li
- [Paper]
- [Python Reference]
- SSN: Learning Sparse Switchable Normalization via Sparsestmax (CVPR 2019)
- Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, Ping Luo
- [Paper]
- A Mean Field Theory of Batch Normalization
- Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz
- [Paper]
- Restructuring Batch Normalization to Accelerate CNN Training
- Wonkyung Jung, Daejin Jung, Byeongho Kim, Sunjung Lee, Wonjong Rhee, Jung Ho Ahn
- [Paper]
- A Novel Convolutional Neural Network for Image Steganalysis with Shared Normalization (TMM 2019)
- Songtao Wu, Sheng-hua Zhong, and Yan Liu
- [Paper]
- Training Faster by Separating Modes of Variation in Batch-normalized Models (TPAMI 2019)
- Mahdi M. Kalayeh, Mubarak Shah
- [Paper]
- Uncertainty Estimation via Stochastic Batch Normalization (ISNN 2019)
- Andrei Atanov, Arsenii Ashukha, Dmitry Molchanov, Kirill Neklyudov, Dmitry Vetrov
- [Paper]
- [Python Reference]
2018
- Decorrelated Batch Normalization (CVPR 2018)
- Lei Huang, Dawei Yang, Bo Lang, Jia Deng
- [Paper]
- [Lua Reference]
- MegDet: A Large Mini-Batch Object Detector (CVPR 2018)
- Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, Jian Sun
- [Paper]
- Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks (NIPS 2018)
- Hyeonseob Nam, Hyo-Eun Kim
- [Paper]
- Kalman Normalization: Normalizing Internal Representations Across Network Layers (NIPS 2018)
- Guangrun Wang, Jiefeng Peng, Ping Luo, Xinjiang Wang, Liang Lin
- [Paper]
- Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
- Ping Luo, Zhanglin Peng, Jiamin Ren, Ruimao Zhang
- [Paper]
- L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks (TNNLS)
- Shuang Wu , Guoqi Li, Lei Deng, Liu Liu, Dong Wu, Yuan Xie, Luping Shi
- [Paper]
- In-Place Activated BatchNorm for Memory-Optimized Training of DNNs (CVPR 2018)
- Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder
- [Paper]
- [Python Reference]
- Group Normalization (ECCV 2018)
- Yuxin Wu, Kaiming He
- [Paper]
- [Python Reference]
- Spectral Normalization for Generative Adversarial
- Takeru Miyato, Toshiki Kataoka, Masanori Koyama, Yuichi Yoshida
- [Paper]
- [Python Reference]
2017
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models (NIPS 2017)
- Sergey Ioffe
- [Paper]
- [Python Reference]
- Modulating Early Visual Processing by Language (NIPS 2017)
- Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, Aaron Courville
- [Paper]
- [Python Reference]
- Instance Normalization: the Missing Ingredient for Fast Stylization
- Dmitry Ulyanov, Andrea Vedaldi
- [Paper]
- [PyTorch Reference]
- Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks
- Luo Chunjie, Zhan Jianfeng, Wang Lei, Yang Qiang
- [Paper]
- Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization (ICCV 2017)
- Xun Huang, Serge Belongie
- [Paper]
- [Python Reference]
- Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
- Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H. Sinz, Richard S. Zemel
- [Paper]
- Recurrent Batch Normalization
- Tim Cooijmans, Nicolas Ballas, César Laurent, Çag ̆lar Gülçehre, Aaron Courville
- [Paper]
2016
-
Layer Normalization
- Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton
- [Paper]
- [PyTorch Reference]
-
Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
- Devansh Arpit, Yingbo Zhou, Bhargava U. Kota, Venu Govindaraju, SUNY Buffalo
- [Paper]
-
Weight Normalization: a Simple Reparameterization to Accelerate Training of Deep Neural Networks (NIPS 2016)
- Tim Salimans, Diederik P. Kingma
- [Paper]
2015
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Sergey Ioffe, Christian Szegedy
- [Paper]
- [PyTorch Reference]