A Survey on Rain Removal from Video and Single Image
Hong Wang, Yichen Wu, Minghan Li, Qian Zhao, and Deyu Meng
Citation
@article{wang2022survey,
title={Survey on rain removal from videos or a single image},
author={Wang, Hong and Wu, Yichen and Li, Minghan and Zhao, Qian and Meng, Deyu},
journal={Science China Information Sciences},
volume={65},
number={1},
pages={1--23},
year={2022},
publisher={Springer}
}
@article{WangA,
title={A Survey on Rain Removal from Video and Single Image},
author={Wang, Hong and Wu, Yichen and Li, Minghan and Zhao, Qian and Meng, Deyu},
journal={arXiv preprint arXiv:1909.08326},
year={2019}
}
Physical Properties of Raindrops
- Gemometric Property
- Brightness Property
- Chromatic Property
- Spatial and Temporal Propety
Video Deraining Methods
- Time Domain
- Detection and removal of rain from videos (CVPR2004), Garg et al [Project][PDF]
- When does camera see rain? (ICCV2005), Garg et al [Project][PDF]
- Rain removal using kalman filter in video (ICSMA2008), Park et al [PDF]
- Using the shape characteristics of rain to identify and remove rain from video (S+SSPR2008), Brewer et al [PDF]
- The application of histogram on rain detection in video (JCIS2008), Zhao et al [PDF]
- Rain or snow detection in image sequences through use of a histogram of orientation of streaks (IJCV2011), Bossu et al [PDF]
- A probabilistic approach for detection and removal of rain from videos (IETE JR2011), Tripathi et al [PDF]
- Video post processing: low latency spatiotemporal approach for detection and removal of rain (IET IP2012), Tripathi et al [PDF]
- Removal of rain from videos: a review (SIVP2014), Tripathi et al [PDF]
- Stereo video deraining and desnowing based on spatiotemporal frame warping (ICIP2014), Kim et al [PDF]
- Frequency Domain
- Low Rank and Sparsity
- A generalized low-rank appearance model for spatio-temporally correlated rain streaks (ICCV2013), Chen et al [PDF]
- A rain pixel recovery algorithm for videos with highly dynamic scenes (TIP2013), Chen et al [PDF]
- Video deraining and desnowing using temporal correlation and low-rank matrix completion (TIP2015), Kim et al [PDF] [Code]
- Adherent raindrop modeling, detection and removal in video (TPAMI2016), You et al. [Project] [PDF]
- Video desnowing and deraining based on matrix decomposition (CVPR2017), Ren et al [PDF] [Code]
- A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors (CVPR2017), Jiang et al [PDF]
- Should We encode rain streaks in video as deterministic or stochastic? (ICCV2017), Wei et al [PDF] [Code]
- A directional global sparse model for single image rain removal (AMM2018), Deng et al [PDF] [Code]
- Video rain streak removal by multiscale convolutional sparse coding (CVPR2018), Li et al [Project] [PDF] [Code]
- Fastderain: A novel video rain streak removal method using directional gradient priors (TIP2019), Jiang et al [PDF] [Code]
- Deep Learning
- Robust video content alignment and compensation for rain removal in a cnn framework (CVPR2018), Chen et al [PDF] [Code]
- Erase or fill? deep joint recurrent rain removal and reconstruction in videos (CVPR2018), Liu et al. [Project] [PDF] [Code]
- D3R-Net: dynamic routing residue recurrent network for video rain removal (TIP2018), Liu et al. [PDF]
- Frame consistent recurrent video deraining with dual-level flow (CVPR2019), Yang et al. [Code]
- Self-Learning Video Rain Streak Removal: When Cyclic Consistency Meets Temporal Correspondence(CVPR2020), Yang et al.[PDF] [Supplementary Materials] [Code]
- Online Rain/Snow Removal From Surveillance Videos(TIP2021), Li et al. [PDF]
- Triple-Level Model Inferred Collaborative Network Architecture for Video Deraining(TIP2021), Mu et al.[PDF] [Code]
- Let’s See Clearly: Contaminant Artifact Removal for Moving Cameras(ICCV2021), Li et al. [PDF] [Results Display]
- Self-Aligned Video Deraining With Transmission-Depth Consistency(CVPR2021), Yan et al. [PDF] [Supplementary Materials] [Code]
- Semi-Supervised Video Deraining With Dynamical Rain Generator(CVPR2021), Yue et al. [PDF][Supplementary Materials] [Code]
- Robust online rain removal for surveillance videos with dynamic rains(KBS2021), Yi et al. [PDF]
- Feature-Aligned Video Raindrop Removal With Temporal Constraints(TIP2022), Yan et al. [PDF]
- Reivew paper
Single Image Deraining Methods
- Filter based methods
- Guided image filtering (ECCV2010), He et al. [Project] [PDF] [Code]
- Removing rain and snow in a single image using guided filter (CSAE2012), Xu et al. [PDF]
- An improved guidance image based method to remove rain and snow in a single image (CIS2012), Xu et al. [PDF]
- Single-image deraining using an adaptive nonlocal means filter (ICIP2013), Kim et al. [PDF]
- Single-image-based rain and snow removal using multi-guided filter (NIPS2013), Zheng et al. [PDF]
- Single image rain and snow removal via guided L0 smoothing filter (Multimedia Tools and Application2016), Ding et al. [PDF]
- Prior based methods
- Automatic single-image-based rain streaks removal via image decomposition (TIP2012), Kang et al [PDF] [Code]
- Self-learning-based rain streak removal for image/video (ISCS2012), Kang et al. [PDF]
- Single-frame-based rain removal via image decomposition (ICA2013), Fu et al. [PDF]
- Exploiting image structural similarity for single image rain removal (ICIP2014), Sun et al. [PDF]
- Visual depth guided color image rain streaks removal using sparse coding (TCSVT2014), Chen et al [PDF]
- Removing rain from a single image via discriminative sparse coding (ICCV2015), Luo et al [PDF] [Code] pwd: d229
- Rain streak removal using layer priors (CVPR2016), Li et al [PDF] [Code]
- Single image rain streak decomposition using layer priors (TIP2017), Li et al [PDF]
- Error-optimized dparse representation for single image rain removal (IEEE TIE2017), Chen et al [PDF]
- A hierarchical approach for rain or snow removing in a single color image (TIP2017), Wang et al. [PDF]
- Joint bi-layer optimization for single-image rain streak removal (ICCV2017), Zhu et al. [PDF]
- Convolutional sparse and low-rank codingbased rain streak removal (WCACV2017), Zhang et al [PDF]
- Joint convolutional analysis and synthesis sparse representation for single image layer separation (CVPR2017), Gu et al [PDF] [Code]
- Single image deraining via decorrelating the rain streaks and background scene in gradient domain (PR2018)， Du et al [PDF]
- Deep Learning
- Restoring an image taken through a window covered with dirt or rain (ICCV2013), Eigen et al. [Project] [PDF] [Code]
- Attentive generative adversarial network for raindrop removal from a single image (CVPR2018), Qian et al [Project] [PDF]
- Clearing the skies: A deep network architecture for single-image rain streaks removal (TIP2017), Fu et al. [Project] [PDF] [Code]
- Removing rain from single images via a deep detail network (CVPR2017), Fu et al. [Project] [PDF] [Code]
- Image de-raining using a conditional generative adversarial network (Arxiv2017), Zhang et al [PDF] [Code]
- Deep joint rain detection and removal from a single image (CVPR2017), Yang et al.[Project] [PDF] [Code]
- Residual guide feature fusion network for single image deraining (ACMMM2018), Fan et al. [Project] [PDF]
- Fast single image rain removal via a deep decomposition-composition network (Arxiv2018), Li et al [Project]) [PDF] [Code]
- Density-aware single image de-raining using a multi-stream dense network (CVPR2018), Zhang et al [PDF] [Code]
- Recurrent squeeze-and-excitation context aggregation net for single image deraining (ECCV2018), Li et al. [PDF] [Code]
- Rain streak removal for single image via kernel guided cnn (Arxiv2018), Wang et al [PDF]
- Physics-based generative adversarial models for image restoration and beyond (Arxiv2018), Pan et al [PDF]
- Learning dual convolutional neural networks for low-level vision (CVPR2018), Pan et al [Project] [PDF] [Code]
- Non-locally enhanced encoder-decoder network for single image de-raining (ACMMM2018), Li et al [PDF] [Code]
- Single image rain removal via a deep decomposition-composition network (CVIU2019), Li et al.
- Unsupervised single image deraining with self-supervised constraints (ICIP2019), Jin et al [PDF]
- Residual multiscale based single image deraining (BMVC2019), Zheng et al.
- Erl-net: Entangled representation learning for single image de-raining (ICCV2019), Wang et al. [code]
- Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining (CVPR2019), Rajeev Yasarla et al.[Code]
- Heavy rain image restoration: Integrating physics model and conditional adversarial learning (CVPR2019), Li et al.[Code]
- Progressive image deraining networks: A better and simpler baseline (CVPR2019), Ren et al [PDF] [Code]
- Spatial attentive single-image deraining with a high quality real rain dataset (CVPR2019), Wang et al [Project] [PDF] [Code]
- Lightweight pyramid networks for image deraining (TNNLS2019), Fu et al [PDF] [Code]
- Joint rain detection and removal from a single image with contextualized deep networks (TPAMI2019), Yang et al [PDF] [Code]
- Scale-free single image deraining via visibility-enhanced recurrent wavelet learning (TIP2019), Yang et al.[PDF]
- Towards scale-free rain streak removal via selfsupervised fractal band learning (AAAI2020), Yang et al.[Code]
- Structural Residual Learning for Single Image Rain Removal(Arxiv2020), Wang et al. [PDF]
- All in One Bad Weather Removal Using Architectural Search (CVPR2020), Li et al.[PDF]
- Syn2Real Transfer Learning for Image Deraining Using Gaussian Processes(CVPR2020), Rajeev Yasarla et al. [Code]
- Multi-Scale Progressive Fusion Network for Single Image Deraining(CVPR2020), Jiang et al. [Code]
- Detail-recovery Image Deraining via Context Aggregation Networks(CVPR2020), Deng et al.[Code]
- Variational image deraining(WACV2020), Du et al.[PDF]
- DerainCycleGAN: Rain Attentive CycleGAN for Single Image Deraining and Rainmaking(TIP2021), Wei et al.[PDF][Code]
- Unsupervised Image Deraining: Optimization Model Driven Deep CNN(ACM MM2021), Yu et al.[PDF] [Code]
- Multi-Level Memory Compensation Network for Rain Removal via Divide-and-Conquer Strategy(JSTSP2021), Jiang et al. [PDF]
- Robust Representation Learning with Feedback for Single Image Deraining (CVPR2021), Chen et al. [PDF] [Code]
- Memory Oriented Transfer Learning for Semi-Supervised Image Deraining (CVPR2021), Huang et al. [PDF] [Code]
- Image De-raining via Continual Learning (CVPR2021), Zhou et al. [PDF]
- Improving De-raining Generalization via Neural Reorganization (ICCV2021), Xiao et al. [PDF]
- Learning Multiple Adverse Weather Removal via Two-stage Knowledge Learning and Multi-contrastive Regularization: Toward a Unified Model (CVPR2022), Chen et al. [PDF] [Code]
- GIQE: Generic Image Quality Enhancement via Nth Order Iterative Degradation (CVPR2022), Pranjay Shyam et al. [PDF] [Supplementary Materials]
- Dual Heterogeneous Complementary Networks for Single Image Deraining (CVPR2022), Yuuto Nanba et al. [PDF]
- TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions (CVPR2022), Jeya Maria Jose Valanarasu et al. [PDF] [Code]
- Unpaired Deep Image Deraining Using Dual Contrastive Learning (CVPR2022), Chen et al. [Project] [PDF]
- Unsupervised Deraining: Where Contrastive Learning Meets Self-similarity (CVPR2022), Ye et al. [PDF]
- Single Image Deraining Network with Rain Embedding Consistency and Layered LSTM (WACV2022), Li et al. [PDF]
- FLUID: Few-Shot Self-Supervised Image Deraining (WACV2022), Shyam Nandan Rai et al. [PDF] [Code]
- Single image rain removal using recurrent scale-guide networks (Neurocomputing2022), Wang et al. [PDF]
- Non-local channel aggregation network for single image rain removal (Neurocomputing2022), Su et al. [PDF]
- Joint Model-driven and Data-driven
- Deep Layer Prior Optimization for Single Image Rain Streaks Removal (ICASSP2018), Liu et al [PDF]
- Learning bilevel layer priors for single image rain streaks removal (SP Letters 2018), Mu et al [PDF]
- Semi-supervised transfer learning for image rain removal (CVPR2019), Wei et al [PDF] [Code]
- Knowledge-driven deep unrolling for robust image layer separation (TNNLS2019), Liu et al.
- A Model-driven Deep Neural Network for Single Image Rain Removal (CVPR2020), Wang et al. [PDF] [Supp] [Code]
- Unpaired Learning for Deep Image Deraining with Rain Direction Regularizer (ICCV2021), Liu et al. [Project] [PDF]
- Structure-Preserving Deraining with Residue Channel Prior Guidance (ICCV2021), Yi et al. [PDF] [Code]
- Multi-Decoding Deraining Network and Quasi-Sparsity Based Training (CVPR2021), Wang et al [PDF]
- Towards Robust Rain Removal Against Adversarial Attacks: A Comprehensive Benchmark Analysis and Beyond (CVPR2022), Yu et al. [PDF] [Code]
- Reivew paper
Datasets and Discriptions
-
Video
- Synthetic Datasets: highway and park.
- Real Datasets: compfinal and night. Please download from [Baidu Netdisk] provided by Li Minghan.
-
Single Image
-
Synthetic Datasets: RainTrainL/Rain100L, RainTrainH/Rain100H, Rain12600/Rain1400, and Rain12. Please download from [Baidu Netdisk] provided by Ren Dongwei.
-
Real Datasets: Please download SPA-Data from [Baidu Netdisk, key: 4fwo] provided by Wang Tianyu and Internet-Data from the link provided by Weiwei .
-
Other important datasets: Rain800 [Link]， Rain12000[Link]， RainCityscapes[Link]， NYU-Rain[Link]， MPID [Link]
-
*We note that:
i. RainTrainL/Rain100L and RainTrainH/Rain100H are synthesized by Yang Wenhan. Rain12600/Rain1400 is from Fu Xueyang and Rain12 is from Li Yu.
ii. In video experiment, the rain-removed results of the deep learning method are provided by the author Yang Wenhan. Really thanks!
iii. In single image experiment, we seperately retrain all the recent state-of-the-art methods via the three training datasets: RainTrainL(200 input/clean image pairs), RainTrainH(1800 pairs), and Rain12600(12600 pairs), and then evaluate their rain removal performance based on the correponding test datasets: Rain100L(100 pairs), Rain100H(100 pairs), and Rain1400(1400 pairs). Besides, the trained model obtained by RainTrainL is adpoted to predict rain-removed results of Rain12(12 pairs). Moreover, we utilize the Internet-Data(147 input images) and SPA-Data(1000 pairs) to compare the generalization ability.
iiii. In single image experiment, when training the semi-supervised method--SIRR, we always utilize Internet-Data as unsupervised samples.
Image Quality Metrics
- Reference
- PSNR (Peak Signal-to-Noise Ratio) [PDF] [Matlab Code] [Python Code]
- SSIM (Structural Similarity) [PDF] [Matlab Code] [Python Code]
- VIF (Visual Quality) [PDF] [Matlab Code]
- FSIM (Feature Similarity) [PDF] [Matlab Code])
- Non-reference
- NIQE (Naturalness Image Quality Evaluator)[PDF] [Matlab Code]
- BRISQUE (Blind/Referenceless Image Spatial Quality Evaluator)[PDF] [Matlab Code]
- SSEQ (Spatial-Spectral Entropy-based Quality)[PDF] [Matlab Code]
*Please note that all quantitative results in our survey paper are computed based on Y channel.
Contact
If you have any question, please feel free to concat Hong Wang (Email: [email protected]).