Multi-Stage Progressive Image Restoration (CVPR 2021)

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao

News

We are happy to see that our work has inspired the Winning Solutions in NTIRE 2021 challenges:

• Dual-pixel Defocus Deblurring Challenge -- MRNet: Multi Refinement Network for Dual-pixel Images Defocus Deblurring
• Image Deblurring Challenge -- HINet: Half Instance Normalization Network for Image Restoration

Abstract: Image restoration tasks demand a complex balance between spatial details and high-level contextualized information while recovering images. In this paper, we propose a novel synergistic design that can optimally balance these competing goals. Our main proposal is a multi-stage architecture, that progressively learns restoration functions for the degraded inputs, thereby breaking down the overall recovery process into more manageable steps. Specifically, our model first learns the contextualized features using encoder-decoder architectures and later combines them with a high-resolution branch that retains local information. At each stage, we introduce a novel per-pixel adaptive design that leverages in-situ supervised attention to reweight the local features. A key ingredient in such a multi-stage architecture is the information exchange between different stages. To this end, we propose a two-faceted approach where the information is not only exchanged sequentially from early to late stages, but lateral connections between feature processing blocks also exist to avoid any loss of information. The resulting tightly interlinked multi-stage architecture, named as MPRNet, delivers strong performance gains on ten datasets across a range of tasks including image deraining, deblurring, and denoising. For example, on the Rain100L, GoPro and DND datasets, we obtain PSNR gains of 4 dB, 0.81 dB and 0.21 dB, respectively, compared to the state-of-the-art.

Network Architecture

Overall Framework of MPRNet	Supervised Attention Module (SAM)

Installation

The model is built in PyTorch 1.1.0 and tested on Ubuntu 16.04 environment (Python3.7, CUDA9.0, cuDNN7.5).

For installing, follow these intructions

conda create -n pytorch1 python=3.7
conda activate pytorch1
conda install pytorch=1.1 torchvision=0.3 cudatoolkit=9.0 -c pytorch
pip install matplotlib scikit-image opencv-python yacs joblib natsort h5py tqdm

Install warmup scheduler

cd pytorch-gradual-warmup-lr; python setup.py install; cd ..

Quick Run

To test the pre-trained models of Deblurring, Deraining, Denoising on your own images, run

python demo.py --task Task_Name --input_dir path_to_images --result_dir save_images_here

Here is an example to perform Deblurring:

python demo.py --task Deblurring --input_dir ./samples/input/ --result_dir ./samples/output/

Training and Evaluation

Training and Testing codes for deblurring, deraining and denoising are provided in their respective directories.

Results

Experiments are performed for different image processing tasks including, image deblurring, image deraining and image denoising. Images produced by MPRNet can be downloaded from Google Drive links: Deblurring, Deraining, and Denoising.

Image Deblurring (click to expand)

Deblurring on Synthetic Datasets.	Deblurring on Real Dataset.

Image Deraining (click to expand)

Citation

If you use MPRNet, please consider citing:

@inproceedings{Zamir2021MPRNet,
    title={Multi-Stage Progressive Image Restoration},
    author={Syed Waqas Zamir and Aditya Arora and Salman Khan and Munawar Hayat
            and Fahad Shahbaz Khan and Ming-Hsuan Yang and Ling Shao},
    booktitle={CVPR},
    year={2021}
}

Contact

Should you have any question, please contact [email protected]

Our Related Works

• Learning Enriched Features for Fast Image Restoration and Enhancement, TPAMI 2022. Paper | Code
• Restormer: Efficient Transformer for High-Resolution Image Restoration, CVPR 2022. Paper | Code
• Learning Enriched Features for Real Image Restoration and Enhancement, ECCV 2020. Paper | Code
• CycleISP: Real Image Restoration via Improved Data Synthesis, CVPR 2020. Paper | Code