• Stars
    star
    137
  • Rank 266,121 (Top 6 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 3 years ago
  • Updated 9 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Exploring Image Deblurring via Blur Kernel Space (CVPR'21)

Exploring Image Deblurring via Encoded Blur Kernel Space

About the project

We introduce a method to encode the blur operators of an arbitrary dataset of sharp-blur image pairs into a blur kernel space. Assuming the encoded kernel space is close enough to in-the-wild blur operators, we propose an alternating optimization algorithm for blind image deblurring. It approximates an unseen blur operator by a kernel in the encoded space and searches for the corresponding sharp image. Due to the method's design, the encoded kernel space is fully differentiable, thus can be easily adopted in deep neural network models.

Blur kernel space

Detail of the method and experimental results can be found in our following paper:

@inproceedings{m_Tran-etal-CVPR21, 
โ€ƒ author = {Phong Tran and Anh Tran and Quynh Phung and Minh Hoai}, 
โ€ƒ title = {Explore Image Deblurring via Encoded Blur Kernel Space}, 
โ€ƒ year = {2021}, 
โ€ƒ booktitle = {Proceedings of the {IEEE} Conference on Computer Vision and Pattern Recognition (CVPR)} 
}

Please CITE our paper whenever this repository is used to help produce published results or incorporated into other software.

Open In Colab

Table of Content

Getting started

Prerequisites

  • Python >= 3.7
  • Pytorch >= 1.4.0
  • CUDA >= 10.0

Installation

git clone https://github.com/VinAIResearch/blur-kernel-space-exploring.git
cd blur-kernel-space-exploring


conda create -n BlurKernelSpace -y python=3.7
conda activate BlurKernelSpace
conda install --file requirements.txt

Training and evaluation

Preparing datasets

You can download the datasets in the model zoo section.

To use your customized dataset, your dataset must be organized as follow:

root
โ”œโ”€โ”€ blur_imgs
    โ”œโ”€โ”€ 000
    โ”œโ”€โ”€โ”€โ”€ 00000000.png
    โ”œโ”€โ”€โ”€โ”€ 00000001.png
    โ”œโ”€โ”€โ”€โ”€ ...
    โ”œโ”€โ”€ 001
    โ”œโ”€โ”€โ”€โ”€ 00000000.png
    โ”œโ”€โ”€โ”€โ”€ 00000001.png
    โ”œโ”€โ”€โ”€โ”€ ...
โ”œโ”€โ”€ sharp_imgs
    โ”œโ”€โ”€ 000
    โ”œโ”€โ”€โ”€โ”€ 00000000.png
    โ”œโ”€โ”€โ”€โ”€ 00000001.png
    โ”œโ”€โ”€โ”€โ”€ ...
    โ”œโ”€โ”€ 001
    โ”œโ”€โ”€โ”€โ”€ 00000000.png
    โ”œโ”€โ”€โ”€โ”€ 00000001.png
    โ”œโ”€โ”€โ”€โ”€ ...

where root, blur_imgs, and sharp_imgs folders can have arbitrary names. For example, let root, blur_imgs, sharp_imgs be REDS, train_blur, train_sharp respectively (That is, you are using the REDS training set), then use the following scripts to create the lmdb dataset:

python create_lmdb.py --H 720 --W 1280 --C 3 --img_folder REDS/train_sharp --name train_sharp_wval --save_path ../datasets/REDS/train_sharp_wval.lmdb
python create_lmdb.py --H 720 --W 1280 --C 3 --img_folder REDS/train_blur --name train_blur_wval --save_path ../datasets/REDS/train_blur_wval.lmdb

where (H, C, W) is the shape of the images (note that all images in the dataset must have the same shape), img_folder is the folder that contains the images, name is the name of the dataset, and save_path is the save destination (save_path must end with .lmdb).

When the script is finished, two folders train_sharp_wval.lmdb and train_blur_wval.lmdb will be created in ./REDS.

Training

To do image deblurring, data augmentation, and blur generation, you first need to train the blur encoding network (The F function in the paper). This is the only network that you need to train. After creating the dataset, change the value of dataroot_HQ and dataroot_LQ in options/kernel_encoding/REDS/woVAE.yml to the paths of the sharp and blur lmdb datasets that were created before, then use the following script to train the model:

python train.py -opt options/kernel_encoding/REDS/woVAE.yml

where opt is the path to yaml file that contains training configurations. You can find some default configurations in the options folder. Checkpoints, training states, and logs will be saved in experiments/modelName. You can change the configurations (learning rate, hyper-parameters, network structure, etc) in the yaml file.

Testing

Data augmentation

To augment a given dataset, first, create an lmdb dataset using scripts/create_lmdb.py as before. Then use the following script:

python data_augmentation.py --target_H=720 --target_W=1280 \
			    --source_H=720 --source_W=1280\
			    --augmented_H=256 --augmented_W=256\
                            --source_LQ_root=datasets/REDS/train_blur_wval.lmdb \
                            --source_HQ_root=datasets/REDS/train_sharp_wval.lmdb \
			    --target_HQ_root=datasets/REDS/test_sharp_wval.lmdb \
                            --save_path=results/GOPRO_augmented \
                            --num_images=10 \
                            --yml_path=options/data_augmentation/default.yml

(target_H, target_W), (source_H, source_W), and (augmented_H, augmented_W) are the desired shapes of the target images, source images, and augmented images respectively. source_LQ_root, source_HQ_root, and target_HQ_root are the paths of the lmdb datasets for the reference blur-sharp pairs and the input sharp images that were created before. num_images is the size of the augmented dataset. model_path is the path of the trained model. yml_path is the path to the model configuration file. Results will be saved in save_path.

Data augmentation examples

Generate novel blur kernels

To generate a blur image given a sharp image, use the following command:

python generate_blur.py --yml_path=options/generate_blur/default.yml \
		        --image_path=imgs/sharp_imgs/mushishi.png \
			--num_samples=10
			--save_path=./res.png

where model_path is the path of the pre-trained model, yml_path is the path of the configuration file. image_path is the path of the sharp image. After running the script, a blur image corresponding to the sharp image will be saved in save_path. Here is some expected output: kernel generating examples Note: This only works with models that were trained with --VAE flag. The size of input images must be divisible by 128.

Generic Deblurring

To deblur a blurry image, use the following command:

python generic_deblur.py --image_path imgs/blur_imgs/blur1.png --yml_path options/generic_deblur/default.yml --save_path ./res.png

where image_path is the path of the blurry image. yml_path is the path of the configuration file. The deblurred image will be saved to save_path.

Image deblurring examples

Deblurring using sharp image prior

First, you need to download the pre-trained styleGAN or styleGAN2 networks. If you want to use styleGAN, download the mapping and synthesis networks, then rename and copy them to experiments/pretrained/stylegan_mapping.pt and experiments/pretrained/stylegan_synthesis.pt respectively. If you want to use styleGAN2 instead, download the pretrained model, then rename and copy it to experiments/pretrained/stylegan2.pt.

To deblur a blurry image using styleGAN latent space as the sharp image prior, you can use one of the following commands:

python domain_specific_deblur.py --input_dir imgs/blur_faces \
		    --output_dir experiments/domain_specific_deblur/results \
		    --yml_path options/domain_specific_deblur/stylegan.yml  # Use latent space of stylegan
python domain_specific_deblur.py --input_dir imgs/blur_faces \
		    --output_dir experiments/domain_specific_deblur/results \
		    --yml_path options/domain_specific_deblur/stylegan2.yml  # Use latent space of stylegan2

Results will be saved in experiments/domain_specific_deblur/results. Note: Generally, the code still works with images that have the size divisible by 128. However, since our blur kernels are not uniform, the size of the kernel increases as the size of the image increases.

PULSE-like Deblurring examples

Model Zoo

Pretrained models and corresponding datasets are provided in the below table. After downloading the datasets and models, follow the instructions in the testing section to do data augmentation, generating blur images, or image deblurring.

Model name dataset(s) status
REDS woVAE REDS โœ”๏ธ
GOPRO woVAE GOPRO โœ”๏ธ
GOPRO wVAE GOPRO โœ”๏ธ
GOPRO + REDS woVAE GOPRO, REDS โœ”๏ธ

Notes and references

The training code is borrowed from the EDVR project: https://github.com/xinntao/EDVR

The backbone code is borrowed from the DeblurGAN project: https://github.com/KupynOrest/DeblurGAN

The styleGAN code is borrowed from the PULSE project: https://github.com/adamian98/pulse

The stylegan2 code is borrowed from https://github.com/rosinality/stylegan2-pytorch

More Repositories

1

PhoGPT

PhoGPT: Generative Pre-training for Vietnamese (2023)
Python
720
star
2

PhoBERT

PhoBERT: Pre-trained language models for Vietnamese (EMNLP-2020 Findings)
658
star
3

BERTweet

BERTweet: A pre-trained language model for English Tweets (EMNLP-2020)
Python
573
star
4

WaveDiff

Official Pytorch Implementation of the paper: Wavelet Diffusion Models are fast and scalable Image Generators (CVPR'23)
Python
372
star
5

CPM

๐Ÿ’„ Lipstick ain't enough: Beyond Color-Matching for In-the-Wild Makeup Transfer (CVPR 2021)
Python
364
star
6

XPhoneBERT

XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech (INTERSPEECH 2023)
Python
292
star
7

Anti-DreamBooth

Anti-DreamBooth: Protecting users from personalized text-to-image synthesis (ICCV 2023)
Python
205
star
8

LFM

Official PyTorch implementation of the paper: Flow Matching in Latent Space
Python
184
star
9

PhoNLP

PhoNLP: A BERT-based multi-task learning model for part-of-speech tagging, named entity recognition and dependency parsing (NAACL 2021)
Python
137
star
10

dict-guided

Dictionary-guided Scene Text Recognition (CVPR-2021)
Python
126
star
11

VinAI_Translate

A Vietnamese-English Neural Machine Translation System (INTERSPEECH 2022)
123
star
12

MagNet

Progressive Semantic Segmentation (CVPR-2021)
Python
114
star
13

Warping-based_Backdoor_Attack-release

WaNet - Imperceptible Warping-based Backdoor Attack (ICLR 2021)
Python
111
star
14

HyperInverter

HyperInverter: Improving StyleGAN Inversion via Hypernetwork (CVPR 2022)
Python
111
star
15

BARTpho

BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese (INTERSPEECH 2022)
99
star
16

PhoWhisper

PhoWhisper: Automatic Speech Recognition for Vietnamese (2024)
96
star
17

ISBNet

ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution (CVPR 2023)
Python
93
star
18

Dataset-Diffusion

Dataset Diffusion: Diffusion-based Synthetic Data Generation for Pixel-Level Semantic Segmentation (NeurIPS2023)
Jupyter Notebook
87
star
19

JointIDSF

BERT-based joint intent detection and slot filling with intent-slot attention mechanism (INTERSPEECH 2021)
Python
84
star
20

3D-UCaps

3D-UCaps: 3D Capsules Unet for Volumetric Image Segmentation (MICCAI 2021)
Python
65
star
21

PhoNER_COVID19

COVID-19 Named Entity Recognition for Vietnamese (NAACL 2021)
63
star
22

PCC-pytorch

A pytorch implementation of the paper "Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control"
Python
59
star
23

Counting-DETR

Few-shot Object Counting and Detection (ECCV 2022)
Python
56
star
24

PSENet-Image-Enhancement

PSENet: Progressive Self-Enhancement Network for Unsupervised Extreme-Light Image Enhancement (WACV 2023)
Python
54
star
25

LeMul

Toward Realistic Single-View 3D Object Reconstruction with Unsupervised Learning from Multiple Images (ICCV 2021)
Python
51
star
26

DSW

Distributional Sliced-Wasserstein distance code
Python
47
star
27

PhoMT

PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation (EMNLP 2021)
40
star
28

single_image_hdr

Single-Image HDR Reconstruction by Multi-Exposure Generation (WACV 2023)
Python
38
star
29

SwiftBrush

SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation (CVPR 2024)
Python
37
star
30

tise-toolbox

TISE: Bag of Metrics for Text-to-Image Synthesis Evaluation (ECCV 2022)
Python
33
star
31

Point-Unet

Point-Unet: A Context-aware Point-based Neural Network for Volumetric Segmentation (MICCAI 2021)
Python
32
star
32

COVID19Tweet

WNUT-2020 Task 2: Identification of informative COVID-19 English Tweets
Python
30
star
33

CREPS

Efficient Scale-Invariant Generator with Column-Row Entangled Pixel Synthesis (CVPR 2023)
Python
30
star
34

ViText2SQL

ViText2SQL: A dataset for Vietnamese Text-to-SQL semantic parsing (EMNLP-2020 Findings)
28
star
35

input-aware-backdoor-attack-release

Input-aware Dynamic Backdoor Attack (NeurIPS 2020)
Python
27
star
36

QC-StyleGAN

QC-StyleGAN - Quality Controllable Image Generation and Manipulation (NeurIPS 2022)
Python
26
star
37

fsvc-ata

Inductive and Transductive Few-Shot Video Classification via Appearance and Temporal Alignments (ECCV 2022)
Python
23
star
38

GeoFormer

Geodesic-Former: a Geodesic-Guided Few-shot 3D Point Cloud Instance Segmenter (ECCV 2022)
Python
23
star
39

PhoST

A High-Quality and Large-Scale Dataset for English-Vietnamese Speech Translation (INTERSPEECH 2022)
19
star
40

MISCA

MISCA: A Joint Model for Multiple Intent Detection and Slot Filling with Intent-Slot Co-Attention (EMNLP 2023 - Findings)
Python
18
star
41

PC3-pytorch

Predictive Coding for Locally-Linear Control (ICML-2020)
Python
16
star
42

Open3DIS

Open3DIS: Open-vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024)
Python
16
star
43

EFHQ

Code and data for the CVPR24 paper "EFHQ: Multi-purpose ExtremePose-Face-HQ dataset" [CVPR'24]
Python
15
star
44

TPC-tensorflow

Temporal Predictive Coding For Model-Based Planning In Latent Space (ICML-2021)
Python
14
star
45

iFS-RCNN

iFS-RCNN: An Incremental Few-shot Instance Segmenter (CVPR 2022)
Python
14
star
46

GaPro

GaPro: Box-Supervised 3D Point Cloud Instance Segmentation Using Gaussian Processes as Pseudo Labelers (ICCV 2023)
Python
13
star
47

HyperCUT

HyperCUT: Video Sequence from a Single Blurry Image using Unsupervised Ordering (CVPR'23)
Python
12
star
48

LP-OVOD

LP-OVOD: Open-Vocabulary Object Detection by Linear Probing (WACV 2024)
Python
11
star
49

selfsup_pcd

Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud Analysis (ACCV 2022)
Python
8
star
50

PointSWD

Point-set Distances for Learning Representations of 3D Point Clouds (ICCV 2021)
Python
7
star
51

PhoATIS_Disfluency

From Disfluency Detection to Intent Detection and Slot Filling (INTERSPEECH 2022)
7
star
52

JPIS

JPIS: A Joint Model for Profile-Based Intent Detection and Slot Filling with Slot-to-Intent Attention (ICASSP 2024)
Python
6
star
53

SA-DPM

Official PyTorch implementation of "On Inference Stability for Diffusion Models" (AAAI'24)
Python
5
star
54

PhoDisfluency

Disfluency Detection for Vietnamese (WNUT 2022)
4
star
55

DiverseDream

DiverseDream: A Technique to Generate Diverse 3D Objects from the Same Text Prompt (ECCV '24)
Python
3
star
56

robust-bayesian-recourse

Robust Bayesian Recourse: a robust model-agnostic algorithmic recourse method (UAI'22)
Python
2
star
57

RDUOT

Official code for ECCV 2024 paper โ€œA high-quality robust diffusion framework for corrupted datasetโ€
Python
1
star
58

LAMPAT

LAMPAT: Low-rank Adaptation Multilingual Paraphrasing using Adversarial Training (AAAI'24)
Python
1
star