Learning Blind Motion Deblurring
TensorFlow implementation of multi-frame blind deconvolution:
Learning Blind Motion Deblurring
Patrick Wieschollek, Michael Hirsch, Bernhard Schölkopf, Hendrik P.A. Lensch
ICCV 2017
Download results from the paper. We propose to use the saccade-viewer to compare images qualitatively.
Prerequisites
1. Get YouTube videos
The first step is to gather videos from some arbitrary sources. We use YouTube to get some videos with diverse content and recording equipment. To download these videos, we use the python-tool youtube-dl
.
pip install youtube-dl --user
Some examples are given in download_videos.sh
. Note, you can use whatever mp4 video you want to use for this task. In fact, for this re-implementation we use some other videos, which also work well.
2. Generate Synthetic Motion Blur
Now, we use optical flow to synthetically add motion blur. We used the most simple OpticalFlow method, wich provides reasonable results (we average frames anyway):
cd synthblur
mkdir build && cd build
cmake ..
make all
To convert a video input.mp4
into a blurry version, run
./synthblur/build/convert "input.mp4"
This gives you multiple outputs:
- 'input.mp4_blurry.mp4'
- 'input.mp4_sharp.mp4'
- 'input.mp4_flow.mp4'
Adding blur from synthetic camera shake is done on-the-fly (see psf.py
).
3. Building a Database
For performance reasons we randomly sample frames from all videos beforehand and store 5+5 consecutive frames (sharp+blurry) into an LMDB file (for training/validation/testing).
I use
#!/bin/bash
for i in `seq 1 30`; do
python data_sampler.py --pattern '/graphics/scratch/wieschol/YouTubeDataset/train/*_blurry.mp4' --lmdb /graphics/scratch/wieschol/YouTubeDataset/train$i.lmdb --num 5000
done
for i in `seq 1 10`; do
python data_sampler.py --pattern '/graphics/scratch/wieschol/YouTubeDataset/val/*_blurry.mp4' --lmdb /graphics/scratch/wieschol/YouTubeDataset/val$i.lmdb --num 5000
done
To visualize the training examples just run
python data_provider.py --lmdb /graphics/scratch/wieschol/YouTubeDataset/train1.lmdb --show --num 5000
Training
This re-implementation uses TensorPack instead of the used custom library for the paper. Starting training is done by
python learning_blind_motion_deblurring.py --gpu 0,1 --data path/to/lmdb-files/
Results
See the release section for full-resolution images produced by our approach.
Further experiments
We further tried a convLSTM/convGRU and a multi-scale approach (instead of the simple test from the paper). These script are available in additional_scripts
.
Notes
I re-trained a slightly larger model in TensorPack just for testing the TensorPack library some months ago. It seems to have similar performance (although it is not compatible with this GitHub project). Find the inference code/weights here.
Please note, TensorFlow introduce some changes over time. This setup is tested under
- Python 2.7
- Tensorflow-gpu v1.9.0
- Cuda 9.0
- Tensorpack 0.1.6 (from 17 Feb 2017)