DeepRemaster: Temporal Source-Reference Attention Networks for Comprehensive Video Enhancement

Satoshi Iizuka and Edgar Simo-Serra

Overview

This code provides an implementation of the research paper:

  "DeepRemaster: Temporal Source-Reference Attention Networks for Comprehensive Video Enhancement"
  Satoshi Iizuka and Edgar Simo-Serra
  ACM Transaction on Graphics (Proc. of SIGGRAPH ASIA 2019), 2019

We learn to semi-automatically remaster vintage videos with a deep convolutional network. Our network is based on temporal convolutions with source-reference attention mechanisms trained on videos with example-based deterioration simulation, which allows us to automatically remove film noises, improve contrast and sharpness, and add color based on reference color frames created manually. See our project page for more detailed information.

License

  Copyright (C) <2019> <Satoshi Iizuka and Edgar Simo-Serra>

  This work is licensed under the Creative Commons
  Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy
  of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or
  send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

  Satoshi Iizuka, University of Tsukuba
  [email protected], http://iizuka.cs.tsukuba.ac.jp/index_eng.html
  
  Edgar Simo-Serra, Waseda University
  [email protected], https://esslab.jp/~ess/

Dependencies

• PyTorch (0.4.1+) torchvision
• FFmpeg (requires to be configured with --enable-libx264)
• opencv (3.4.1+)
• scikit-image
• tqdm

For information on how to install PyTorch, please refer to the PyTorch website. FFmpeg should be installed with libx264 support, which can be installed in Anaconda by using conda install x264 ffmpeg -c conda-forge.

Usage

First, download the model by running the download script:

bash download_model.sh

Basic usage is:

python remaster.py --input <input_video> --reference_dir <directory_of_reference_images>

The input video will be automatically restored and colorized based on the reference color frames using the model. If you want to perform restoration only, use --disable_colorization option.

Other options:

• --gpu: Use GPU for the computation (recommended). Defaults to false.
• --disable_colorization: Disable colorization and only perform restoration with enhancement. Defaults to false.
• --mindim: Minimum edge dimension of the input video. Defaults to 320.

For example:

python remaster.py --input example/a-bomb_blast_effects_part.mp4 --reference_dir example/references --gpu

Preparing Reference Images

To prepare reference color images for your own video, it is recommended to first extract reference frames from the video using a scene detection technique such as pyscenedetect. Afterwards, colorize them by leveraging image editing software or recent interactive colorization techniques such as the Interactive Deep Colorization [Zhang et al. 2017].

Notes

• This is developed on a Linux machine running Ubuntu 18.04 during late 2018.
• We recommend using GPU with 4GB+ memory for fast computation.
• Provided model and sample code are under a non-commercial creative commons license.

Dataset

The list of video URLs used for training the model is available here (unfortunately several links are no longer available).

The noise data used for simulating old film degradation is available here (898MB).

Citing

If you use this code please cite:

@Article{IizukaSIGGRAPHASIA2019,
  author = {Satoshi Iizuka and Edgar Simo-Serra},
  title = {{DeepRemaster: Temporal Source-Reference Attention Networks for Comprehensive Video Enhancement}},
  journal = "ACM Transactions on Graphics (Proc. of SIGGRAPH ASIA)",
  year = 2019,
  volume = 38,
  number = 6,
  pages = 1--13,
  articleno = 176,
}