• Stars
    star
    174
  • Rank 219,104 (Top 5 %)
  • Language
    Jupyter Notebook
  • License
    Other
  • Created over 2 years ago
  • Updated 7 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[WACV 2023] Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand

FcF-Inpainting

Open In Colab Huggingface space Framework: PyTorch License

Jitesh Jain†, Yuqian Zhou†, Ning Yu, Humphrey Shi, WACV 2023

† Equal Contribution

[Project Page] [arXiv] [pdf] [BibTeX]

This repo contains the code for our paper Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand.

FcFGAN

News

  • [October 6, 2022]: You can host your own FcF-Inpainting demo using streamlit by following the instructions here.
  • [September 5, 2022]: FcF-Inpainting is now available in the image inpainting tool Lama Cleaner. Thanks to @Sanster for integrating FcF-Inpainting into Lama Cleaner!
  • [August 16, 2022]: FcF-Inpainting is accepted to WACV 2023!
  • [August 5, 2022]: Project Page, ArXiv Preprint and GitHub Repo are public!

Contents

  1. Setup Instructions
  2. Dataset Preparation
  3. Training and Evaluation
  4. Citing FcF-Inpainting

1. Setup Instructions

  • Clone the repo:

    git clone https://github.com/SHI-Labs/FcF-Inpainting.git
    cd FcF-Inpainting
  • Create a conda environment:

    conda create --name fcfgan python=3.7
    conda activate fcfgan
  • Install Pytorch 1.7.1 and other dependencies:

    pip3 install -r requirements.txt
    export TORCH_HOME=$(pwd) && export PYTHONPATH=.
  • Download the models for the high receptive perceptual loss:

    mkdir -p ade20k/ade20k-resnet50dilated-ppm_deepsup/
    wget -P ade20k/ade20k-resnet50dilated-ppm_deepsup/ http://sceneparsing.csail.mit.edu/model/pytorch/ade20k-resnet50dilated-ppm_deepsup/encoder_epoch_20.pth

2. Dataset Preparation

CelebA-HQ Dataset

Training Data

  • Download data256x256.zip from gdrive.

    mkdir -p datasets/
    # unzip & split into train/test/visualization
    bash tools/prepare_celebahq.sh
    
    datasets
    ├── celeba-hq-dataset
    │   ├── train_256
    │   ├── val_source_256
    │   ├── visual_test_source_256

Evaluation Data

  • Generate 2k (image, mask) pairs to be used for evaluation.

    bash tools/prepare_celebahq_evaluation.sh

Places2 Dataset

Training Data

  • Download the Places2 dataset:

    mkdir -p datasets/
    mkdir datasets/places2_dataset/
    wget http://data.csail.mit.edu/places/places365/train_large_places365challenge.tar
    tar -xvf train_large_places365challenge.tar -C datasets/places2_dataset/
    mv datasets/places2_datasets/data_large datasets/places2_dataset/train
    
    wget http://data.csail.mit.edu/places/places365/val_large.tar
    tar -xvf val_large.tar -C datasets/places2_dataset/
    mv datasets/places2_dataset/val_large datasets/places2_dataset/val
    
    datasets
    ├── places2_dataset
    │   ├── train
    │   ├── val
  • Generate 10k (image, mask) pairs to be used for validation during training.

    bash tools/prepare_places_val.sh

Evaluation Data

Irregular Mask Strategy
  • Generate 30k (image, mask) pairs to be used for evaluation.

    bash tools/prepare_places_evaluation.sh
Segmentation Mask strategy
  • Install Detectron2-v0.5.

    python -m pip install detectron2==0.5 -f \
    https://dl.fbaipublicfiles.com/detectron2/wheels/cu110/torch1.7/index.html
  • Download networks for segmentation masks:

    mkdir -p ade20k/ade20k-resnet50dilated-ppm_deepsup/
    wget -P ade20k/ade20k-resnet50dilated-ppm_deepsup/ http://sceneparsing.csail.mit.edu/model/pytorch/ade20k-resnet50dilated-ppm_deepsup/encoder_epoch_20.pth
    wget -P ade20k/ade20k-resnet50dilated-ppm_deepsup/ http://sceneparsing.csail.mit.edu/model/pytorch/ade20k-resnet50dilated-ppm_deepsup/decoder_epoch_20.pth
  • Generate (image, mask) pairs to be used for segmentation mask based evaluation.

    bash tools/prepare_places_segm_evaluation.sh

Note: The pairs are only generated for images with detected instances.

3. Training and Evaluation

places

Training on 256x256

  • Execute the following command to start training for 25M images on 8 gpus with 16 images per gpu:

    python train.py \
        --outdir=training-runs-inp \
        --img_data=datasets/places2_dataset/train \
        --gpus 8 \
        --kimg 25000 \
        --gamma 10 \
        --aug 'noaug' \
        --metrics True \
        --eval_img_data datasets/places2_dataset/evaluation/random_segm_256
        --batch 128

Note: If the process hangs on Setting up PyTorch plugin ..., refer to this issue.

Evaluation

Pretrained Models

checkpoint Description
places_512.pkl Model trained on 512x512 for 25M Places2 images
places.pkl Model trained on 256x256 for 25M Places2 images
celeba-hq.pkl Model trained on 256x256 for 25M CelebA-HQ images
  • Run the following command to calculate the metric scores (fid, ssim and lpips) using 8 gpus:

    python evaluate.py \
        --img_data=datasets/places2_dataset/evaluation/random_segm_256 \
        --network=[path-to-checkpoint] \
        --num_gpus=8

celeba

Demo

  • Run the following command and find the results in the visualizations/ folder:

    python demo.py \
    --img_data=datasets/demo/places2 \
    --network=[path-to-checkpoint] \
    --resolution 256

4. Citing FcF-Inpainting

@inproceedings{jain2022keys,
  title={Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand},
  author={Jitesh Jain and Yuqian Zhou and Ning Yu and Humphrey Shi},
  booktitle={WACV},
  year={2023}
} 

Acknowledgement

Code is heavily based on the following repositories: stylegan2-ada-pytorch and lama.

More Repositories

1

OneFormer

OneFormer: One Transformer to Rule Universal Image Segmentation, arxiv 2022 / CVPR 2023
Jupyter Notebook
1,461
star
2

Versatile-Diffusion

Versatile Diffusion: Text, Images and Variations All in One Diffusion Model, arXiv 2022 / ICCV 2023
Python
1,300
star
3

Neighborhood-Attention-Transformer

Neighborhood Attention Transformer, arxiv 2022 / CVPR 2023. Dilated Neighborhood Attention Transformer, arxiv 2022
Python
1,037
star
4

Prompt-Free-Diffusion

Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models, arxiv 2023 / CVPR 2024
Python
727
star
5

Matting-Anything

Matting Anything Model (MAM), an efficient and versatile framework for estimating the alpha matte of any instance in an image with flexible and interactive visual or linguistic user prompt guidance.
Python
607
star
6

Compact-Transformers

Escaping the Big Data Paradigm with Compact Transformers, 2021 (Train your Vision Transformers in 30 mins on CIFAR-10 with a single GPU!)
Python
492
star
7

Cross-Scale-Non-Local-Attention

PyTorch code for our paper "Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining" (CVPR2020).
Python
401
star
8

Pyramid-Attention-Networks

[IJCV] Pyramid Attention Networks for Image Restoration: new SOTA results on multiple image restoration tasks: denoising, demosaicing, compression artifact reduction, super-resolution
Python
382
star
9

NATTEN

Neighborhood Attention Extension. Bringing attention to a neighborhood near you!
Cuda
333
star
10

Smooth-Diffusion

Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models arXiv 2023 / CVPR 2024
Python
305
star
11

VCoder

VCoder: Versatile Vision Encoders for Multimodal Large Language Models, arXiv 2023 / CVPR 2024
Python
259
star
12

Rethinking-Text-Segmentation

[CVPR 2021] Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach
Python
241
star
13

Agriculture-Vision

[CVPR 2020 & 2021 & 2022 & 2023] Agriculture-Vision Dataset, Prize Challenge and Workshop: A joint effort with many great collaborators to bring Agriculture and Computer Vision/AI communities together to benefit humanity!
199
star
14

Self-Similarity-Grouping

Self-similarity Grouping: A Simple Unsupervised Cross Domain Adaptation Approach for Person Re-identification (ICCV 2019, Oral)
Python
188
star
15

Decoupled-Classification-Refinement

Revisiting RCNN: On Awakening the Classification Power of Faster RCNN (ECCV 2018)
Python
167
star
16

Convolutional-MLPs

[Preprint] ConvMLP: Hierarchical Convolutional MLPs for Vision, 2021
Python
163
star
17

3D-Point-Cloud-Learning

131
star
18

CuMo

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
Python
130
star
19

Forget-Me-Not

Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models, 2023
Python
107
star
20

VMFormer

[Preprint] VMFormer: End-to-End Video Matting with Transformer
Python
106
star
21

Semi-Supervised-Transfer-Learning

[CVPR 2021] Adaptive Consistency Regularization for Semi-Supervised Transfer Learning
Jupyter Notebook
101
star
22

SGL-Retinal-Vessel-Segmentation

[MICCAI 2021] Study Group Learning: Improving Retinal Vessel Segmentation Trained with Noisy Labels: New SOTA on both DRIVE and CHASE_DB1.
Jupyter Notebook
101
star
23

StyleNAT

New flexible and efficient image generation framework that sets new SOTA on FFHQ-256 with FID 2.05, 2022
Python
97
star
24

Unsupervised-Domain-Adaptation-with-Differential-Treatment

[CVPR 2020] Differential Treatment for Stuff and Things: A Simple Unsupervised Domain Adaptation Method for Semantic Segmentation
Python
88
star
25

Text2Video-Zero-sd-webui

Python
79
star
26

GFR-DSOD

Improving Object Detection from Scratch via Gated Feature Reuse (BMVC 2019)
Python
65
star
27

SH-GAN

[WACV 2023] Image Completion with Heterogeneously Filtered Spectral Hints
Python
62
star
28

VIM

Python
54
star
29

UltraSR-Arbitrary-Scale-Super-Resolution

[Preprint] UltraSR: Spatial Encoding is a Missing Key for Implicit Image Function-based Arbitrary-Scale Super-Resolution, 2021
53
star
30

Any-Precision-DNNs

Any-Precision Deep Neural Networks (AAAI 2021)
Python
44
star
31

Horizontal-Pyramid-Matching

Horizontal Pyramid Matching for Person Re-identification (AAAI 2019)
Python
39
star
32

Pseudo-IoU-for-Anchor-Free-Object-Detection

Pseudo-IoU: Improving Label Assignment in Anchor-Free Object Detection
Python
30
star
33

Human-Object-Interaction-Detection

25
star
34

CompFeat-for-Video-Instance-Segmentation

CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation (AAAI 2021)
19
star
35

Diffusion-Driven-Test-Time-Adaptation-via-Synthetic-Domain-Alignment

Everything to the Synthetic: Diffusion-driven Test-time Adaptation via Synthetic-Domain Alignment
Python
17
star
36

OneFormer-Colab

[Colab Demo Code] OneFormer: One Transformer to Rule Universal Image Segmentation.
Python
13
star
37

DiSparse-Multitask-Model-Compression

[CVPR 2022] DiSparse: Disentangled Sparsification for Multitask Model Compression
Jupyter Notebook
13
star
38

Interpretable-Visual-Reasoning

[ICCV 2021] Interpretable Visual Reasoning via Induced Symbolic Space
9
star
39

Mask-Selection-Networks

[CVPR 2021] Youtube-VIS 2021 3rd place, [CVPR 2020] winner DAVIS 2020. Code for mask selection based methods.
6
star
40

Activity-Recognition

5
star
41

Boosted-Dynamic-Networks

Boosted Dynamic Neural Networks, AAAI 2023
Python
4
star
42

Aneurysm-Segmentation-with-Multi-Teacher-Pseudo-Labels

1
star