• Stars
    star
    135
  • Rank 269,297 (Top 6 %)
  • Language
    Python
  • License
    BSD 3-Clause "New...
  • Created over 3 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Differentiable Drawing and Sketching

Differentiable Drawing and Sketching

This repository contains the official implementation of the paper 'Differentiable Drawing and Sketching.'

ArXiv • Papers With Code • About • Usage • Experiments

About

This is a bottom-up relaxed differentiable relaxation of point/line/curve rasterisation and sketch generation implemented entirely as GPU-accelerated tensor operations in PyTorch. We relax the act of rasterising primitive objects such that there is a gradient with respect to the primitive's parameters from every pixel in the image. Rasterisations of primitives can then be composed in different ways (so long as they are differentiable) to achieve a resultant output which can be optimised.

We demonstrate that it is possible to perform gradient descent directly through the rasteriser to make the generated image match a target image according to some loss (which could be a simple function, or even a complex neural network -- the image above was created using the LPIPS Perceptual Loss with a VGG16 network to optimise 2000 straight line segments and their respective colours). We also demonstrate the use of the rasteriser in auto-encoder models that learn to both produce reconstructions whilst simultaneously learning the actual parameters of underlying vector primitives, and thus differentiably perform automatic image tracing (aka autotracing).

Usage

Installation

Clone the source, then from the source directory install the dependencies (pip install -r requirements.txt) and run python setup.py install to install the library and commandline tools.

Demonstration Notebooks

The samples/ directory contains a number of jupyter notebooks demonstrating how to implement different types of primitive parameterisations, as well as demonstrating direct optimisation of primitive parameters using gradient descent through the rasteriser.

Tools

We've provided some simple commandline tools to perform image optimisation and run experiments from the paper.

Image Optimisation

The imageopt tool lets you optimise an image against a set of points and line segments:

usage: imageopt [--width WIDTH] [--lines LINES] [--points POINTS] --loss
            {BCELoss,BlurredMSELoss,DoGPyrMSELoss,LPIPSLoss,MSELoss,PyrMSELoss}
            [--iters ITERS] [--init-sigma2 INIT_SIGMA2]
            [--final-sigma2 FINAL_SIGMA2] [--sigma2-factor SIGMA2_FACTOR]
            [--sigma2-step SIGMA2_STEP] [--seed SEED] [--lr LR]
            [--init-raster INIT_RASTER] [--init-pdf INIT_PDF]
            [--final-raster FINAL_RASTER] [--final-pdf FINAL_PDF]
            [--snapshots-path SNAPSHOTS_PATH]
            [--snapshots-steps SNAPSHOTS_STEPS] [--invert]
            [--optimiser OPTIMISER] [--device DEVICE] [--colour]
            image

For example, the image at the top of this page was created with:

imageopt --loss LPIPSLoss --net vgg --invert --seed 1234 --width 300 --lines 2000 --init-sigma2 1.0 --final-sigma2 1.0 --iters 500 --lr 0.01 --init-raster results/vancouver/init.png --final-raster results/vancouver/final.png --init-pdf results/vancouver/init.pdf --final-pdf results/vancouver/final.pdf --snapshots-path results/vancouver data/vancouver.jpg --snapshots-steps 100 --colour

Auto-encoder Experiments

The auto-encoder experiments in the paper can be reproduced with the autoencoder-experiment tool, which has a train mode for model training:

autoencoder-experiment train [-h] [--variational] --encoder
                                {BetterCNNEncoder,Conv2DEncoder,SimpleCNNEncoder,SimpleMLPEncoder,StrokeNetEncoder}
                                --decoder
                                {RNNBezierDecoder,RecurrentCRSDecoder,RecurrentDecoder,RecurrentPolyConnectDecoder,RecurrentPolyLineDecoder,RecurrentSimpleLineDecoder,SinglePassBezierConnectDecoder,SinglePassCRSDecoder,SinglePassPolyConnectDecoder,SinglePassPolyLineDecoder,SinglePassSimpleBezierDecoder,SinglePassSimpleLineDecoder}
                                --dataset
                                {C28pxOmniglot,MNIST,Omniglot,ScaledMNIST}
                                [--num-reconstructions NUM_RECONSTRUCTIONS]
                                --output OUTPUT [--device DEVICE] --loss
                                {BCELoss,BlurredMSELoss,DoGPyrMSELoss,LPIPSLoss,MSELoss,PyrMSELoss}
                                [--epochs EPOCHS]
                                [--learning-rate LEARNING_RATE]
                                [--weight-decay WEIGHT_DECAY]
                                [--snapshot-interval SNAPSHOT_INTERVAL]

and an eval mode for evaluating:

autoencoder-experiment eval [-h] [--variational] --encoder
                               {BetterCNNEncoder,Conv2DEncoder,SimpleCNNEncoder,SimpleMLPEncoder,StrokeNetEncoder}
                               --decoder
                               {RNNBezierDecoder,RecurrentCRSDecoder,RecurrentDecoder,RecurrentPolyConnectDecoder,RecurrentPolyLineDecoder,RecurrentSimpleLineDecoder,SinglePassBezierConnectDecoder,SinglePassCRSDecoder,SinglePassPolyConnectDecoder,SinglePassPolyLineDecoder,SinglePassSimpleBezierDecoder,SinglePassSimpleLineDecoder}
                               --dataset
                               {C28pxOmniglot,MNIST,Omniglot,ScaledMNIST}
                               [--num-reconstructions NUM_RECONSTRUCTIONS]
                               --output OUTPUT [--device DEVICE] --weights
                               WEIGHTS
                               [--classifier-weights CLASSIFIER_WEIGHTS]
                               [--classifier-model {LakeThesisCNN,MnistCNN,OmniglotCNN,ScaledMnistCNN}]

The evaluation mode requires a pretrained classifier for measuring performance. We've provided our MNIST and Scaled-MNIST classifiers in the expt-models folder. However, you can train your own classifiers with the provided train_classifiers tool.

Citation

If you find this repository useful for your research, please use the following.

@misc{mihai2021differentiable,
      title={Differentiable Drawing and Sketching}, 
      author={Daniela Mihai and Jonathon Hare},
      year={2021},
      eprint={2103.16194},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

More Repositories

1

COMP3204

COMP3204/COMP6223 Computer Vision (Part II Lecture Materials)
Java
28
star
2

COMP6237

COMP6237 "Data Mining" Lecture Material
Java
24
star
3

COMP6208

COMP6208 Advanced Machine Learning Lecture Material
Java
14
star
4

SimpleFaceDetector

A simple face detection tool for COMP6223
Java
12
star
5

DISCnetMachineLearningCourse

DISCnetMachineLearningCourse
Jupyter Notebook
10
star
6

os-deep-learning-labs

Python
8
star
7

LloydsRegistryMachineLearningCourse

Notes, demos and materials for learning Machine Learning
Python
7
star
8

COMP3005

COMP3005 Computer Vision Part II - Lecture Materials
Java
6
star
9

StreetviewExtractor

Extract streetview images along a route between two locations
Java
3
star
10

vlc-deep-learning-labs

Python
3
star
11

explore.vision

Code for explore.vision
PHP
3
star
12

Smallpeice-Lecture

Java
2
star
13

ecs-summer-school-vision-lecture

Interactive slides for the computer vision lecture at the ECS Summer School
Java
2
star
14

git-example

C
2
star
15

WEBS2002-1415-Alpha

Code resources for team A
Java
1
star
16

jmatio

A branch of the JMatio project by Craig Watcham
Java
1
star
17

WAISFest14

WAISFest 2014: Predicting the seasons
Java
1
star
18

learning2communicate

Slides from the OptML workshop
TeX
1
star
19

MakeItDigital-Vision-Talk

Interactive slides for my computer vision talk at the BBC Make It Digital Roadshow
Java
1
star
20

timelapse

Cross-platform command-line time-lapse capture from a webcam
Java
1
star
21

COMP3013-MIAR

Interactive Slides for COMP3013
Java
1
star
22

WAISFest13

WAISFest13 Flickr Visualisation
Java
1
star
23

js-mr-demo

Simple node server for demonstrating distributed data processing
JavaScript
1
star
24

DifferentiableProgrammingIntroLecture

TeX
1
star
25

OnAir

OnAir: An IoT live-status device
C++
1
star
26

jsworksheets

Interactive Javascript Worksheets
JavaScript
1
star
27

ecs-taster-vision-lecture

ECS Taster Vision Lecture
Java
1
star
28

archai-sprint

Python
1
star
29

cs-outreach

Computer Science Outreach Stuff
Python
1
star
30

WAIS-ML101

Material for the 2016 WAIS Away Day Machine Learning 101 tutorial
Java
1
star
31

WhiteboardSerialDuo

Serial communication driver for 2-stepper plotbot
Arduino
1
star
32

NGCM_MachineLearningCourse

Notes for the NGCM Summer Academy "Machine Learning" course
Jupyter Notebook
1
star