Awesome-Sketch-Synthesis

A collection of papers about Sketch Synthesis (Generation). Mainly focus on stroke-level vector sketch synthesis.

Feel free to create a PR or an issue.

Outlines

0. Survey
1. Datasets
2. Sketch-Synthesis Approaches
3. Vector Graphics Generation

0. Survey

Paper	Source	Code/Project Link
Deep Learning for Free-Hand Sketch: A Survey	TPAMI 2022	[code]

1. Datasets

Here Vector strokes means having svg data. With photos means having the photo-sketch paired data.

Level	Dataset	Source	Vector strokes	With photos	Notes
Characters	Omniglot		✔️	❌	Alphabets characters
	KanjiVG		✔️	❌	Chinese characters
	Kuzushiji		❌	❌	Japanese characters
Instance-level	TU-Berlin	SIGGRAPH 2012	✔️	❌	Multi-category hand sketches
	Sketchy	SIGGRAPH 2016	✔️	✔️	Multi-category photo-sketch paired
	QuickDraw	ICLR 2018	✔️	❌	Multi-category hand sketches
	QMUL-Shoe-Chair-V2	CVPR 2016	✔️	✔️	Only two categories
	Sketch Perceptual Grouping (SPG)	ECCV 2018	✔️	❌	With part-level semantic segmentation information
	FaceX	AAAI 2019	✔️	❌	Labeled facial sketches
	Creative Sketch	ICLR 2021	✔️	❌	With annotated part segmentation
	ImageNet-Sketch	NeurIPS 2019	❌	❌	50 images for each of the 1000 ImageNet classes
Scene-level	SketchyScene	ECCV 2018	❌	✔️	With semantic/instance segmentation information
	CMPlaces	TPAMI 2018	❌	✔️	Cross-modal scene dataset
	Context-Skecth	Expressive 2018	❌	✔️	Context-based scene sketches for co-classification
	SketchyCOCO	CVPR 2020	❌	✔️	Scene sketch, segmentation and normal images
	FS-COCO	ECCV 2022	✔️	✔️	Scene sketches with text description
	SFSD	VC 2022	✔️	✔️	Completely hand-drawn scene sketches with label annotation
Drawing from photos	Photo-Sketching	WACV 2019	✔️	✔️	ScenePhoto-sketch paired
Drawing from photos	Tracing-vs-Freehand	SIGGRAPH 2021	✔️	✔️	Tracings and freehand drawings of images
Drawing from 3D models	DifferSketching	SIGGRAPH Asia 2022	✔️	❌	3D model-sketch paired, with novice and professional ones
Portrait	APDrawing	CVPR 2019	❌	✔️	Portrait-sketch paired
Anime portrait	Danbooru-Parsing	TOG 2023	❌	✔️	For anime portrait parsing and anime translation
Children's Drawing	Amateur Drawings	TOG 2023	❌	✔️	With character bounding boxes, segmentation masks, and joint location annotations
Rough sketch	Da Vinci	CGI 2018	❌	✔️	Line drawing restoration dataset
Rough sketch	Rough Sketch Benchmark	SIGGRAPH Asia 2020	✔️	✔️	Rough and clean sketch pairs (only for evaluation)
CAD	ld3d	SIGGRAPH 2008	❌	❌	Line Drawings of 3D Shapes
	OpenSketch	SIGGRAPH Asia 2019	✔️	❌	Product Design Sketches
	SketchGraphs	ICML 2020 Workshop	✔️	❌	Sketches extracted from real-world CAD models
	Fusion 360 Gallery	SIGGRAPH 2021	✔️	❌	For 'sketch and extrude' designs
	FloorPlanCAD	ICCV 2021	✔️	❌	With instance and semantic annotations

2. Sketch-Synthesis Approaches

1) Semantic Concept-to-sketch

Level	Paper	Source	Code/Project Link
Instance-level	A Neural Representation of Sketch Drawings (sketch-rnn)	ICLR 2018	[Code] [Project] [Demo]
	Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories		[Code]
	AI-Sketcher : A Deep Generative Model for Producing High-Quality Sketches	AAAI 2019	[Project]
	Stroke-based sketched symbol reconstruction and segmentation (stroke-rnn)	CGA 2019
	BézierSketch: A generative model for scalable vector sketches	ECCV 2020	[Code]
	Pixelor: A Competitive Sketching AI Agent. So you think you can beat me?	SIGGRAPH Asia 2020	[Project] [Code]
	Creative Sketch Generation	ICLR 2021	[Project] [Code]
	Computer-Aided Design as Language	arxiv 2105
	DoodleFormer: Creative Sketch Drawing with Transformers	ECCV 2022	[Project] [Code]
	SketchKnitter: Vectorized Sketch Generation with Diffusion Models	ICLR 2023	[Code]
	Self-Organizing a Latent Hierarchy of Sketch Patterns for Controllable Sketch Synthesis	TNNLS 2023	[Code]

2) Photo-to-sketch

vector image generation

Data type	Paper	Source	Code/Project Link
Facial	Style and abstraction in portrait sketching	TOG 2013
Instance-level	Free-Hand Sketch Synthesis with Deformable Stroke Models	IJCV 2017	[Project] [code]
	Learning to Sketch with Shortcut Cycle Consistency	CVPR 2018	[Code1] [Code2]
	Learning Deep Sketch Abstraction	CVPR 2018
	CLIPasso: Semantically-Aware Object Sketching	SIGGRAPH 2022	[Project] [Code]
Scene-level	CLIPascene: Scene Sketching with Different Types and Levels of Abstraction	arxiv 22.11	[Project]
Scene-level	Sketch Generation with Drawing Process Guided by Vector Flow and Grayscale	AAAI 2021	[Code]
Technical Drawings	Deep Vectorization of Technical Drawings	ECCV 2020	[Project] [code]

raster image generation

Type	Paper	Source	Code/Project Link
Facial	ArtLine	Online demo	[Code]
	APDrawingGAN: Generating Artistic Portrait Drawings from Face Photos with Hierarchical GANs	CVPR 2019	[Code] [Demo]
	Unpaired Portrait Drawing Generation via Asymmetric Cycle Mapping	CVPR 2020	[Code]
	Line Drawings for Face Portraits From Photos Using Global and Local Structure Based GANs	TPAMI 2020	[Code]
	Quality Metric Guided Portrait Line Drawing Generation from Unpaired Training Data	TPAMI 2022	[Code]
Instance-level	Deep Factorised Inverse-Sketching	ECCV 2018
Instance-level	Synthesizing human-like sketches from natural images using a conditional convolutional decoder	WACV 2020	[Code]
Anime	sketchKeras	online demo	[Code]
	LineDistiller	online demo	[Code]
	Anime2Sketch	online demo	[Code]
	Reference Based Sketch Extraction via Attention Mechanism	SIGGRAPH Asia 2022	[Code]
Scene-level	Photo-Sketching: Inferring Contour Drawings from Images	WACV 2019	[Code] [Project]
Scene-level	Learning to generate line drawings that convey geometry and semantics	CVPR 2022	[Code] [Project]
Arbitrary	Semi-supervised reference-based sketch extraction using a contrastive learning	SIGGRAPH 2023	[Code] [Project]

3) Text/Attribute-to-sketch

raster image generation

Type	Paper	Source
Facial	Text2Sketch: Learning Face Sketch from Facial Attribute Text	ICIP 2018
Scene-level	Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications	UIST 2019
Scene-level	Scones: Towards Conversational Authoring of Sketches	IUI 2020

vector image generation

Type	Paper	Source	Code/Project Link
Arbitrary	CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders	arxiv 21.06	[code]
Arbitrary	Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts	arxiv 21.09	[code] [project]
Arbitrary	VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models	CVPR 2023	[project]
Icon	IconShop: Text-Based Vector Icon Synthesis with Autoregressive Transformers	arxiv 23.04	[project] [code]
Arbitrary	DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models	arxiv 23.06	[project] [code]

4) 3D shape-to-sketch

Paper	Source	Code/Project Link
DeepShapeSketch : Generating hand drawing sketches from 3D objects	IJCNN 2019
Neural Contours: Learning to Draw Lines from 3D Shapes	CVPR 2020	[project] [code]
Cloud2Curve: Generation and Vectorization of Parametric Sketches	CVPR 2021	[project]
Neural Strokes: Stylized Line Drawing of 3D Shapes	ICCV 2021	[code]
Learning a Style Space for Interactive Line Drawing Synthesis from Animated 3D Models	PG 2022
CAD2Sketch: Generating Concept Sketches from CAD Sequences	SIGGRAPH Asia 2022	[project]

5) Art-to-sketch

Here we list sketch synthesis based on other image types, like Manga, line art, rough sketch, etc.

a) Line art

Vector-to-Vector

Paper	Source	Code/Project Link	*-based
Closure-aware Sketch Simplification	SIGGRAPH Asia 2015	[Project]	Optimization
StrokeAggregator: Consolidating Raw Sketches into Artist-Intended Curve Drawings	SIGGRAPH 2018	[Project]	Optimization
StrokeStrip: Joint Parameterization and Fitting of Stroke Clusters	SIGGRAPH 2021	[Project] [code]	Optimization

Raster-to-Vector (a.k.a. Vectorization)

Paper	Source	Code/Project Link	*-based
Topology-Driven Vectorization of Clean Line Drawings	TOG 2013		Optimization
Fidelity vs. Simplicity: a Global Approach to Line Drawing Vectorization	SIGGRAPH 2016	[Project]	Optimization
A Delaunay triangulation based approach for cleaning rough sketches	C&G 2018	[Code]	Optimization
Semantic Segmentation for Line Drawing Vectorization Using Neural Networks	EG 2018	[project] [code]	Learning
Deep Line Drawing Vectorization via Line Subdivision and Topology Reconstruction	PG 2019		Learning
Inertia-based Fast Vectorization of Line Drawings	PG 2019		Optimization
Vectorization of Line Drawings via Polyvector Fields	TOG 2019	[Code]	Optimization
Integer-Grid Sketch Simplification and Vectorization	SGP 2020	[Project] [Code]	Optimization
Deep Vectorization of Technical Drawings	ECCV 2020	[project] [code]	Learning
General Virtual Sketching Framework for Vector Line Art	SIGGRAPH 2021	[project] [code]	Learning
Keypoint-Driven Line Drawing Vectorization via PolyVector Flow	SIGGRAPH Asia 2021	[project]	Learning + Optimization
End-to-end Line Drawing Vectorization	AAAI 2022		Learning
Vectorizing Line Drawings of Arbitrary Thickness via Boundary-based Topology Reconstruction	CGF 2022		Heuristic

b) Rough sketch simplification / cleanup

Datasets and benchmark

Paper	Source	Code/Project Link
A Benchmark for Rough Sketch Cleanup	SIGGRAPH Asia 2020	[Project] [Code]

Methods

Paper	Source	Code/Project Link
Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup	SIGGRAPH 2016	[Code] [Project]
Mastering Sketching: Adversarial Augmentation for Structured Prediction	SIGGRAPH 2018	[Code] [Project]
Real-Time Data-Driven Interactive Rough Sketch Inking	SIGGRAPH 2018	[Code] [Project]
Perceptual-aware Sketch Simplification Based on Integrated VGG Layers	TVCG 2019

c) Manga (Comics)

Paper	Source	Code/Project Link
Deep extraction of manga structural lines	SIGGRAPH 2017	[Code]
Manga Filling Style Conversion with Screentone Variational Autoencoder	SIGGRAPH Asia 2020	[Project] [Code]
Generating Manga from Illustrations via Mimicking Manga Workflow	CVPR 2021	[Project] [Code]
MangaGAN: Unpaired Photo-to-Manga Translation Based on The Methodology of Manga Drawing	AAAI 2021
Vectorization of Raster Manga by Deep Reinforcement Learning	arxiv 2110

3. Vector Graphics Generation

Here we focus on learning-based vector graphics generation without depending on vector training data, and traditional vectorization algorithms.

Learning with external black-box (non-differentiable) rendering simulator

Paper	Source	Code/Project Link
Synthesizing Programs for Images using Reinforced Adversarial Learning	ICML 2018	[Code]
Unsupervised Doodling and Painting with Improved SPIRAL	arxiv 1910	[Project]

Learning with built-in differentiable rendering module

Paper	Source	Code/Project Link
Intelli-Paint: Towards Developing More Human-Intelligible Painting Agents	ECCV 2022	[project]
Towards Layer-wise Image Vectorization	CVPR 2022	[code] [project]
Paint Transformer: Feed Forward Neural Painting with Stroke Prediction	ICCV 2021	[code]
Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings	CVPR 2021	[code]
Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes	CVPR 2021	[code]
Im2Vec: Synthesizing Vector Graphics without Vector Supervision	CVPR 2021	[Project] [code]
Stylized Neural Painting	CVPR 2021	[Code] [project]
Learning to Paint With Model-based Deep Reinforcement Learning	ICCV 2019	[code]
Strokenet: A neural painting environment	ICLR 2019	[Code]
Neural Painters: A learned differentiable constraint for generating brushstroke paintings	arxiv 1904	[Code]
Learning to Sketch with Deep Q Networks and Demonstrated Strokes	arxiv 1810
Unsupervised Image to Sequence Translation with Canvas-Drawer Networks	arxiv 1809

Vectorization

Paper	Source	Code/Project Link
Depixelizing pixel art	SIGGRAPH 2011
Perception-Driven Semi-Structured Boundary Vectorization	SIGGRAPH 2018	[Webpage]
PolyFit: Perception-aligned Vectorization of Raster Clip-art via Intermediate Polygonal Fitting	SIGGRAPH 2020	[Webpage] [Code]
ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis	TVCG 2021
TCB-Spline-Based Image Vectorization	TOG 2022
Image vectorization and editing via linear gradient layer decomposition	SIGGRAPH 2023

MarkMoHR/Awesome-Sketch-Synthesis

MarkMoHR

Reviews

Repository Details