• Stars
    star
    1,717
  • Rank 26,036 (Top 0.6 %)
  • Language
    Rust
  • License
    Apache License 2.0
  • Created over 4 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

🎨 Example-based texture synthesis written in Rust πŸ¦€

🎨 texture-synthesis

Embark Embark Crates.io Docs dependency status Build Status

A light Rust API for Multiresolution Stochastic Texture Synthesis [1], a non-parametric example-based algorithm for image generation.

The repo also includes multiple code examples to get you started (along with test images), and you can find a compiled binary with a command line interface under the release tab.

Also see our talk More Like This, Please! Texture Synthesis and Remixing from a Single Example which explains this technique and the background more in-depth:

Video thumbnail

Maintenance note

We at Embark are not actively using or developing these crates and would be open to transferring them to a maintainer or maintainers that would be more active. See #166.

Features and examples

1. Single example generation

Imgur

Generate similar-looking images from a single example.

use texture_synthesis as ts;

fn main() -> Result<(), ts::Error> {
    //create a new session
    let texsynth = ts::Session::builder()
        //load a single example image
        .add_example(&"imgs/1.jpg")
        .build()?;

    //generate an image
    let generated = texsynth.run(None);

    //save the image to the disk
    generated.save("out/01.jpg")
}

CLI

cargo run --release -- --out out/01.jpg generate imgs/1.jpg

You should get the following result with the images provided in this repo:

2. Multi example generation

Imgur

We can also provide multiple example images and the algorithm will "remix" them into a new image.

use texture_synthesis as ts;

fn main() -> Result<(), ts::Error> {
    // create a new session
    let texsynth = ts::Session::builder()
        // load multiple example image
        .add_examples(&[
            &"imgs/multiexample/1.jpg",
            &"imgs/multiexample/2.jpg",
            &"imgs/multiexample/3.jpg",
            &"imgs/multiexample/4.jpg",
        ])
        // we can ensure all of them come with same size
        // that is however optional, the generator doesnt care whether all images are same sizes
        // however, if you have guides or other additional maps, those have to be same size(s) as corresponding example(s)
        .resize_input(ts::Dims {
            width: 300,
            height: 300,
        })
        // randomly initialize first 10 pixels
        .random_init(10)
        .seed(211)
        .build()?;

    // generate an image
    let generated = texsynth.run(None);

    // save the image to the disk
    generated.save("out/02.jpg")?;

    //save debug information to see "remixing" borders of different examples in map_id.jpg
    //different colors represent information coming from different maps
    generated.save_debug("out/")
}

CLI

cargo run --release -- --rand-init 10 --seed 211 --in-size 300x300 -o out/02.png --debug-out-dir out generate imgs/multiexample/1.jpg imgs/multiexample/2.jpg imgs/multiexample/3.jpg imgs/multiexample/4.jpg

You should get the following result with the images provided in this repo:

3. Guided Synthesis

Imgur

We can also guide the generation by providing a transformation "FROM"-"TO" in a form of guide maps

use texture_synthesis as ts;

fn main() -> Result<(), ts::Error> {
    let texsynth = ts::Session::builder()
        // NOTE: it is important that example(s) and their corresponding guides have same size(s)
        // you can ensure that by overwriting the input images sizes with .resize_input()
        .add_example(ts::Example::builder(&"imgs/2.jpg").with_guide(&"imgs/masks/2_example.jpg"))
        // load target "heart" shape that we would like the generated image to look like
        // now the generator will take our target guide into account during synthesis
        .load_target_guide(&"imgs/masks/2_target.jpg")
        .build()?;

    let generated = texsynth.run(None);

    // save the image to the disk
    generated.save("out/03.jpg")
}

CLI

cargo run --release -- -o out/03.png generate --target-guide imgs/masks/2_target.jpg --guides imgs/masks/2_example.jpg -- imgs/2.jpg

NOTE: Note the use of -- to delimit the path to the example imgs/2.jpg, if you don't specify --, the path to the example will be used as another guide path and there won't be any examples.

You should get the following result with the images provided in this repo:

4. Style Transfer

Imgur

Texture synthesis API supports auto-generation of example guide maps, which produces a style transfer-like effect.

use texture_synthesis as ts;

fn main() -> Result<(), ts::Error> {
    let texsynth = ts::Session::builder()
        // load example which will serve as our style, note you can have more than 1!
        .add_examples(&[&"imgs/multiexample/4.jpg"])
        // load target which will be the content
        // with style transfer, we do not need to provide example guides
        // they will be auto-generated if none were provided
        .load_target_guide(&"imgs/tom.jpg")
        .guide_alpha(0.8)
        .build()?;

    // generate an image that applies 'style' to "tom.jpg"
    let generated = texsynth.run(None);

    // save the result to the disk
    generated.save("out/04.jpg")
}

CLI

cargo run --release -- --alpha 0.8 -o out/04.png transfer-style --style imgs/multiexample/4.jpg --guide imgs/tom.jpg

You should get the following result with the images provided in this repo:

5. Inpaint

Imgur

We can also fill-in missing information with inpaint. By changing the seed, we will get different version of the 'fillment'.

use texture_synthesis as ts;

fn main() -> Result<(), ts::Error> {
    let texsynth = ts::Session::builder()
        // let the generator know which part we would like to fill in
        // if we had more examples, they would be additional information
        // the generator could use to inpaint
        .inpaint_example(
            &"imgs/masks/3_inpaint.jpg",
            // load a "corrupted" example with missing red information we would like to fill in
            ts::Example::builder(&"imgs/3.jpg")
                // we would also like to prevent sampling from "corrupted" red areas
                // otherwise, generator will treat that those as valid areas it can copy from in the example,
                // we could also use SampleMethod::Ignore to ignore the example altogether, but we
                // would then need at least 1 other example image to actually source from
                // example.set_sample_method(ts::SampleMethod::Ignore);
                .set_sample_method(&"imgs/masks/3_inpaint.jpg"),
            // Inpaint requires that inputs and outputs be the same size, so it's a required
            // parameter that overrides both `resize_input` and `output_size`
            ts::Dims::square(400),
        )
        // Ignored
        .resize_input(ts::Dims::square(200))
        // Ignored
        .output_size(ts::Dims::square(100))
        .build()?;

    let generated = texsynth.run(None);

    //save the result to the disk
    generated.save("out/05.jpg")
}

CLI

Note that the --out-size parameter determines the size for all inputs and outputs when using inpaint!

cargo run --release -- --out-size 400 --inpaint imgs/masks/3_inpaint.jpg -o out/05.png generate imgs/3.jpg

You should get the following result with the images provided in this repo:

6. Inpaint Channel

bricks

Instead of using a separate image for our inpaint mask, we can instead obtain the information from a specific channel. In this example, the alpha channel is a circle directly in the middle of the image.

use texture_synthesis as ts;

fn main() -> Result<(), ts::Error> {
    let texsynth = ts::Session::builder()
        // Let the generator know that it is using 
        .inpaint_example_channel(
            ts::ChannelMask::A,
            &"imgs/bricks.png",
            ts::Dims::square(400),
        )
        .build()?;

    let generated = texsynth.run(None);

    //save the result to the disk
    generated.save("out/06.jpg")
}

CLI

cargo run --release -- --inpaint-channel a -o out/06.png generate imgs/bricks.jpg

You should get the following result with the images provided in this repo:

7. Tiling texture

We can make the generated image tile (meaning it will not have seams if you put multiple images together side-by-side). By invoking inpaint mode together with tiling, we can make an existing image tile.

use texture_synthesis as ts;

fn main() -> Result<(), ts::Error> {
    // Let's start layering some of the "verbs" of texture synthesis
    // if we just run tiling_mode(true) we will generate a completely new image from scratch (try it!)
    // but what if we want to tile an existing image?
    // we can use inpaint!

    let texsynth = ts::Session::builder()
        // load a mask that specifies borders of the image we can modify to make it tiling
        .inpaint_example(
            &"imgs/masks/1_tile.jpg",
            ts::Example::new(&"imgs/1.jpg"),
            ts::Dims::square(400),
        )
        //turn on tiling mode!
        .tiling_mode(true)
        .build()?;

    let generated = texsynth.run(None);

    generated.save("out/07.jpg")
}

CLI

cargo run --release -- --inpaint imgs/masks/1_tile.jpg --out-size 400 --tiling -o out/07.bmp generate imgs/1.jpg

You should get the following result with the images provided in this repo:

8. Repeat texture synthesis transform on a new image

We can re-apply the coordinate transformation performed by texture synthesis onto a new image.

use texture_synthesis as ts;

fn main() -> Result<(), ts::Error> {
    // create a new session
    let texsynth = ts::Session::builder()
        //load a single example image
        .add_example(&"imgs/1.jpg")
        .build()?;

    // generate an image
    let generated = texsynth.run(None);

    // now we can apply the same transformation of the generated image
    // onto a new image (which can be used to ensure 1-1 mapping between multiple images)
    // NOTE: it is important to provide same number of input images as the 
    // otherwise, there will be coordinates mismatch
    let repeat_transform_img = generated
        .get_coordinate_transform()
        .apply(&["imgs/1_bw.jpg"])?;

    // save the image to the disk
    // 08 and 08_repeated images should match perfectly
    repeat_transform_img.save("out/08_repeated.jpg").unwrap();
    generated.save("out/08.jpg")
}

CLI

  1. First, we need to create a transform that can be reused

The notable bit here is the --save-transform out/multi.xform which creates the file that can be used to generate new outputs with.

cargo run --release -- --rand-init 10 --seed 211 --in-size 300x300 -o out/02.png generate --save-transform out/multi.xform imgs/multiexample/1.jpg imgs/multiexample/2.jpg imgs/multiexample/3.jpg imgs/multiexample/4.jpg

  1. Next, we use the repeat subcommand to repeat transform with different inputs

The important bits here are the use of the repeat subcommand instead of generate, and --transform out/multi.xform which tells what transform to apply to the inputs. The only restriction is that the number of images you specify must match the original number of examples exactly. If the input images have different dimensions than the example images, they will be automatically resized for you.

cargo run --release -- -o out/02-repeated.png repeat --transform out/multi.xform imgs/multiexample/1.jpg imgs/multiexample/2.jpg imgs/multiexample/4.jpg imgs/multiexample/3.jpg

Also note that the normal parameters that are used with generate don't apply to the repeat subcommand and will be ignored.

9. Sample masks

Sample masks allow you to specify how an example image is sampled during generation.

use texture_synthesis as ts;

fn main() -> Result<(), ts::Error> {
    let session = ts::Session::builder()
        .add_example(
            ts::Example::builder(&"imgs/4.png").set_sample_method(ts::SampleMethod::Ignore),
        )
        .add_example(ts::Example::builder(&"imgs/5.png").set_sample_method(ts::SampleMethod::All))
        .seed(211)
        .output_size(ts::Dims::square(200))
        .build()?;

    // generate an image
    let generated = session.run(None);

    // save the image to the disk
    generated.save("out/09.png")
}

CLI

cargo run --release -- --seed 211 --out-size 200 --sample-masks IGNORE ALL --out 09_sample_masks.png generate imgs/4.png imgs/5.png

You should get the following result with the images provided in this repo:

10. Combining texture synthesis 'verbs'

We can also combine multiple modes together. For example, multi-example guided synthesis:

Or chaining multiple stages of generation together:

For more use cases and examples, please refer to the presentation "More Like This, Please! Texture Synthesis and Remixing from a Single Example"

Additional CLI functionality

Some functionality is only exposed through the CLI and not built into the library.

flip-and-rotate

This subcommand takes each example and performs flip and rotation transformations to it to generate additional example inputs for generation. This subcommand doesn't support target or example guides.

Example: cargo run --release -- -o out/output.png flip-and-rotate imgs/1.jpg

Command line binary

  • Download the binary for your OS.
  • Or Install it from source.
    • Install Rust - The minimum required version is 1.37.0
    • Clone this repo
    • In a terminal cd to the directory you cloned this repository into
    • Run cargo install --path=cli
    • Or if you wish to see the texture as it is being synthesized cargo install --path=cli --features="progress"
  • Open a terminal
  • Navigate to the directory where you downloaded the binary, if you didn't just cargo install it
  • Run texture_synthesis --help to get a list of all of the options and commands you can run
  • Refer to the examples section in this readme for examples of running the binary

Notes

  • By default, generating output will use all of your logical cores
  • When using multiple threads for generation, the output image is not guaranteed to be deterministic with the same inputs. To have 100% determinism, you must use a thread count of one, which can by done via
    • CLI - texture-synthesis --threads 1
    • API - SessionBuilder::max_thread_count(1)

Limitations

  • Struggles with complex semantics beyond pixel color (unless you guide it)
  • Not great with regular textures (seams can become obvious)
  • Cannot infer new information from existing information (only operates on what’s already there)
  • Designed for single exemplars or very small datasets (unlike Deep Learning based approaches)

Additional Dependencies

If you're compiling for Linux, you'll need to have libxkbcommon development libraries installed. For ubuntu this is libxkbcommon-x11-dev.

Links/references

[1] [Opara & Stachowiak] "More Like This, Please! Texture Synthesis and Remixing from a Single Example"

[2] [Harrison] Image Texture Tools

[3] [Ashikhmin] Synthesizing Natural Textures

[4] [Efros & Leung] Texture Synthesis by Non-parametric Sampling

[5] [Wey & Levoy] Fast Texture Synthesis using Tree-structured Vector Quantization

[6] [De Bonet] Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images

[7] All the test images in this repo are from Unsplash

Contributing

Contributor Covenant

We welcome community contributions to this project.

Please read our Contributor Guide for more information on how to get started.

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

More Repositories

1

rust-gpu

πŸ‰ Making Rust a first-class language and ecosystem for GPU shaders 🚧
Rust
6,796
star
2

kajiya

πŸ’‘ Experimental real-time global illumination renderer πŸ¦€
Rust
4,524
star
3

wg-ui

WireGuard Web UI for self-serve client configurations, with optional auth.
Go
1,468
star
4

cargo-deny

❌ Cargo plugin for linting your dependencies πŸ¦€
Rust
1,254
star
5

puffin

🐦 Friendly little instrumentation profiler for Rust πŸ¦€
Rust
973
star
6

rust-ecosystem

Rust wants & tracking for Embark πŸ¦€
Rust
843
star
7

physx-rs

🎳 Rust binding for NVIDIA PhysX πŸ¦€
Rust
611
star
8

blender-tools

🐡 Embark Addon for Blender
Python
392
star
9

cargo-about

πŸ“œ Cargo plugin to generate list of all licenses for a crate πŸ¦€
Rust
320
star
10

tryhard

πŸ’« Easily retry futures πŸ¦€
Rust
182
star
11

presser

A crate to help you copy things into raw buffers without invoking spooky action at a distance (undefined behavior).
Rust
150
star
12

rpmalloc-rs

🐏 rpmalloc global memory allocator for Rust πŸ¦€
Rust
126
star
13

crash-handling

Collection of crates to deal with crashes
Rust
122
star
14

poll-promise

A Rust promise for games and immediate mode GUIs
Rust
119
star
15

discord-sdk

An open implementation of the Discord Game SDK in Rust
Rust
108
star
16

skyhook

Simple Python communication system for DCC's and Game Engines
Python
104
star
17

spirt

SPIR-πŸ‡Ή: shader-focused IR to target, transform and translate from πŸ¦€
Rust
101
star
18

superluminal-perf-rs

πŸ”† Superluminal Performance profiler Rust API πŸ¦€
Rust
94
star
19

ash-molten

πŸŒ‹ Statically linked MoltenVK for Vulkan on Mac using Ash πŸ¦€
Rust
91
star
20

cargo-deny-action

❌ GitHub Action for cargo-deny πŸ¦€
Shell
89
star
21

cargo-fetcher

🎁 cargo plugin for quickly fetching dependencies πŸ¦€
Rust
83
star
22

relnotes

Automatic GitHub Release Notes
Rust
77
star
23

opensource-template

🌻 Template for creating new repositories
74
star
24

tame-oauth

πŸ” Small OAuth crate that follows the sans-io approach πŸ¦€
Rust
68
star
25

mirror-mirror

πŸͺž Powerful reflection library for Rust
Rust
68
star
26

tiny-bench

A tiny benchmarking library
Rust
55
star
27

k8s-buildkite-plugin

Run any buildkite build step as a Kubernetes Job
Jsonnet
48
star
28

krates

πŸ“¦ Creates graphs of crates from cargo metadata πŸ¦€
Rust
45
star
29

opensource

Open source processes, policies, and info
Rust
42
star
30

tame-gcs

πŸ“‚ A small library with a limited set of Google Cloud Storage operations πŸ¦€
Rust
37
star
31

cervo

Utility wrappers for tract
Rust
36
star
32

server-framework

Framework for running network services, opinions included
Rust
36
star
33

tracing-logfmt

A logfmt formatter for tracing subscriber logs
Rust
35
star
34

octobors

Rust program for automerging PRs based on a few rules
Rust
35
star
35

nfd2

OS native dialogs for Windows, MacOS, and Linux
Rust
32
star
36

cfg-expr

A parser and evaluator for Rust cfg() expressions. Targets as of Rust 1.58.0 are supported.
Rust
32
star
37

opa-policies

Contains OPA Policies for Dockerfiles, Kubernetes YAMLs, Terraform, etc
Open Policy Agent
29
star
38

gsutil

Minimal gsutil replacement
Rust
27
star
39

spdx

πŸ†” Helper crate for SPDX expressions. πŸ¦€
Rust
27
star
40

pdm-plugin-torch

A tool for managing torch-variants with PDM.
Python
22
star
41

opensource-website

🌐 Hub for Embark's open source efforts
HTML
22
star
42

buildkite-jobify

πŸ‘· Kubekite, but in Rust, using configuration from your repos πŸ¦€
Rust
21
star
43

spirv-tools-rs

πŸ›  Wrapper crate for SPIRV-Tools πŸ¦€
C++
20
star
44

tame-index

Small crate for interacting with cargo registry indices
Rust
18
star
45

emote

Reinforcement learning library from Embark Studios
Python
17
star
46

rymder

Unofficial agones client SDK
Rust
17
star
47

fsr-rs

Rust bindings for AMD FidelityFXβ„’ Super Resolution
C
16
star
48

proto-gen

A protobuf generation runner/cli using tonic build
Rust
15
star
49

sentry-contrib-rust

Integrates crashdump reporting with Sentry
Rust
15
star
50

tame-oidc

🧬 Small OAuth crate that follows the sans-io approach πŸ¦€
Rust
11
star
51

cloud-dns

A wrapper for the Google Cloud DNS API (https://cloud.google.com/dns)
Rust
10
star
52

tame-webpurify

πŸ’© Small HTTP client for the Webpurify API following the sans-io approach πŸ¦€
Rust
9
star
53

tracing-ext-ffi-subscriber

Simple subscriber for forwarding tracing spans to a C or C++ profiling API.
Rust
4
star
54

server-code-exciser

A program that contains grammars and functionality to remove server only code from code bases.
C#
4
star
55

boh

Rust
4
star
56

gke-accelerated-xorg-example

Example project for running remote rendering on GKE
Dockerfile
3
star
57

helix-oidc

🧬 Helix Perforce OIDC validator
Go
3
star
58

cassini

Topology-aware distributed cache
Go
2
star
59

minwin

Rust
2
star
60

container-packer-qemu

πŸ“¦ Dockerized packer with qemu
Dockerfile
1
star
61

rustc-compile-time-regress

Rust
1
star
62

.github

Default community health files for GitHub (https://help.github.com/en/github/building-a-strong-community/creating-a-default-community-health-file-for-your-organization)
1
star
63

missing-symbols

Rust
1
star
64

prefabricator-ue4

Prefabricator - Prefab plugin for Unreal Engine 4 https://prefabricator.io
C++
1
star