• Stars
    star
    367
  • Rank 116,257 (Top 3 %)
  • Language
    Rust
  • Created about 6 years ago
  • Updated almost 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Asset pipeline system for game engines & editor suites.

Rust

Distill

Distill is an asset pipeline for games, reading artist-friendly formats from disk, processing them into your engine-ready formats, and delivering them to your game runtime. Distill handles dependencies between assets, import & build caching, cross-device hot reloading during development, packing assets for a shippable game build, and more.

Vision

To create an open-source go-to solution for asset processing in games.

Features

The project contains a number of different components, and some can be used independently of others. You can combine them in different ways to tailor them to your workflow. Checkmarks indicate feature support - some features are dreamed up but not implemented.

Daemon

The daemon watches for filesystem events, imports source files to produce assets, manages metadata and serves asset load requests. It is primarily intended for use during development, but can also be used in a distributed game if appropriate. The daemon is very resource efficient and only does work when either a file changes or work is requested. Other components interact with the daemon through a transport-agnostic RPC protocol.

βœ“ Asset UUIDs & Dependency Graphs

Every asset is identified by a 16-byte UUID that is generated when a source file is imported for the first time. Importers also produce an asset's build and load dependencies in terms of UUIDs which can be used to efficiently traverse the dependency graph of an asset without touching the filesystem.

βœ“ Source file change detection

The daemon watches for filesystem changes and ensures source files are only imported when they change. Metadata and hashes are indexed locally in LMDB and version controlled in .meta files. Filesystem modification time and hashes are used to reduce redundant imports across your whole team to the greatest extent possible.

βœ“ Import Caching

Assets imported from a source file are cached by a hash of their source file content and its ID, avoiding expensive parsing and disk operations.

βœ“ Asset Change Log

Asset metadata is maintained in LMDB, a transactional database. The database's consistency guarantees and snapshot support provides a way to synchronize external data stores with the current state of the asset metadata using the Asset Change Log of asset changes.

βœ“ Metadata Tracking & Caching

When assets are imported from source files, metadata is generated and stored in `.meta` files together with source file, as well as cached in a database. Commit these to version control along with your source files.

βœ“ Move & Rename Source Files Confidently

Since metadata is stored with the source file and UUIDs are used to identify individual assets, users can move, rename and share source files with others without breaking references between assets.

βœ“ Bring Your Own Asset Types

Asset types are not included in this project. You define your own asset types and source file formats by implementing the `Importer` trait and registering these with a file extension. The Daemon will automatically run your `Importer` for files with the registered extension as required. All asset types must implement `serde::Serialize` + `serde::Deserialize` + `TypeUuidDynamic` + `Send`.

βœ“ RON Importer - *OPTIONAL*

An optional Importer and derive macro is included to simplify usage of serialized Rust types as source files using `serde`.

Type definition:

#[derive(Serialize, Deserialize, TypeUuid, SerdeImportable)]
#[uuid = "fab4249b-f95d-411d-a017-7549df090a4f"]
pub struct CustomAsset {
    pub cool_string: String,
    pub handle_from_path: Handle<crate::image::Image>,
    pub handle_from_uuid: Handle<crate::image::Image>,
}

custom_asset.ron:

{
    "fab4249b-f95d-411d-a017-7549df090a4f": 
    (
        cool_string: "thanks",
        // This references an asset from a file in the same directory called "amethyst.png"
        handle_from_path: "amethyst.png", 
        // This references an asset with a UUID (see associated .meta file for an asset's UUID)
        handle_from_uuid: "6c5ae1ad-ae30-471b-985b-7d017265f19f"
    )
}

Loader

The Loader module loads assets and their dependencies for a user-implemented AssetStorage trait to handle. Loader supports a pluggable LoaderIO trait for customizing where assets and their metadata are loaded from.

βœ“ Hot Reloading

The built-in `RpcIO` implementation of `LoaderIO` talks to the `Daemon` and automatically reloads assets when an asset has changed.

βœ“ Automatic Loading of Dependencies

When a source file is imported and an asset is produced, dependencies are gathered for the asset and saved as metadata. The Loader automatically ensures that dependencies are loaded before the asset is loaded, and that dependencies are unloaded when they are no longer needed.

βœ“ serde` Support for Handles πŸŽ‰πŸ’―

An optional Handle type is provided with support for deserialization and serialization using `serde`. Handles can be deserialized as either a UUID or a path.

βœ“ Automatic Registration of Handle Dependencies πŸŽ‰πŸ’―

Handle references that are serialized as part of an asset are automatically registered and the referenced assets are guaranteed to be loaded by the Loader before the depending asset is loaded. This means Handles in assets are always guaranteed to be valid and loaded.

βœ“ Packing for distribution

To distribute your game, you will want to pack assets into files with enough metadata to load them quickly. The CLI supports packing assets into a file format which the `PackfileIO` implementation supports loading.

TODO

Networked artifact caching

Results of imports and builds can be re-used across your whole team using a networked cache server.

Platform-specific builds

Provide customized build parameters when building an asset and tailor the build artifact for a specific platform.

Scalable build pipeline

Once assets are imported from sources, the build system aims to be completely pure in the functional programming sense. Inputs to asset builds are all known and declared in the import step. This design enables parallelizable and even distributed builds.

Searching

Search tags can be produced at import and are automatically indexed by tantivy which enables super fast text search. The search index is incrementally maintained by subscribing to the Asset Change Log.

Cross-Platform Support

The project aims to support as many platforms as possible with the Loader module, while the Daemon may never be able to run on platforms without solid filesystem support such as WASM. Current known supported platforms:

Linux/Mac/Windows: Loader + Daemon

iOS: Loader

Examples

To run:

  • cd examples/handle_integration
  • cargo run
  • The example includes an image asset type, so try to put some images (png, jpg, tga) in the assets folder!

Have a look at the generated .meta files in the assets folder!

Get involved

This project is primarily used by Amethyst and casual communication around development happens in the #engine-general channel of the Amethyst Discord server. Feel free to drop by for a chat. Contributions or questions are very welcome!

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

See LICENSE-APACHE and LICENSE-MIT.

License

Licensed under either of

at your option.

PLEASE NOTE that some dependencies may be licensed under other terms. These are listed in deny.toml under licenses.exceptions on a best-effort basis, and are validated in every CI run using cargo-deny.

Vendored Code

In addition to crate dependencies, this project contains some vendored code:

More Repositories

1

amethyst

Data-oriented and data-driven game engine written in Rust
Rust
7,982
star
2

specs

Specs - Parallel ECS
Rust
2,483
star
3

rlua

High level Lua bindings to Rust
C
1,628
star
4

legion

High performance Rust ECS library
Rust
1,616
star
5

bracket-lib

The Roguelike Toolkit (RLTK), implemented for Rust.
Rust
1,500
star
6

rustrogueliketutorial

Roguelike Tutorial in Rust - using RLTK
Rust
894
star
7

rendy

State of the art "build your own engine" kit powered by gfx-hal
Rust
815
star
8

shred

Shared resource dispatcher
Rust
231
star
9

evoli

An ecosystem-simulation game made with Amethyst
Rust
217
star
10

amethyst-starter-2d

Seed project for 2D games
Rust
201
star
11

space-menace

An action 2D platformer made with Amethyst game engine
Rust
180
star
12

shotcaller

A moddable RTS/MOBA game made with bracket-lib and minigene.
Rust
143
star
13

serde-diff

Utility for comparing two structs and re-applying the differences to other structs
Rust
120
star
14

hibitset

Hierarchical bit set container
Rust
114
star
15

voxel-mapper

Make beautiful voxel maps.
Rust
113
star
16

specs-physics

nphysics integration for the Specs entity component system
Rust
94
star
17

sheep

Modular and lightweight spritesheet packer πŸ‘
Rust
89
star
18

tools

Game development tools for the Amethyst engine
Rust
80
star
19

grumpy_visitors

πŸ§™β€β™‚οΈπŸ§™β€β™€οΈ A prototype of a top-down EvilInvasion-like 2D arcade/action (with co-op!)
Rust
77
star
20

amethyst-imgui

imgui integration for Amethyst
Rust
66
star
21

legion_transform

A Unity-inspired hierarchical transform implementation using Legion ECS
Rust
51
star
22

editor-core

Crate that allows an Amethyst game to communicate with an editor.
Rust
44
star
23

rfcs

RFCs are documents that contain major plans and decisions for the engine
32
star
24

dwarf_seeks_fortune

A 2D puzzle platformer made with the Amethyst game engine.
Rust
25
star
25

amethyst_iced

Iced rendering plugin for the Amethyst game engine
Rust
24
star
26

pong_wasm

WASM end-to-end proof of concept -- work in progress
Rust
22
star
27

web_worker

Rust
19
star
28

ludumdare42

A game made by the Amethyst team for Ludum Dare 42
Rust
16
star
29

website-legacy

Project website and blog (DEPRECATED)
HTML
11
star
30

website

Official Amethyst website
JavaScript
9
star
31

amethyst-rhusics

A bridge between Amethyst and rhusics (unmaintained)
Rust
9
star
32

awesome-specs

A curated list of projects that use or help with using Specs.
6
star
33

crystal-editor

Svelte
6
star
34

ludumdare43

Rust
2
star
35

resources

Files that are important to keep around but not tied to any specific code base
1
star
36

builder

The docker container used in amethyst's CI/CD infrastructure.
Dockerfile
1
star
37

wasm_rush_report

Report about adding WASM support to the Amethyst game engine.
1
star
38

laminar-ffi

Crate that exposes Laminar functionality to C
Rust
1
star
39

amethyst-docs-builder

Webhook server that builds amethyst's book and documentation
Go
1
star
40

amethystup

Setup script for Amethyst dependencies
Shell
1
star
41

documentation

Non-rustdoc documentation and policies
1
star