• Stars
    star
    103
  • Rank 333,046 (Top 7 %)
  • Language
    Rust
  • License
    Apache License 2.0
  • Created about 3 years ago
  • Updated 10 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

SPIR-πŸ‡Ή: shader-focused IR to target, transform and translate from πŸ¦€

SPIR-πŸ‡Ή

β‹―πŸ’’ πŸ‡Ήarget πŸ † πŸ‡Ήransform πŸ † πŸ‡Ήranslate β‹―πŸ’’

Embark Crates.io Docs Git Docs dependency status Build status

SPIR-πŸ‡Ή is a research project aimed at exploring shader-oriented IR designs derived from SPIR-V, and producing a framework around such an IR to facilitate advanced compilation pipelines, beyond what existing SPIR-V tooling allows for.

Such a need arose in the Rust-GPU project, which requires a variety of legalization passes to turn general-purpose (Rust1) code operating on untyped memory, into GPU-friendly direct data-flow.
Our goal is to replace the existing Rust-GPU SPIR-V legalizations passes with SPIR-πŸ‡Ή equivalents - but even more imporantly, SPIR-πŸ‡Ή should allow writing much more powerful legalization/optimization passes, that would've been unfathomable2 for direct SPIR-V manipulation.


1 Rust is not unique in its needs here, and more languages (or IRs) could eventually make use of such a framework, but the initial design and implementation work has focused on Rust-GPU

2 not outright impossible, but requiring excessive development/maintenance cost, having to constantly balance correctness and power (more conservative passes are easier to trust), etc.

Disclaimer

This project is not affiliated, associated, authorized, endorsed by, or in any way officially connected with Khronos Group Inc., or any of its subsidiaries or its affiliates. The official Khronos Group Inc. website can be found at https://www.khronos.org.
The names SPIR, SPIR-V, as well as related names, marks, emblems and images are trademarks of their respective owners.

Additional context: the name of this project is a pun on SPIR-V, and entirely unrelated to SPIR (the older IR standard).

Status

🚧 This project is in active design and development, many details can and will change 🚧

If you're interested in using SPIR-πŸ‡Ή yourself, you may want to first take a look at the issue tracker for relevant issues, and even open new ones describing your usecase.
With the initial focus being on Rust-GPU's usecase, various (otherwise desirable) functionality/APIs/docs may be lacking, or rapidly changing - at the same time, discussions around widening the scope and usability of SPIR-πŸ‡Ή in the long term are still welcome.

Non-goals (at least in the short term)

  • supporting the ("OpenCL") Kernel dialect of SPIR-V
    • Kernel SPIR-V is much closer to LLVM IR, than Shader SPIR-V, and as such tooling oriented around LLVM is more likely to be a better fit
  • textual syntax that can be parsed back
    • i.e. the pretty-printer output is purely a visualization

Designed and implemented so far

IR data types:

  • allowing near-arbitrary SPIR-V instructions for any unrecognized opcodes
    • IDs are replaced with interned/"entity" handles (see below)
  • interning for attributes (decorations & similar), types and constants
    • i.e. automatic deduplication, efficient indexing, and no concept of "definition" (only uses of interned handles can lead to a module being considered to contain a specific type/constant)
  • "entity" system for e.g. definitions in a module, instructions in a function, etc.
    • disallows iteration in favor of/forcing the use of efficient indexing
  • structured control-flow "regions" inspired by RVSDG, stricter than SPIR-V (see ControlRegionDef's docs for more details)

Framework utilities:

  • visit/transform: immutable/mutable IR traversal
  • print: pretty-printer with (styled and hyperlinked) HTML output

Passes (to/from/on SPIR-πŸ‡Ή):

  • spv::lower: "lowering" from SPIR-V, normalizing away many irrelevant details
    • lossy for some relevant details (these are bugs, though many are non-semantic so lower priority)
  • spv::lift: "lifting" back up to SPIR-V, making arbitrary choices where necessary
    • comparable to e.g. generating GLSL syntax from SPIR-V, just one level down
  • cfg::Structurizer: (re)structurization, from arbitrary control-flow to the stricter structured "regions"
  • passes::link: mapping (linkage) imports to relevant exports

Simple example (with non-trivial control-flow)

#version 450
out int output0;
void main() {
    int o = 1;
    for(int i = 1; i < 10; i++)
    	  o *= i;
    output0 = o;
}

@vertex
fn main() -> @location(0) i32 {
    var o: i32 = 1;
    for(var i: i32 = 1; i < 10; i++) {
    	o *= i;
    }
    return o;
}

SPIR-πŸ‡Ή

#[spv.Decoration.Flat]
#[spv.Decoration.Location(Location: 0)]
global_var GV0 in spv.StorageClass.Output: s32

func F0() -> spv.OpTypeVoid {
  loop(v0: s32 <- 1s32, v1: s32 <- 1s32) {
    v2 = spv.OpSLessThan(v1, 10s32): bool
    (v3: s32, v4: s32) = if v2 {
      v5 = spv.OpIMul(v0, v1): s32
      v6 = spv.OpIAdd(v1, 1s32): s32
      (v5, v6)
    } else {
      (spv.OpUndef: s32, spv.OpUndef: s32)
    }
    (v3, v4) -> (v0, v1)
  } while v2
  spv.OpStore(Pointer: &GV0, Object: v0)
}

%typeof_output0 = OpTypePointer Output %i32
%output0 = OpVariable %typeof_output0 Output

%typeof_main = OpTypeFunction %void
%main = OpFunction %void None %typeof_main
  %entry = OpLabel
    OpBranch %bb0
  %bb0 = OpLabel
    OpBranch %bb1
  %bb1 = OpLabel
    %o = OpPhi %i32 %1_i32 %bb0 %o_next %bb5
    %i = OpPhi %i32 %0_i32 %bb0 %i_next %bb5
    OpLoopMerge %bb6 %bb5 None
    OpBranch %bb2
  %bb2 = OpLabel
    %cond = OpSLessThan %bool %i %10_i32
    OpSelectionMerge %bb4 None
  OpBranchConditional %cond %bb4 %bb3
  %bb3 = OpLabel
    OpBranch %bb6
  %bb4 = OpLabel
    %o_next = OpIMul %i32 %o %i
    OpBranch %bb5
  %bb5 = OpLabel
    %i_next = OpIAdd %i32 %i %1_i32
    OpBranch %bb1
  %bb6 = OpLabel
    OpStore %output0 %o
    OpReturn
OpFunctionEnd

GPU (shader) IR landscape overview

(and the vision of how SPIR-πŸ‡Ή fits into it)

The distinction being made here is between:

  • Interchange IRs (standards that many tools can use to interoperate)
    • SPIR-V was very much intended as such a standard (outside of the GPU space, wasm is also a great example)
    • they only need to encode the right concepts, not straying too far away from what tools understand, but the design effort is often oriented around being a "serialization" format
  • Compiler IRs (non-standard implementation details of compilers)
    • LLVM is quite well-known, but Mesa's NIR is even closer to SPIR-πŸ‡Ή (both being shader-oriented, and having similar specialized choices of e.g. handling control-flow)
    • these have to handle legalization/optimization passes quite well, and in general a lot of on-the-fly transformations - as their main purpose is to expedite such operations
    • this is where SPIR-πŸ‡Ή sits, as a kind of "relative"/dialect of SPIR-V, but making trade-offs in favor of the "intra-compiler" usage

Contribution

Contributor Covenant

We welcome community contributions to this project.

Please read our Contributor Guide for more information on how to get started. Please also read our Contributor Terms before you make any contributions.

Any contribution intentionally submitted for inclusion in an Embark Studios project, shall comply with the Rust standard licensing model (MIT OR Apache 2.0) and therefore be dual licensed as described below, without any additional terms or conditions:

License

This contribution is dual licensed under EITHER OF

at your option.

For clarity, "your" refers to Embark or any other licensee/user of the contribution.

More Repositories

1

rust-gpu

πŸ‰ Making Rust a first-class language and ecosystem for GPU shaders 🚧
Rust
6,796
star
2

kajiya

πŸ’‘ Experimental real-time global illumination renderer πŸ¦€
Rust
4,753
star
3

wg-ui

WireGuard Web UI for self-serve client configurations, with optional auth.
Go
1,468
star
4

cargo-deny

❌ Cargo plugin for linting your dependencies πŸ¦€
Rust
1,254
star
5

puffin

🐦 Friendly little instrumentation profiler for Rust πŸ¦€
Rust
973
star
6

rust-ecosystem

Rust wants & tracking for Embark πŸ¦€
Rust
843
star
7

physx-rs

🎳 Rust binding for NVIDIA PhysX πŸ¦€
Rust
646
star
8

blender-tools

🐡 Embark Addon for Blender
Python
411
star
9

cargo-about

πŸ“œ Cargo plugin to generate list of all licenses for a crate πŸ¦€
Rust
320
star
10

tryhard

πŸ’« Easily retry futures πŸ¦€
Rust
200
star
11

presser

A crate to help you copy things into raw buffers without invoking spooky action at a distance (undefined behavior).
Rust
157
star
12

crash-handling

Collection of crates to deal with crashes
Rust
135
star
13

poll-promise

A Rust promise for games and immediate mode GUIs
Rust
127
star
14

rpmalloc-rs

🐏 rpmalloc global memory allocator for Rust πŸ¦€
Rust
126
star
15

discord-sdk

An open implementation of the Discord Game SDK in Rust
Rust
108
star
16

skyhook

Simple Python communication system for DCC's and Game Engines
Python
104
star
17

superluminal-perf-rs

πŸ”† Superluminal Performance profiler Rust API πŸ¦€
Rust
94
star
18

ash-molten

πŸŒ‹ Statically linked MoltenVK for Vulkan on Mac using Ash πŸ¦€
Rust
91
star
19

cargo-deny-action

❌ GitHub Action for cargo-deny πŸ¦€
Shell
89
star
20

cargo-fetcher

🎁 cargo plugin for quickly fetching dependencies πŸ¦€
Rust
83
star
21

mirror-mirror

πŸͺž Powerful reflection library for Rust
Rust
79
star
22

relnotes

Automatic GitHub Release Notes
Rust
77
star
23

opensource-template

🌻 Template for creating new repositories
74
star
24

tame-oauth

πŸ” Small OAuth crate that follows the sans-io approach πŸ¦€
Rust
68
star
25

tiny-bench

A tiny benchmarking library
Rust
59
star
26

k8s-buildkite-plugin

Run any buildkite build step as a Kubernetes Job
Jsonnet
49
star
27

krates

πŸ“¦ Creates graphs of crates from cargo metadata πŸ¦€
Rust
45
star
28

opensource

Open source processes, policies, and info
Rust
42
star
29

cervo

Utility wrappers for tract
Rust
40
star
30

tracing-logfmt

A logfmt formatter for tracing subscriber logs
Rust
38
star
31

tame-gcs

πŸ“‚ A small library with a limited set of Google Cloud Storage operations πŸ¦€
Rust
37
star
32

octobors

Rust program for automerging PRs based on a few rules
Rust
35
star
33

server-framework

Framework for running network services, opinions included
Rust
35
star
34

nfd2

OS native dialogs for Windows, MacOS, and Linux
Rust
32
star
35

cfg-expr

A parser and evaluator for Rust cfg() expressions. Targets as of Rust 1.58.0 are supported.
Rust
32
star
36

pdm-plugin-torch

A tool for managing torch-variants with PDM.
Python
31
star
37

opa-policies

Contains OPA Policies for Dockerfiles, Kubernetes YAMLs, Terraform, etc
Open Policy Agent
29
star
38

gsutil

Minimal gsutil replacement
Rust
27
star
39

spdx

πŸ†” Helper crate for SPDX expressions. πŸ¦€
Rust
27
star
40

tame-index

Small crate for interacting with cargo registry indices
Rust
23
star
41

fsr-rs

Rust bindings for AMD FidelityFXβ„’ Super Resolution
C
22
star
42

opensource-website

🌐 Hub for Embark's open source efforts
HTML
22
star
43

buildkite-jobify

πŸ‘· Kubekite, but in Rust, using configuration from your repos πŸ¦€
Rust
21
star
44

spirv-tools-rs

πŸ›  Wrapper crate for SPIRV-Tools πŸ¦€
C++
20
star
45

emote

Reinforcement learning library from Embark Studios
Python
19
star
46

proto-gen

A protobuf generation runner/cli using tonic build
Rust
18
star
47

rymder

Unofficial agones client SDK
Rust
17
star
48

sentry-contrib-rust

Integrates crashdump reporting with Sentry
Rust
15
star
49

tame-oidc

🧬 Small OAuth crate that follows the sans-io approach πŸ¦€
Rust
11
star
50

tame-webpurify

πŸ’© Small HTTP client for the Webpurify API following the sans-io approach πŸ¦€
Rust
9
star
51

cloud-dns

A wrapper for the Google Cloud DNS API (https://cloud.google.com/dns)
Rust
7
star
52

boh

Rust
5
star
53

tracing-ext-ffi-subscriber

Simple subscriber for forwarding tracing spans to a C or C++ profiling API.
Rust
4
star
54

server-code-exciser

A program that contains grammars and functionality to remove server only code from code bases.
C#
4
star
55

gke-accelerated-xorg-example

Example project for running remote rendering on GKE
Dockerfile
3
star
56

helix-oidc

🧬 Helix Perforce OIDC validator
Go
3
star
57

cassini

Topology-aware distributed cache
Go
2
star
58

minwin

Rust
2
star
59

container-packer-qemu

πŸ“¦ Dockerized packer with qemu
Dockerfile
1
star
60

rustc-compile-time-regress

Rust
1
star
61

missing-symbols

Rust
1
star
62

.github

Default community health files for GitHub (https://help.github.com/en/github/building-a-strong-community/creating-a-default-community-health-file-for-your-organization)
1
star
63

toml-span

Span preserving toml deserializer
Rust
1
star