• Stars
    star
    157
  • Rank 238,399 (Top 5 %)
  • Language
    Rust
  • License
    Apache License 2.0
  • Created about 2 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A crate to help you copy things into raw buffers without invoking spooky action at a distance (undefined behavior).

πŸ—œ presser

Utilities to help make copying data around into raw, possibly-uninitialized buffers easier and safer.

Embark Embark Crates.io Published Docs Git Docs dependency status

presser can help you when copying data into raw buffers. One primary use-case is copying data into graphics-api-allocated buffers which will then be accessed by the GPU. Common methods for doing this right now in Rust can often invoke UB in subtle and hard-to-see ways. For example, viewing an allocated but uninitialized buffer as an &mut [u8] is instantly undefined behavior*, and transmuteing even a T: Copy type which has any padding bytes in its layout as a &[u8] to be the source of a copy is also instantly undefined behavior, in both cases because it is invalid to create a reference to an invalid value (and uninitialized memory is an invalid u8), even if your code never actually accesses that memory. This immediately makes what seems like the most straightforward way to copy data into buffers unsound 😬

* If you're currently thinking to yourself "bah! what's the issue? surely an uninit u8 is just any random bit pattern and that's fine we don't care," check out this blog post by @RalfJung, one of the people leading the effort to better define Rust's memory and execution model. As is explored in that blog post, an uninit piece of memory is not simply an arbitrary bit pattern, it is a wholly separate state about a piece of memory, outside of its value, which lets the compiler perform optimizations that reorder, delete, and otherwise change the actual execution flow of your program in ways that cannot be described simply by "the value could have some possible bit pattern". LLVM and Clang are changing themselves to require special noundef attribute to perform many important optimizations that are otherwise unsound. For a concrete example of the sorts of problems this can cause, see this issue @scottmcm hit.

A bad example

#[derive(Clone, Copy)]
#[repr(C)]
struct MyDataStruct {
    a: u8,
    b: u32,
}

let my_data = MyDataStruct { a: 0, b: 42 };

// 🚨 MyDataStruct contains 3 padding bytes after `a`, which are
// uninit, therefore getting a slice that includes them is UB!
let my_data_bytes: &[u8] = transmute(&my_data);

// allocate an uninit buffer of some size
let my_buffer: MyBufferType = some_api.alloc_buffer_size(2048);

// 🚨 this is UB for the same reason, these bytes are uninit!
let buffer_as_bytes: &mut [u8] =
    slice::from_raw_parts(my_buffer.ptr(), my_buffer.size());

// 🚨 this is UB because not only are both slices invalid,
// this is not ensuring proper alignment!
buffer_as_bytes.copy_from_slice(my_data_bytes);

presser helps with this by allowing you to view raw allocated memory of some size as a "Slab" of memory and then provides safe, valid ways to copy data into that memory:

A good example

#[derive(Clone, Copy)]
#[repr(C)]
struct MyDataStruct {
    a: u8,
    b: u32,
}

let my_data = MyDataStruct { a: 0, b: 42 };

// allocate an uninit buffer of some size
let my_buffer: MyBufferType = some_api.alloc_buffer_size(2048);

// use `RawAllocation` helper to allow access to a presser `Slab`.
// alternatively, you could implement the `Slab` on your buffer type directly if that
// type is owned by your code!
let mut raw_allocation = presser::RawAllocation::from_raw_parts(my_buffer.ptr(), my_buffer.size());

// here we assert that we have exclusive access to the data in the buffer, and get the actual
// `Slab` to use to copy into.
let mut slab = unsafe { raw_allocation.borrow_as_slab() };

// now we may safely copy `my_data` into `my_buffer`, starting at a minimum offset of 0 into the buffer
let copy_record = presser::copy_to_offset(&my_data, &mut slab, 0)?;

// note that due to alignment requirements of `my_data`, the *actual* start of the bytes of
// `my_data` may be placed at a different offset than requested. so, we check the returned
// `CopyRecord` to check the actual start offset of the copied data.
let actual_start_offset = copy_record.copy_start_offset;

Note that actually accessing the copied data is a completely separate issue which presser does not (as of now) concern itself with. BE CAREFUL!

See more in the git main docs or the released version docs.

Contribution

Contributor Covenant

We welcome community contributions to this project.

Please read our Contributor Guide for more information on how to get started. Please also read our Contributor Terms before you make any contributions.

Any contribution intentionally submitted for inclusion in an Embark Studios project, shall comply with the Rust standard licensing model (MIT OR Apache 2.0) and therefore be dual licensed as described below, without any additional terms or conditions:

License

This contribution is dual licensed under EITHER OF

at your option.

For clarity, "your" refers to Embark or any other licensee/user of the contribution.

More Repositories

1

rust-gpu

πŸ‰ Making Rust a first-class language and ecosystem for GPU shaders 🚧
Rust
6,796
star
2

kajiya

πŸ’‘ Experimental real-time global illumination renderer πŸ¦€
Rust
4,753
star
3

wg-ui

WireGuard Web UI for self-serve client configurations, with optional auth.
Go
1,468
star
4

cargo-deny

❌ Cargo plugin for linting your dependencies πŸ¦€
Rust
1,254
star
5

puffin

🐦 Friendly little instrumentation profiler for Rust πŸ¦€
Rust
973
star
6

rust-ecosystem

Rust wants & tracking for Embark πŸ¦€
Rust
843
star
7

physx-rs

🎳 Rust binding for NVIDIA PhysX πŸ¦€
Rust
646
star
8

blender-tools

🐡 Embark Addon for Blender
Python
411
star
9

cargo-about

πŸ“œ Cargo plugin to generate list of all licenses for a crate πŸ¦€
Rust
320
star
10

tryhard

πŸ’« Easily retry futures πŸ¦€
Rust
200
star
11

crash-handling

Collection of crates to deal with crashes
Rust
135
star
12

poll-promise

A Rust promise for games and immediate mode GUIs
Rust
127
star
13

rpmalloc-rs

🐏 rpmalloc global memory allocator for Rust πŸ¦€
Rust
126
star
14

discord-sdk

An open implementation of the Discord Game SDK in Rust
Rust
108
star
15

skyhook

Simple Python communication system for DCC's and Game Engines
Python
104
star
16

spirt

SPIR-πŸ‡Ή: shader-focused IR to target, transform and translate from πŸ¦€
Rust
103
star
17

superluminal-perf-rs

πŸ”† Superluminal Performance profiler Rust API πŸ¦€
Rust
94
star
18

ash-molten

πŸŒ‹ Statically linked MoltenVK for Vulkan on Mac using Ash πŸ¦€
Rust
91
star
19

cargo-deny-action

❌ GitHub Action for cargo-deny πŸ¦€
Shell
89
star
20

cargo-fetcher

🎁 cargo plugin for quickly fetching dependencies πŸ¦€
Rust
83
star
21

mirror-mirror

πŸͺž Powerful reflection library for Rust
Rust
79
star
22

relnotes

Automatic GitHub Release Notes
Rust
77
star
23

opensource-template

🌻 Template for creating new repositories
74
star
24

tame-oauth

πŸ” Small OAuth crate that follows the sans-io approach πŸ¦€
Rust
68
star
25

tiny-bench

A tiny benchmarking library
Rust
59
star
26

k8s-buildkite-plugin

Run any buildkite build step as a Kubernetes Job
Jsonnet
49
star
27

krates

πŸ“¦ Creates graphs of crates from cargo metadata πŸ¦€
Rust
45
star
28

opensource

Open source processes, policies, and info
Rust
42
star
29

cervo

Utility wrappers for tract
Rust
40
star
30

tracing-logfmt

A logfmt formatter for tracing subscriber logs
Rust
38
star
31

tame-gcs

πŸ“‚ A small library with a limited set of Google Cloud Storage operations πŸ¦€
Rust
37
star
32

octobors

Rust program for automerging PRs based on a few rules
Rust
35
star
33

server-framework

Framework for running network services, opinions included
Rust
35
star
34

nfd2

OS native dialogs for Windows, MacOS, and Linux
Rust
32
star
35

cfg-expr

A parser and evaluator for Rust cfg() expressions. Targets as of Rust 1.58.0 are supported.
Rust
32
star
36

pdm-plugin-torch

A tool for managing torch-variants with PDM.
Python
31
star
37

opa-policies

Contains OPA Policies for Dockerfiles, Kubernetes YAMLs, Terraform, etc
Open Policy Agent
29
star
38

gsutil

Minimal gsutil replacement
Rust
27
star
39

spdx

πŸ†” Helper crate for SPDX expressions. πŸ¦€
Rust
27
star
40

tame-index

Small crate for interacting with cargo registry indices
Rust
23
star
41

fsr-rs

Rust bindings for AMD FidelityFXβ„’ Super Resolution
C
22
star
42

opensource-website

🌐 Hub for Embark's open source efforts
HTML
22
star
43

buildkite-jobify

πŸ‘· Kubekite, but in Rust, using configuration from your repos πŸ¦€
Rust
21
star
44

spirv-tools-rs

πŸ›  Wrapper crate for SPIRV-Tools πŸ¦€
C++
20
star
45

emote

Reinforcement learning library from Embark Studios
Python
19
star
46

proto-gen

A protobuf generation runner/cli using tonic build
Rust
18
star
47

rymder

Unofficial agones client SDK
Rust
17
star
48

sentry-contrib-rust

Integrates crashdump reporting with Sentry
Rust
15
star
49

tame-oidc

🧬 Small OAuth crate that follows the sans-io approach πŸ¦€
Rust
11
star
50

tame-webpurify

πŸ’© Small HTTP client for the Webpurify API following the sans-io approach πŸ¦€
Rust
9
star
51

cloud-dns

A wrapper for the Google Cloud DNS API (https://cloud.google.com/dns)
Rust
7
star
52

boh

Rust
5
star
53

tracing-ext-ffi-subscriber

Simple subscriber for forwarding tracing spans to a C or C++ profiling API.
Rust
4
star
54

server-code-exciser

A program that contains grammars and functionality to remove server only code from code bases.
C#
4
star
55

gke-accelerated-xorg-example

Example project for running remote rendering on GKE
Dockerfile
3
star
56

helix-oidc

🧬 Helix Perforce OIDC validator
Go
3
star
57

cassini

Topology-aware distributed cache
Go
2
star
58

minwin

Rust
2
star
59

container-packer-qemu

πŸ“¦ Dockerized packer with qemu
Dockerfile
1
star
60

rustc-compile-time-regress

Rust
1
star
61

missing-symbols

Rust
1
star
62

.github

Default community health files for GitHub (https://help.github.com/en/github/building-a-strong-community/creating-a-default-community-health-file-for-your-organization)
1
star
63

toml-span

Span preserving toml deserializer
Rust
1
star