• This repository has been archived on 10/Oct/2023
  • Stars
    star
    487
  • Rank 90,352 (Top 2 %)
  • Language
    Rust
  • License
    Mozilla Public Li...
  • Created over 2 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Riff automatically provides external dependencies for Rust projects, with support for other languages coming soon.

Riff

logo-light logo-dark

Riff is a tool that automatically provides external dependencies1 for software projects. To enter a shell environment with all your project's external dependencies installed, run this at the project root:

riff shell

You can also directly run commands with the shell environment applied but without entering the shell:

riff run cargo build

Riff currently supports Rust with support for other languages coming soon. It uses the Nix package manager to handle dependencies but doesn't require you to know or use Nix.

For a video demo of Riff in action, see below.

Requirements

To use Riff, you need to install these binaries on your system:

Installation

Using Nix

To install Riff using Nix (make sure to have flakes enabled):

nix profile install github:DeterminateSystems/riff

Using Homebrew

To install Riff on macOS using Homebrew:

brew install DeterminateSystems/riff/riff

Note: The riff Homebrew formula does not install Nix or Cargo.

Using cURL

You can find instructions for installing Riff using cURL on the releases page.

GitHub Actions

You can install Riff in your GitHub Actions pipelines using install-riff-action. Here's an example configuration:

steps:
  - uses: actions/checkout@v3
  - name: Install Nix
    uses: DeterminateSystems/nix-installer-action@main
  - name: Install Rust
    uses: actions-rs/toolchain@v1
    with:
      toolchain: stable
  - name: Install Riff
    uses: DeterminateSystems/install-riff-action@v1
  - name: Build Rust app
    run: riff run cargo build -- --release

Prompt Customization

You can customize your shell's prompt to display when you're in a Riff shell environment by relying on the $IN_NIX_SHELL environment variable.

If you use Starship, you get this information for free because the Nix shell module is enabled by default. However, if you want to add this to your shell prompt yourself, you can do that by adding $name to your prompt when $IN_NIX_SHELL is set.

In Bash, this might look something like:

export PS1="$PWD \${IN_NIX_SHELL:+\$name }\$ "

Note the escaping of \${IN_NIX_SHELL} and \$name. This prevents Bash from taking the current values of those environment variables and using them even after one of them has changed.

In Zsh, it might look similar to:

export PROMPT="$PWD \${IN_NIX_SHELL:+\$name }\$ "

And in Fish, you might use something like:

function fish_prompt
    echo -n "$PWD "
    if set -q IN_NIX_SHELL
        echo -n "$name "
    end
    echo -n "\$ "
end

Escaping $name is unnecessary here because Fish doesn't capture the value of $name until the function is run when your prompt is displayed.

The idea extends to any shell: if you can change the prompt and prevent it from evaluating the values of $IN_NIX_SHELL and $name until the prompt is displayed, you can add this information to your prompt.

What Riff provides

Most programming languages use language-specific package managers to handle dependencies, such as Cargo for the Rust language. But these language-specific tools typically don't handle dependencies written in other languages very well. They expect you to install those dependencies using some other tool and fail in mysterious ways when they're missing. Here's an example error from trying to build the octocrab crate without OpenSSL installed:

--- stderr
thread 'main' panicked at '

Could not find directory of OpenSSL installation, and this `-sys` crate cannot
proceed without this knowledge. If OpenSSL is installed and this crate had
trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
compilation process.

Make sure you also have the development packages of openssl installed.
For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

In cases like this, it's up to you to install missing external dependencies, which can be laborious, error prone, and hard to reproduce.

Riff enables you to bypass this problem entirely. It uses your your project's language-specific configuration to infer which external dependencies are required and creates a shell environment with those dependencies both installed and properly linked. In cases where those dependencies can't be inferred, for example in your build.rs script, you can explicitly declare them in your Cargo.toml.

These environments are transient in the sense that they don't affect anything outside the shell; they install dependencies neither globally nor in your current project, so you don't have to worry about Riff breaking anything on your system. When you exit the Riff shell, the dependencies are gone.

Offline mode

In cases where you want to limit Riff's access to the Internet, you can run it in offline mode, which disables all network usage except what's required by the nix develop command (which Riff runs in the background). You can enable offline mode using either the --offline flag or the RIFF_OFFLINE environment variable. Here are some examples:

# Via flag
riff run --offline

# Via environment variable
RIFF_OFFLINE=true riff shell

Example usage

In this example, we'll build the Prost project from source. Prost has an external dependency on OpenSSL, without which commands like cargo build and cargo run are doomed to fail. Riff provides those dependencies automatically, without you needing to install them in your regular environment. Follow these steps to see dependency inference in action:

git clone https://github.com/tokio-rs/prost.git
cd prost

# Enter the Riff shell environment
riff shell
# ✓ 🦀 rust: cargo, cmake, curl, openssl, pkg-config, rustc, rustfmt, zlib

# Check for the presence of openssl
which openssl
# The path should look like this:
# /nix/store/f3xbf94zykbh6drw6wfg9hdrfgwrkck7-openssl-1.1.1q-bin/bin/openssl
# This means that Riff is using the Nix-provided openssl

# Build the project
cargo build

# Leave the shell environment
exit

# Check for openssl again
which openssl
# This should either point to an openssl executable on your PATH or fail

How to declare package inputs

While Riff does its best to infer external dependencies from your project's crate dependencies, you can explicitly declare external dependencies if necessary by adding a riff block to the package.metadata block in your Cargo.toml. Riff currently supports three types of inputs:

  • build-inputs are external dependencies that some crates may need to link against.
  • environment-variables are environment variables you want to set in your dev shell.
  • runtime-inputs are libraries you want to add to your LD_LIBRARY_PATH to ensure that your dev shell works as expected.

Both build-inputs and runtime-inputs can be any packages available in Nixpkgs. You may find this particularly useful for build.rs scripts.

Here's an example Cargo.toml with an explicitly supplied Riff configuration:

[package]
name = "riff-example"
version = "0.1.0"
edition = "2021"

[package.metadata.riff]
build-inputs = [ "openssl" ]
runtime-inputs = [ "libGL" ]

[package.metadata.riff.environment-variables]
HI = "BYE"

# Other configuration

When you run riff shell in this project, Riff

  • adds OpenSSL to your build environment
  • sets the LD_LIBRARY_PATH environment variable to include libGL's library path
  • sets the HI environment variable to have a value of BYE

Target-specific dependencies

If a project has OS-, architecture-, or vendor-specific dependencies, you can define them in a targets block under package.metadata.riff. Here's an example for Apple M1 (aarch64-apple-darwin) systems:

[package.metadata.riff.targets.aarch64-apple-darwin]
build-inputs = [
  "darwin.apple_sdk.frameworks.CoreServices",
  "darwin.apple_sdk.frameworks.Security"
]

The Rust project maintains a list of well-known targets that you can view by running nix run nixpkgs#rustup target list. This field can also contain custom targets, such as riscv32imac-unknown-xous-elf, although riff makes no effort to support cross compiling at this time.

When target-specific dependencies are present, the build-inputs and runtime-inputs sections are unioned (joined), while the target-specific environment variables override default environment variables.

macOS framework dependencies

macOS users may encounter issues with so-called "framework" dependencies, such as Foundation, CoreServices, and Security. When these dependencies are missing, you may see error messages like this:

= note: ld: framework not found CoreFoundation

You can solve this by adding framework dependencies to your build-inputs as darwin.apple_sdk.frameworks.<framework>, for example darwin.apple_sdk.frameworks.Security. Here's an example Cargo.toml configuration that adds multiple framework dependencies:

[package.metadata.riff.targets.x86_64-apple-darwin]
build-inputs = [
  "darwin.apple_sdk.frameworks.CoreServices",
  "darwin.apple_sdk.frameworks.Security"
]

[package.metadata.riff.targets.aarch64-apple-darwin]
build-inputs = [
  "darwin.apple_sdk.frameworks.CoreServices",
  "darwin.apple_sdk.frameworks.Security"
]

Riff understands dependencies transitively

If you add Riff metadata to Cargo.toml, this doesn't just make it easier to build and run your project: it actually benefits consumers of your crate as well. That's because Riff can use this metadata transitively to infer which external dependencies are necessary across the entire crate dependency graph. Let's say that you release a crate called make-it-pretty that has an external dependency on libGL and you add that to your Cargo.toml:

[package.metadata.riff]
runtime-inputs = [ "libGL" ]

Now let's say that another Rust dev releases a crate called artify that depends on your make-it-pretty crate. If someone tries to build artify using Cargo, they may receive an error if they don't have libGL installed. But if they use Riff to build artify, Riff knows to install libGL without any user input.

The implication is that adding Riff metadata to your crates—if they have external dependencies—can benefit the Rust ecosystem more broadly.

How it works

When you run riff shell in a Rust project, Riff

  • reads your Cargo.toml configuration manifest to determine which external dependencies your project requires and then
  • uses the Nix package manager—in the background and without requiring any intervention on your part—to install any external dependencies, such as OpenSSL or Protobuf, and also sets any environment variables necessary to discover those tools. Once it knows which external tools are required, it
  • builds a custom shell environment that enables you to use commands like cargo build and cargo run without encountering the missing dependency errors that so often dog Rust development.

This diagram provides a basic visual description of that process:

Riff reads your Cargo.toml to infer external dependencies and then
      uses Nix to build a shell environment that provides those dependencies

Because Riff uses Nix, all of the dependencies that it installs are stored in your local Nix store, by default under /nix/store. For a more thorough breakdown of how Riff works, see the architecture docs.

Video demo

You can see a video demo of Riff in action here (click on the image for a larger version):

Asciicast video demo of Riff with preview image

In the video, running cargo build in the Prost project fails due to missing external dependencies. But running riff run cargo build succeeds because Riff is able to infer which external dependencies are missing and provide them in the background using Nix.

Garbage Collection

One of the disadvantages of Riff using Nix is that Nix can end up using substantial disk space. After using Riff on a few projects, you may notice the /nix/store directory grow in size. Luckily, it is relatively easy to reclaim this space by running nix-collect-garbage. Note that garbage collection may clean up any Riff shells you have used, which in turn may cause your next riff invocation to take slightly longer because Riff—using Nix—needs to fetch the dependencies that were cleaned up.

Direnv Integration

You can add Riff support to Direnv on a project-specific or global basis. To enable Riff in a project, create a .envrc file that contains this:

# reload when these files change
watch_file Cargo.toml Cargo.lock
# add any other files you might want to trigger a riff reload
# load the riff dev env
eval "$(riff print-dev-env)"

You can enable Riff support globally by either adding a use_riff function either to your ~/.config/direnv/direnvrc file or a new ~/.config/direnv/lib/riff.sh file. The use_riff function should look something like this:

use_riff() {
  watch_file Cargo.toml watch_file Cargo.lock
  eval "$(riff print-dev-env)"
}

With Direnv now aware of this function, you can enable Riff in any directory with:

echo "use riff" > .envrc

When you run direnv allow you will automatically enter the Riff shell every time you navigate to the project directory.

Privacy policy

For the sake of improving user experience, Riff does collect some telemetry. You can read the full privacy policy for Determinate Systems, the creators of Riff, here.

To disable telemetry on any Riff command invocation, you can either

  • Use the --disable-telemetry flag or
  • Set the RIFF_DISABLE_TELEMETRY environment variable to any value except false,0, or an empty string ("").

Here are some examples:

# Via flag
riff shell --disable-telemetry

# Via environment variable
RIFF_DISABLE_TELEMETRY=true riff run cargo build

Telemetry

When you use Riff, Riff generates a random version 4 UUID for you. It contains no personally identifiable information about you and is used to know how many people use the tool and to focus our limited research and development. You can delete this file at any time to create a new ID.

To see exactly what data we send, you may run Riff as follows:

$ RUST_LOG=riff::telemetry=debug riff run echo 'Hello, Riff!'
✓ 🦀 rust: cargo, openssl, pkg-config, rustc, rustfmt
  2022-09-29T21:29:01.476342Z DEBUG riff::telemetry: Sent telemetry data to https://registry.riff.determinate.systems/telemetry, telemetry: Telemetry { distinct_id: Some(Secret([REDACTED riff::telemetry::DistinctId])), system_os: "linux", system_arch: "x86_64", os_release_name: Some("NixOS"), os_release_version_id: Some("22.11"), riff_version: "1.0.1", nix_version: Some("nix (Nix) 2.12.0pre20220928_c3c0682"), is_tty: true, subcommand: Some("run"), detected_languages: {Rust}, in_ci: false }
    at src/telemetry.rs:129
    in riff::telemetry::send
    in riff::flake_generator::generate_flake_from_project_dir with project_dir: None, offline: false

This will also appear when running with --debug, accompanied by other debug logging.

The table below shows the data Riff collects in a more readable format:

Field Use
distinct_id The ID of the Riff installation. More specifically, we at Determinate Systems care about our weekly active users count and this field helps us measure that.
system_os The operating system Riff is running on.
system_arch The architecture Riff is running on.
os_release_name The distribution Riff is running on. This uses the NAME field of /etc/os-release if it exists.
os_release_version_id The version of the distribution Riff is running on. This uses the VERSION_ID field of /etc/os-release if it exists.
riff_version The version of Riff being used.
nix_version The version of Nix being used by Riff.
is_tty Whether Riff is being run interactively.
subcommand The subcommand Riff is executing. This only contains information about the Riff subcommand, and not any commands being run by Riff (i.e. riff run echo 'Hello, Riff!' will not send any telemetry including the fact that Riff ran echo 'Hello, Riff!').
detected_languages Which languages Riff detected in the project.
in_ci Whether Riff is being used in CI (e.g. GitHub Actions).

Community

If you'd like to discuss Riff with other users, join our Discord (also bridged to a Matrix room).

Footnotes

  1. We define external dependencies as those that are written in another language and thus can't be installed using the same language-specific package manager that you use to build your code. ↩

More Repositories

1

nix-installer

Install Nix and flakes with the fast and reliable Determinate Nix Installer, with over 2 million installs.
Rust
1,919
star
2

zero-to-nix

Zero to Nix is your guide to learning Nix and flakes. Created by Determinate Systems.
MDX
890
star
3

magic-nix-cache-action

Save 30-50%+ of CI time without any effort or cost. Use Magic Nix Cache, a totally free and zero-configuration binary cache for Nix on GitHub Actions.
TypeScript
354
star
4

magic-nix-cache

Save 30-50%+ of CI time without any effort or cost. Use Magic Nix Cache, a totally free and zero-configuration binary cache for Nix on GitHub Actions.
Rust
231
star
5

update-flake-lock

Automatically refresh your Nix Flakes.
TypeScript
226
star
6

nuenv

A Nushell environment for Nix
Nushell
196
star
7

nix-netboot-serve

Make any NixOS system netbootable with 10s cycle times.
Rust
196
star
8

flake-checker

Health checks for your Nix flakes
Rust
160
star
9

nix-installer-action

The Github Action for the Determinate Nix Installer
TypeScript
150
star
10

fh

The official CLI for FlakeHub: search for flakes, and add new inputs to your Nix flake.
Rust
110
star
11

bootspec-secureboot

Rust
88
star
12

nix-to-kubernetes

An example of deploying Nix-built Docker images to Kubernetes.Not act
Nix
78
star
13

flake-schemas

Schemas for common flake output types
Nix
77
star
14

macos-ephemeral

Scripts and instructions for making ephemeral macOS machines with Mosyle MDM support.
Shell
76
star
15

flake-checker-action

A GitHub Action that performs health checks for your Nix flake.lock files
TypeScript
45
star
16

nixos-vault-service

Nix
43
star
17

terraform-provider-hydra

Declaratively configure your Hydra server with Terraform.
Go
43
star
18

nix-github-actions

An example project showing how to use Nix to replace third-party GitHub Actions
Nix
43
star
19

nix-policy

Experiments with Nix and Open Policy Agent
Nix
27
star
20

bonk

Erase macOS machines over Tailscale.
Go
26
star
21

flakehub-push

Release your Nix flake to FlakeHub.com.
Rust
26
star
22

bootspec

Implementation of RFC-0125's datatype and synthesis tooling.
Rust
23
star
23

nix-wasm-example

Nix
19
star
24

nix-copy-deploy

An example project for the nix copy utility
Shell
13
star
25

zpool-auto-expand-partitions

Rust
12
star
26

hydra-github-jobsets-generator

Generate declarative jobsets for a project's GitHub repository.
Rust
12
star
27

dhcpv6macd

A DHCPv6 service which assigns IPv6 addresses strictly on MAC address only. Potentially violates RFCs and specs.
Nix
11
star
28

nixos-example-openstack

An example of building and pushing images for Openstack.
Nix
10
star
29

apple-sdks.nix

Experimental extraction of Apple SDKs.
Nix
9
star
30

nix-cpio-generator

Rust
9
star
31

hydra-examples

Example files for Hydra.
Nix
8
star
32

cpiotools

Tools for examining CPIOs.
Rust
7
star
33

prometheus-weather-gov

An example Nix project using Python, with a NixOS Module, NixOS test, and Docker image.
Python
7
star
34

hydra-nixos-org-configuration

A mirror of the project and jobset configurations on hydra.nixos.org.
HCL
6
star
35

nix

Tracks the fallback paths for Nix, to allow upgrades on our schedule.
Nix
6
star
36

nix-config-parser

Rust
6
star
37

asset-tagger

Print asset tags.
Shell
4
star
38

inspect

Inspect a flake's outputs with flake schemas.
Nix
4
star
39

search-detsys-dev

Python
4
star
40

flakehub-mirror

TypeScript
4
star
41

templates

Ready-made flake templates to get started with Nix
Nix
3
star
42

export

Export arbitrary bytes from Rust to various shells and interpreters.
Rust
3
star
43

elixir-nix

Nix
2
star
44

coldsnap.nix

Nix
2
star
45

hydra-scale-equinix-metal

Scale Equinix Metal builders based on Hydra usage.
Rust
2
star
46

scanoss-nix

Packages for the ScanOSS tool set.
Nix
2
star
47

detsys-ts

TypeScript
2
star
48

hydra-runcommand-logs

Experiments logging to the journal from RunCommand.
Nix
2
star
49

ci

Determinate CI is the one-stop shop for effortless Nix CI in GitHub Actions.
2
star
50

flakehub-docs

FlakeHub documentation
MDX
2
star
51

install-riff-action

A GitHub Action for installing Riff
Shell
1
star
52

homebrew-riff

A Homebrew formula for Riff
Ruby
1
star
53

karonte-nix

Python
1
star
54

github-actions-oidc-claims

Rust
1
star
55

nix-installer-example

Nix
1
star
56

linear-import-nix

Nix
1
star
57

pickup.nix

Nix
1
star
58

hydra-github-jobsets-example-project

1
star
59

hydra-test-migrate-to-s3

Test migrating a Hydra instance from a local cache to an S3-backed cache.
Nix
1
star
60

buildkite-install-nix-macos

A Buildkite workflow to install Nix on macOS
Shell
1
star
61

update-flake-lock-test-template

A template repository for testing changes to https://github.com/DeterminateSystems/update-flake-lock/
Nix
1
star
62

marp-nix

Nix
1
star
63

fh-init-example-project

Python
1
star
64

.github

Determinate Systems
1
star
65

ipxe-boot-lab

Local lab for testing iPXE boot in user network namespaces.
Rust
1
star
66

mlnx-workspace

Nix
1
star
67

store-paths

Nix
1
star