Riff
Riff is a tool that automatically provides external dependencies1 for software projects. To enter a shell environment with all your project's external dependencies installed, run this at the project root:
riff shell
You can also directly run commands with the shell environment applied but without entering the shell:
riff run cargo build
Riff currently supports Rust with support for other languages coming soon. It uses the Nix package manager to handle dependencies but doesn't require you to know or use Nix.
For a video demo of Riff in action, see below.
Requirements
To use Riff, you need to install these binaries on your system:
Installation
Using Nix
To install Riff using Nix (make sure to have flakes enabled):
nix profile install github:DeterminateSystems/riff
Using Homebrew
To install Riff on macOS using Homebrew:
brew install DeterminateSystems/riff/riff
Note: The
riff
Homebrew formula does not install Nix or Cargo.
Using cURL
You can find instructions for installing Riff using cURL on the releases page.
GitHub Actions
You can install Riff in your GitHub Actions pipelines using
install-riff-action
. Here's an example configuration:
steps:
- uses: actions/checkout@v3
- name: Install Nix
uses: DeterminateSystems/nix-installer-action@main
- name: Install Rust
uses: actions-rs/toolchain@v1
with:
toolchain: stable
- name: Install Riff
uses: DeterminateSystems/install-riff-action@v1
- name: Build Rust app
run: riff run cargo build -- --release
Prompt Customization
You can customize your shell's prompt to display when you're in a Riff shell
environment by relying on the $IN_NIX_SHELL
environment variable.
If you use Starship, you get this information for free because the Nix shell
module is enabled by default. However, if you want to add this to your shell
prompt yourself, you can do that by adding $name
to your prompt when
$IN_NIX_SHELL
is set.
In Bash, this might look something like:
export PS1="$PWD \${IN_NIX_SHELL:+\$name }\$ "
Note the escaping of
\${IN_NIX_SHELL}
and\$name
. This prevents Bash from taking the current values of those environment variables and using them even after one of them has changed.
In Zsh, it might look similar to:
export PROMPT="$PWD \${IN_NIX_SHELL:+\$name }\$ "
And in Fish, you might use something like:
function fish_prompt
echo -n "$PWD "
if set -q IN_NIX_SHELL
echo -n "$name "
end
echo -n "\$ "
end
Escaping
$name
is unnecessary here because Fish doesn't capture the value of$name
until the function is run when your prompt is displayed.
The idea extends to any shell: if you can change the prompt and prevent it from
evaluating the values of $IN_NIX_SHELL
and $name
until the prompt is
displayed, you can add this information to your prompt.
What Riff provides
Most programming languages use language-specific package managers to handle
dependencies, such as Cargo for the Rust language. But these
language-specific tools typically don't handle dependencies written in other
languages very well. They expect you to install those dependencies using some
other tool and fail in mysterious ways when they're missing. Here's an
example error from trying to build the octocrab
crate without
OpenSSL installed:
--- stderr
thread 'main' panicked at '
Could not find directory of OpenSSL installation, and this `-sys` crate cannot
proceed without this knowledge. If OpenSSL is installed and this crate had
trouble finding it, you can set the `OPENSSL_DIR` environment variable for the
compilation process.
Make sure you also have the development packages of openssl installed.
For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.
In cases like this, it's up to you to install missing external dependencies, which can be laborious, error prone, and hard to reproduce.
Riff enables you to bypass this problem entirely. It uses your your project's
language-specific configuration to infer which external dependencies are
required and creates a shell environment with those dependencies both installed
and properly linked. In cases where those dependencies can't be inferred, for
example in your build.rs
script, you can explicitly declare
them in your Cargo.toml
.
These environments are transient in the sense that they don't affect anything outside the shell; they install dependencies neither globally nor in your current project, so you don't have to worry about Riff breaking anything on your system. When you exit the Riff shell, the dependencies are gone.
Offline mode
In cases where you want to limit Riff's access to the Internet, you can run it
in offline mode, which disables all network usage except what's required by
the nix develop
command (which Riff runs in the background). You can enable
offline mode using either the --offline
flag or the RIFF_OFFLINE
environment
variable. Here are some examples:
# Via flag
riff run --offline
# Via environment variable
RIFF_OFFLINE=true riff shell
Example usage
In this example, we'll build the Prost project from source. Prost has an
external dependency on OpenSSL, without which commands like cargo build
and cargo run
are doomed to fail. Riff provides those dependencies
automatically, without you needing to install them in your regular
environment. Follow these steps to see dependency inference in action:
git clone https://github.com/tokio-rs/prost.git
cd prost
# Enter the Riff shell environment
riff shell
# ✓ 🦀 rust: cargo, cmake, curl, openssl, pkg-config, rustc, rustfmt, zlib
# Check for the presence of openssl
which openssl
# The path should look like this:
# /nix/store/f3xbf94zykbh6drw6wfg9hdrfgwrkck7-openssl-1.1.1q-bin/bin/openssl
# This means that Riff is using the Nix-provided openssl
# Build the project
cargo build
# Leave the shell environment
exit
# Check for openssl again
which openssl
# This should either point to an openssl executable on your PATH or fail
How to declare package inputs
While Riff does its best to infer external dependencies from your project's
crate dependencies, you can explicitly declare external dependencies if
necessary by adding a riff
block to the package.metadata
block in your
Cargo.toml
. Riff currently supports three types of inputs:
build-inputs
are external dependencies that some crates may need to link against.environment-variables
are environment variables you want to set in your dev shell.runtime-inputs
are libraries you want to add to yourLD_LIBRARY_PATH
to ensure that your dev shell works as expected.
Both build-inputs
and runtime-inputs
can be any packages available in
Nixpkgs. You may find this particularly useful for build.rs
scripts.
Here's an example Cargo.toml
with an explicitly supplied Riff configuration:
[package]
name = "riff-example"
version = "0.1.0"
edition = "2021"
[package.metadata.riff]
build-inputs = [ "openssl" ]
runtime-inputs = [ "libGL" ]
[package.metadata.riff.environment-variables]
HI = "BYE"
# Other configuration
When you run riff shell
in this project, Riff
- adds OpenSSL to your build environment
- sets the
LD_LIBRARY_PATH
environment variable to include libGL's library path - sets the
HI
environment variable to have a value ofBYE
Target-specific dependencies
If a project has OS-, architecture-, or vendor-specific dependencies, you can
define them in a targets
block under package.metadata.riff
. Here's an
example for Apple M1 (aarch64-apple-darwin
) systems:
[package.metadata.riff.targets.aarch64-apple-darwin]
build-inputs = [
"darwin.apple_sdk.frameworks.CoreServices",
"darwin.apple_sdk.frameworks.Security"
]
The Rust project maintains a list of well-known targets
that you can view by running nix run nixpkgs#rustup target list
. This
field can also contain custom targets, such as riscv32imac-unknown-xous-elf
,
although riff
makes no effort to support cross compiling at this time.
When target-specific dependencies are present, the build-inputs
and
runtime-inputs
sections are unioned (joined), while the target-specific
environment variables override default environment variables.
macOS framework dependencies
macOS users may encounter issues with so-called "framework" dependencies, such
as Foundation
, CoreServices
, and
Security
. When these dependencies are missing, you may see error
messages like this:
= note: ld: framework not found CoreFoundation
You can solve this by adding framework dependencies to your build-inputs
as
darwin.apple_sdk.frameworks.<framework>
, for example
darwin.apple_sdk.frameworks.Security
. Here's an example Cargo.toml
configuration that adds multiple framework dependencies:
[package.metadata.riff.targets.x86_64-apple-darwin]
build-inputs = [
"darwin.apple_sdk.frameworks.CoreServices",
"darwin.apple_sdk.frameworks.Security"
]
[package.metadata.riff.targets.aarch64-apple-darwin]
build-inputs = [
"darwin.apple_sdk.frameworks.CoreServices",
"darwin.apple_sdk.frameworks.Security"
]
Riff understands dependencies transitively
If you add Riff metadata to Cargo.toml
, this
doesn't just make it easier to build and run your project: it actually benefits
consumers of your crate as well. That's because Riff can use this metadata
transitively to infer which external dependencies are necessary across the
entire crate dependency graph. Let's say that you release a crate called
make-it-pretty
that has an external dependency on libGL and you add that
to your Cargo.toml
:
[package.metadata.riff]
runtime-inputs = [ "libGL" ]
Now let's say that another Rust dev releases a crate called artify
that
depends on your make-it-pretty
crate. If someone tries to build artify
using
Cargo, they may receive an error if they don't have libGL installed. But if
they use Riff to build artify
, Riff knows to install libGL without any user
input.
The implication is that adding Riff metadata to your crates—if they have external dependencies—can benefit the Rust ecosystem more broadly.
How it works
When you run riff shell
in a Rust project, Riff
- reads your
Cargo.toml
configuration manifest to determine which external dependencies your project requires and then - uses the Nix package manager—in the background and without requiring any intervention on your part—to install any external dependencies, such as OpenSSL or Protobuf, and also sets any environment variables necessary to discover those tools. Once it knows which external tools are required, it
- builds a custom shell environment that enables you to use commands like
cargo build
andcargo run
without encountering the missing dependency errors that so often dog Rust development.
This diagram provides a basic visual description of that process:
Because Riff uses Nix, all of the dependencies that it installs are stored in
your local Nix store, by default under /nix/store
. For a more thorough
breakdown of how Riff works, see the architecture docs.
Video demo
You can see a video demo of Riff in action here (click on the image for a larger version):
In the video, running cargo build
in the Prost project fails due to missing
external dependencies. But running riff run cargo build
succeeds because Riff
is able to infer which external dependencies are missing and provide them in the
background using Nix.
Garbage Collection
One of the disadvantages of Riff using Nix is that Nix can end up using
substantial disk space. After using Riff on a few projects, you may notice the
/nix/store
directory grow in size. Luckily, it is relatively easy to reclaim
this space by running nix-collect-garbage
. Note that garbage collection may
clean up any Riff shells you have used, which in turn may cause your next riff
invocation to take slightly longer because Riff—using Nix—needs to
fetch the dependencies that were cleaned up.
Direnv Integration
You can add Riff support to Direnv on a project-specific or global basis. To
enable Riff in a project, create a .envrc
file that contains this:
# reload when these files change
watch_file Cargo.toml Cargo.lock
# add any other files you might want to trigger a riff reload
# load the riff dev env
eval "$(riff print-dev-env)"
You can enable Riff support globally by either adding a use_riff
function
either to your ~/.config/direnv/direnvrc
file or a new
~/.config/direnv/lib/riff.sh
file. The use_riff
function should look
something like this:
use_riff() {
watch_file Cargo.toml watch_file Cargo.lock
eval "$(riff print-dev-env)"
}
With Direnv now aware of this function, you can enable Riff in any directory with:
echo "use riff" > .envrc
When you run direnv allow
you will automatically enter the Riff shell every
time you navigate to the project directory.
Privacy policy
For the sake of improving user experience, Riff does collect some telemetry. You can read the full privacy policy for Determinate Systems, the creators of Riff, here.
To disable telemetry on any Riff command invocation, you can either
- Use the
--disable-telemetry
flag or - Set the
RIFF_DISABLE_TELEMETRY
environment variable to any value exceptfalse
,0
, or an empty string (""
).
Here are some examples:
# Via flag
riff shell --disable-telemetry
# Via environment variable
RIFF_DISABLE_TELEMETRY=true riff run cargo build
Telemetry
When you use Riff, Riff generates a random version 4 UUID for you. It contains no personally identifiable information about you and is used to know how many people use the tool and to focus our limited research and development. You can delete this file at any time to create a new ID.
To see exactly what data we send, you may run Riff as follows:
$ RUST_LOG=riff::telemetry=debug riff run echo 'Hello, Riff!'
✓ 🦀 rust: cargo, openssl, pkg-config, rustc, rustfmt
2022-09-29T21:29:01.476342Z DEBUG riff::telemetry: Sent telemetry data to https://registry.riff.determinate.systems/telemetry, telemetry: Telemetry { distinct_id: Some(Secret([REDACTED riff::telemetry::DistinctId])), system_os: "linux", system_arch: "x86_64", os_release_name: Some("NixOS"), os_release_version_id: Some("22.11"), riff_version: "1.0.1", nix_version: Some("nix (Nix) 2.12.0pre20220928_c3c0682"), is_tty: true, subcommand: Some("run"), detected_languages: {Rust}, in_ci: false }
at src/telemetry.rs:129
in riff::telemetry::send
in riff::flake_generator::generate_flake_from_project_dir with project_dir: None, offline: false
This will also appear when running with
--debug
, accompanied by other debug logging.
The table below shows the data Riff collects in a more readable format:
Field | Use |
---|---|
distinct_id |
The ID of the Riff installation. More specifically, we at Determinate Systems care about our weekly active users count and this field helps us measure that. |
system_os |
The operating system Riff is running on. |
system_arch |
The architecture Riff is running on. |
os_release_name |
The distribution Riff is running on. This uses the NAME field of /etc/os-release if it exists. |
os_release_version_id |
The version of the distribution Riff is running on. This uses the VERSION_ID field of /etc/os-release if it exists. |
riff_version |
The version of Riff being used. |
nix_version |
The version of Nix being used by Riff. |
is_tty |
Whether Riff is being run interactively. |
subcommand |
The subcommand Riff is executing. This only contains information about the Riff subcommand, and not any commands being run by Riff (i.e. riff run echo 'Hello, Riff!' will not send any telemetry including the fact that Riff ran echo 'Hello, Riff!' ). |
detected_languages |
Which languages Riff detected in the project. |
in_ci |
Whether Riff is being used in CI (e.g. GitHub Actions). |
Community
If you'd like to discuss Riff with other users, join our Discord (also bridged to a Matrix room).
Footnotes
-
We define external dependencies as those that are written in another language and thus can't be installed using the same language-specific package manager that you use to build your code.
↩