sbt-nix.g8
Get started with Nix and see how you can benefit from it in your Scala or cross-team projects.
- Quick start
- Motivation
- What is Nix?
- Install Nix
- Use cases
- Pinning Nixpkgs
- Caching sbt derivations
- Managing dependencies (jars) with Nix
- Get started with sbt-nix.g8
Quick start
If you're familiar with nix and have it installed, you can create a new project using this template:
nix-shell -p sbt --run "sbt new gvolpe/sbt-nix.g8"Then follow the instructions in the README file inside the generated project's directory.
Motivation
I started out writing this guide with examples after I ran a Twitter poll on September 2020.
The results speak for themselves but I am on a mission to change the current situation because I believe Nix and functional programming are the way forward. If you would like to help, share this guide with as many people as you can, give it a ⭐ and spread the good word!
What is Nix?
Quoting the official website:
A powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. Share your development and build environments across different machines.
Reproducibility is probably the best feature Nix has to offer. It is widely used in other ecosystems, such as Haskell, Python, Rust and Go, but barely used (and known) in the Scala community.
Install Nix
You are only a command away (unless you are on NixOS), as officially documented.
curl -L https://nixos.org/nix/install | sh
Use cases
There are three clear use cases where I think Nix can make a difference in Scala projects.
Reproducible development shell
All the project's dependencies are declared in a shell.nix file. For example, jdk, sbt and coursier (maybe also jekyll, if a microsite depends on it).
{ jdk ? "jdk11" }:
let
pkgs = import ./pkgs.nix { inherit jdk; };
in
pkgs.mkShell {
buildInputs = [
pkgs.coursier
pkgs.${jdk}
pkgs.sbt
];
}Where pkgs.nix defines an exact version of the Nixpkgs (more on this later). The important part is that every member of the team will have access to the exact same packages.
Avoid global installation of Java, Sbt & any other binary
Instead of installing binaries from the web, let Nix manage your dependencies. This is crucial when working in big teams. We will no longer hear "it compiles on my machine".
The benefit is even greater when you work on diverse teams where everybody shares the same shell.nix to run the full application. For example, at work we declare all the dependencies for frontend, backend and infrastructure such as jdk, nodejs and kubectl, among others.
It is very appealing for new members joining the team! On day one, all they need to do is to git-clone the project, install Nix, run nix-shell and all the project's dependencies will become available. Isn't that great?
We can also use nix-direnv so that we don't even need to run nix-shell every time. Upon entering a working directory with a shell.nix, all the declared software will become automatically available. Feels like magic! Actually, this is what we promote using at work.
Reproducible CI builds
We can use the exact same dependencies declared in shell.nix on the CI build. No more discrepancies.
Have a look at .github/workflows/ci.yml to see how it looks like. It couldn't be simpler!
In fairness, we use nix/ci.nix to run the CI build, which is a version of shell.nix that only contains the sbt package. The idea is that we can keep adding packages to our shell.nix for local development, and there might be stuff we don't need at all in the CI, so it will run faster with less dependencies to pull.
Reproducible (and smaller) Docker images
Nowadays, most Scala projects are deployed as a Docker image (sometimes using Kubernetes). Although there are tools such as sbt-native-packager, we can again declare what dependencies make it to our Docker image. There are some immediate benefits in doing this:
- we get to use the exact same JDK / JRE we declare in our Nix file.
- we will more likely get a smaller image than using a base slim one from Docker Hub.
- we will have a reproducible image (no more
apt-get updates, please!).
We can still use sbt-native-packaer to create our Docker images, as demonstrated in the examples below. Another option is to use sbt-assembly and some declarative definition of our Dockerfile.
Note: the three examples shown below can be found under the modules folder. Feel free to clone the repo and play around with them.
Default Docker image using sbt-native-packager
This module shows how users normally use sbt-native-packager to create Docker images. It uses openjdk:11.0.8-jre-slim as the base Docker image.
sbt "sbt-nix-native-default/docker:publishLocal"
docker run -it sbt-nix-bootstrap-default:0.1.0-SNAPSHOT
Here's the resulting Docker image, including the jar dependencies declared in our build.sbt.
REPOSITORY TAG IMAGE ID CREATED SIZE
sbt-nix-bootstrap-default 0.1.0-SNAPSHOT c0320ed1b643 2 minutes ago 221MB
This is the default and it doesn't use Nix at all.
Custom Nix Docker image using sbt-native-packager
We could benefit from Nix by creating a base Docker image using the exact same jre we declare in our project, and still use sbt-native-packager to make the final image.
{ imgName ? "base-jre"
, jdk ? "jdk11"
, jre ? "adoptopenjdk-jre-openj9-bin-11"
}:
let
pkgs = import ./pkgs.nix { inherit jdk; };
in
pkgs.dockerTools.buildLayeredImage {
name = imgName;
tag = "latest";
contents = [ pkgs.${jre} ];
}Note: We are using adoptopenjdk-jre-openj9-bin-11 here whereas for the default image we use openjdk:11.0.8-jre-slim but really, I couldn't find an image for the same JRE on Docker Hub, only for the JDK (if you do please let me know to update the document).
Run it as follows:
nix-build nix/docker.nix -o result-base-jre
docker load -i result-base-jre
sbt "sbt-nix-native-custom/docker:publishLocal"
docker run -it sbt-nix-bootstrap-custom:0.1.0-SNAPSHOT
The resulting Docker image is 42MB smaller than the default! (with the caveat of the JREs not being the same).
REPOSITORY TAG IMAGE ID CREATED SIZE
sbt-nix-bootstrap-custom 0.1.0-SNAPSHOT 94e713b3fa0d 6 seconds ago 179MB
base-jre latest 58028d3adc50 50 years ago 163MB
Note: base-jre shows it was created 50 years ago but we can change that by adding created = "now" to our image definition. However, by doing so, we would be breaking binary reproducibility.
Learn more about creating Docker images with Nix at https://nixos.org/manual/nixpkgs/stable/#sec-pkgs-dockerTools
Custom Nix Docker image using sbt-assembly
Although a bit more manual, this approach also works using other build tools such as Mill, which natively provides an assembly command to create a fat jar.
First of all, we need a basic Dockerfile.
FROM base-jre:latest
COPY app.jar /app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
Then we will create a shell-script with Nix.
{ imgName ? "sbt-nix-assembly"
, jdk ? "jdk11"
, jre ? "adoptopenjdk-jre-openj9-bin-11"
}:
let
pkgs = import ../../../nix/pkgs.nix { inherit jdk; };
base = pkgs.callPackage ../../../nix/docker.nix { inherit jdk jre pkgs; };
in
pkgs.writeShellScriptBin "build" ''
cd ../../
${pkgs.sbt}/bin/sbt "sbt-nix-assembly/assembly"
cd modules/assembly/
docker load -i ${base}
cp target/scala-2.13/app.jar nix/app.jar
docker build -t ${imgName} nix/
rm nix/app.jar
''The result will be a shell-script that will run the assembly command and build our Docker image. Under modules/assembly, you will find a build.sh script that does it all but you could as well do it manually since it only runs two commands.
#! /usr/bin/env bash
nix-build nix/app.nix
result/bin/buildYou can run it as follows:
cd modules/assembly/ && ./build.sh
docker run -it sbt-nix-assembly:latestDocker images
To recap, here are all the different Docker images shown in the examples, for an easy comparison.
REPOSITORY TAG IMAGE ID CREATED SIZE
sbt-nix-bootstrap-custom 0.1.0-SNAPSHOT 94e713b3fa0d 6 seconds ago 179MB
sbt-nix-bootstrap-default 0.1.0-SNAPSHOT c0320ed1b643 2 minutes ago 221MB
sbt-nix-assembly latest 61b30afca2fc 9 minutes ago 179MB
base-jre latest 58028d3adc50 50 years ago 163MB
Pinning Nixpkgs
In previous Nix files, we were referencing a file named pkgs.nix. This is where we define what version of Nixpkgs we want to use in our project.
Whenever you install Nix, you'll have something called channels. Using Channels is not recommended because it goes against reproducible builds but they are useful to try things out with Nix. So instead of using a channel, we will "pin" the Nixpkgs to a specific version, indicated by a URL and a SHA256 hash. In a nutshell, it looks as follows:
{ jdk }:
let
nixpkgs = fetchTarball {
name = "nixos-unstable-2020-09-25";
url = "https://github.com/NixOS/nixpkgs-channels/archive/72b9660dc18b.tar.gz";
sha256 = "1cqgpw263bz261bgz34j6hiawi4hi6smwp6981yz375fx0g6kmss";
};
config = {
packageOverrides = p: {
sbt = p.sbt.override {
jre = p.${jdk};
};
};
};
pkgs = import nixpkgs { inherit config; };
in
pkgsIn practice, though,
nixpkgsis defined atnix/pinned.nixandconfigis defined atnix/config.nixfor modularity.
Let's explain what's going on here.
fetchTarballis one of the many built-in functions.urlwill always be the same, except for the last part. That72b9660dc18bis a commit hash. To find out the latest, you can look here.sha256is calculated from thetar.gzfile. You can runnix-prefetch-url --unpack [URL]to get it.
We also have a config that overrides the sbt package to use the jdk version given as an argument. sbt comes with a default jdk version by default.
Overall, we can say our pkgs.nix defines a function that expects a jdk argument . This is how we've seen it used before.
{ jdk ? "jdk11" }:
import ./pkgs.nix { inherit jdk; }It means that if no other value is given, we will use jdk11 by default.
Using sbt with a different JDK
Since our shell.nix defines an argument with a default value:
{ jdk ? "jdk11" }:We will always use jdk11 when running nix-shell. In order to change that, we can supply the argument as follows:
nix-shell --argstr jdk jdk14In fact, this is what we do in the CI build to get sbt to run our project with the desired JDK version.
Caching sbt derivations
Nix derivations normally result in a binary, and sbt is not an exception. Since we override the default JDK version, every binary result is different, and so it can be cached so you don't have to build it again (and neither does the CI build). We can build the derivation via nix-build (the binary will be available by default under result/bin/sbt).
> nix-build nix/sbt.nix
/nix/store/xyv3z73pfzkl390wk0bd68r8j24lvh4r-sbt-1.3.13
> nix-build nix/sbt.nix --argstr jdk jdk8
/nix/store/kw8q95qax89cm8v9g1n6vxqag7hjaycy-sbt-1.3.13
> nix-build nix/sbt.nix --argstr jdk jdk14
/nix/store/fiqmv96y4m9bmyaw84wp7jpx2i8kkwgy-sbt-1.3.13Notice how every hash is different when using different JDKs.
There is a free service for open-source projects named Cachix, including support for Github actions, where we can upload our binaries (result of a Nix derivation) so then everyone else can benefit from not having to build it again.
Pushing a binary is fairly easy. Once you signed up and generate your signing key, you can pipe the output of a nix-build derivation directly to Cachix.
nix-build nix/sbt.nix | cachix push mycacheTo use the binary cache, you need to run cachix use mycache. We are also using Cachix in our CI build.
name: "Install Cachix ❄️"
uses: cachix/cachix-action@v6
with:
name: neutron
signingKey: "${{ secrets.CACHIX_SIGNING_KEY }}"You can do the same, just make sure you change neutron for the name of your cache (creating one is free).
We have seen how the sbt binary can be cached, though, this applies to any other binary. So next time you come across a derivation that results in a binary, know that you can cache it so your peers don't have to re-build it on their machines, and neither does the CI build!
Managing dependencies (jars) with Nix
There were a few attempts to go full Nix:
Unfortunately, both projects seem abandoned. It is also worth noticing that these projects are very ambitious and Nixifying an entire Scala project is not a trivial task.
sbt-derivation
There is another active project named sbt-derivation, which is not as ambitious as the others but it does what it promises. It basically creates two derivations: one for all the jar dependencies and another one for the project. The former derivation is identified by a depsHash256, so if we add a new dependency, the hash will change and it will fail the build if we forget to update the hash.
The project under modules/nixified showcases the usage of sbt-derivation with sbt-assembly, the default building mechanism. The project is defined as follows:
{ jdk ? "jdk11" }:
let
pinned = import nix/pinned.nix;
config = import nix/config.nix { inherit jdk; };
sbtix = import pinned.sbt-derivation;
pkgs = import pinned.nixpkgs {
inherit config;
overlays = [ sbtix ];
};
in
pkgs.sbt.mkDerivation {
pname = "sbt-nixified";
version = "1.0.0";
depsSha256 = "02xxc6fy73v1m2awmavca7lgyr06fhjyg3q2q08cxr6nmy1s4b23";
src = ./.;
buildPhase = ''
sbt "sbt-nix-derivation/assembly"
'';
installPhase = ''
cp modules/nixified/target/scala-*/*-assembly-*.jar $out
'';
}It can be found in the app.nix file, at the root of the repository. To build it and run it, use the following commands:
nix-build app.nix -o result-jar
java -jar result-jarThe wrapper.nix file defines a similar build for modules/wrapper but it uses sbt-native-packager and it creates a binary wrapper as the output instead of just creating a jar.
Get started with sbt-nix.g8
New Scala projects using sbt are usually created using g8 templates by running sbt new template.g8. However, a bit earlier it was recommended to not install sbt globally. So, if that's the case, how do we create a new project via sbt new? The answer is simple: nix-shell -p sbt. This command will start a new shell with the sbt package available. You can ask for more packages, if desired.
So to get started, this is all we need.
nix-shell -p sbt
sbt new gvolpe/sbt-nix.g8You can actually do it in a single command.
nix-shell -p sbt --run "sbt new gvolpe/sbt-nix.g8"Once we have created the project, follow the instructions in the README file to continue.
Note: the default template follows the approach demonstrated with the example sbt-nix-native-custom (recommended) as well as using sbt-derivation to build a binary as an alternative, but you can also checkout this repository and play around with the different examples.