nixfiles

My NixOS configuration and assorted other crap, powered by flakes. Clone to /etc/nixos.

CI checks ensure that code is formatted and passes linting. Run those locally with:

nix flake check
nix run .#fmt
nix run .#lint

Overview

This is an opinionated config making assumptions which work for me but might not for you:

These are primarily single-user hosts, with me being that user. While security and availability are important, convenience takes priority.
Observability is good but there's no central graphing or alerting stack, every host has to run their own.
Databases should not be shared, each service has its own containerised instance. This means a single host may run several instances of the same database software, but that's an acceptable overhead.
Persistent docker volumes should be backed by bind-mounts to the filesystem.
For ZFS systems, wiping / on boot is good actually.

Everything in shared/default.nix is enabled on every host by default. Notable decisions are:

Every user gets a ~/tmp directory with files cleaned out after 7 days.
Automatic upgrades (including reboots if needed), automatic deletions of generations older than 30 days, and automatic garbage collection are all enabled.
Locale, timezone, and keyboard layout all set to UK / GB values (yes, even on servers).
Firewall and fail2ban are enabled, but pings are explicitly allowed.
SSH accepts pubkey auth only: no passwords.
Syncthing is enabled.

For monitoring and alerting specifically:

Prometheus, Grafana, and Alertmanager are all enabled by default (Alertmanager needs AWS credentials provided to actually send alerts).
The Node Exporter is enabled, along with a dashboard.
cAdvisor is enabled, along with a dashboard.

If using ZFS there are a few more things configured:

All pools are scrubbed monthly.
The auto-trim and auto-snapshot jobs are enabled (for pools which have those configured).
There's a Prometheus alert for pools in a state other than "online".

Everything else in shared/ is available to every host, but disabled by default.

Hosts

Currently I have 3 NixOS machines. The naming convention is:

Local machines: beings (gods, people, etc) of the Cthulhu Mythos.
Remote machines: places of the Cthulhu Mythos.

azathoth

This is my desktop computer.

It dual-boots Windows and NixOS, so it doesn’t run any services, as they won't be accessible half of the time. I don't bother backing up either OS: everything I care about is in Syncthing, on GitHub, or on some other cloud service (eg, Steam).

carcosa

This is a VPS (hosted by Hetzner Cloud).

It serves barrucadu.co.uk and other services on it, such as a bookdb instance and my blog. Websites are served with Caddy, with certs from Let's Encrypt.

It's set up in "erase your darlings" style, so most of the filesystem is wiped on boot and restored from the configuration, to ensure there's no accidentally unmanaged configuration or state hanging around. However, it doesn't reboot automatically, because I also use this server for a persistent IRC connection.

nyarlathotep

This is my home server.

It runs writable instances of the bookdb and bookmarks services, which have any updates copied across to carcosa hourly; it acts as a NAS; and it runs a few utility services, such as a dashboard of finance information from my hledger journal, and a script to automatically tag and organise new podcast episodes or CD rips which I copy over to it.

Like carcosa, this host is set up in "erase your darlings" style but, unlike carcosa, it automatically reboots to install updates: so that takes effect significantly more frequently.

Setting up a new host

Create & format partitions
Optional: Configure wiping / on boot (pre-first-boot steps)
Install NixOS with the standard installer
Reboot into the installed system
Clone this repo to /etc/nixos
Move the generated configuration to hosts/<hostname>/ and edit to fit repo conventions
Add an entry for the host to flake.nix
Optional: Add DNS records
Optional: Configure secrets
Optional: Configure wiping / on boot (post-first-boot steps)
Optional: Configure backups
Optional: Generate SSH key
Build the new system configuration with sudo nixos-rebuild switch --flake '.#<hostname>'
Reboot
Commit, push, & merge

Optional: Configure wiping / on boot

Before installing NixOS, create the local pool and datasets:

zpool create -o mountpoint=legacy -o autotrim=on local <device>

zfs create -o mountpoint=legacy local/volatile
zfs create -o mountpoint=legacy local/volatile/root

zfs create -o mountpoint=legacy local/persistent
zfs create -o mountpoint=legacy -o com.sun:auto-snapshot=true local/persistent/home
zfs create -o mountpoint=legacy -o com.sun:auto-snapshot=true local/persistent/nix
zfs create -o mountpoint=legacy -o com.sun:auto-snapshot=true local/persistent/persist
zfs create -o mountpoint=legacy -o com.sun:auto-snapshot=true local/persistent/var-log

Take a snapshot of the empty root dataset:

zfs snapshot local/volatile/root@blank

Mount all the filesystems under /mnt:

mount -t zfs local/volatile/root /mnt

mkdir /mnt/boot
mkdir /mnt/home
mkdir /mnt/nix
mkdir /mnt/persist
mkdir -p /mnt/var/log

mount /dev/<boot device> /mnt/boot
mount -t zfs local/persistent/home /mnt/home
mount -t zfs local/persistent/nix /mnt/nix
mount -t zfs local/persistent/persist /mnt/persist
mount -t zfs local/persistent/var-log /mnt/var/log

Then run the installer, making sure to add ZFS details to the generated configuration:

networking.hostId = "<random 32-bit hex value>";
boot.supportedFilesystems = [ "zfs" ];

After first boot: copy any needed files (eg, SSH host keys) to the appropriate place in /persist, add the user password to the secrets, and set up nixfiles.eraseYourDarlings:

nixfiles.eraseYourDarlings.enable = true;
nixfiles.eraseYourDarlings.machineId = "<contents of /etc/machine-id>";
nixfiles.eraseYourDarlings.barrucaduPasswordFile = config.sops.secrets."users/barrucadu".path;
sops.secrets."users/barrucadu".neededForUsers = true;

Optional: Add DNS records

Add A / AAAA records to the ops repo and apply the change via Concourse.

Optional: Configure secrets

After first boot, generate an age public key from the host SSH key:

nix-shell -p ssh-to-age --run 'ssh-keyscan <hostname>.barrucadu.co.uk | ssh-to-age'

Add a new section with this key to .sops.yaml:

creation_rules:
  ...
  - path_regex: hosts/<hostname>/secrets(/[^/]+)?\.yaml$
    key_groups:
      - age:
          - *barrucadu
          - '<key>'

Enable sops in the host configuration:

sops.defaultSopsFile = ./secrets.yaml;

Optional: Configure backups

All hosts which run any sort of service with data I care about should take automatic backups.

Firstly, add the backup credentials to the secrets:

nix run .#secrets

Then enable backups in the host configuration:

nixfiles.backups.enable = true;
nixfiles.backups.environmentFile = config.sops.secrets."nixfiles/backups/env".path;
sops.secrets."nixfiles/backups/env" = { };

Most services define their own backup scripts. For any other needs, write a custom script:

nixfiles.backups.scripts.<name> = ''
  <script which copies files to backup to the current working directory>
'';

Optional: Generate SSH key

Generate an ed25519 SSH key:

ssh-keygen -t ed25519

If the host should be able to interact with GitHub: add the public key to the GitHub user configuration as an SSH key.

If the host should be able to push commits to GitHub: add the public key to the GitHub user configuration as a signing key, and also add it to the allowed_signers file.

If the host should be able to connect to other machines: add the public key to shared/default.nix.

Tools

Backups

Backups are managed by shared/backups and uploaded to S3 with Duplicity.

Check the status of a backup collection with:

nix run .#backups                   # for the current host
nix run .#backups status            # for the current host
nix run .#backups status <hostname> # for another host

Restore a backup to /tmp/backup-restore with:

nix run .#backups restore            # for the current host
nix run .#backups restore <hostname> # for another host

Change the restore target by setting $RESTORE_DIR.

Secrets

Secrets are managed with sops-nix. Create / edit secrets with:

nix run .#secrets                   # secrets.yaml for current host
nix run .#secrets <hostname>        # secrets.yaml for <hostname>
nix run .#secrets <hostname> <name> # <name>.yaml for <hostname>

barrucadu/nixfiles

barrucadu

Reviews

Repository Details