• Stars
    star
    106
  • Rank 325,811 (Top 7 %)
  • Language
    Haskell
  • License
    MIT License
  • Created almost 5 years ago
  • Updated almost 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Exploring the options for incremental Haskell builds using Nix

ghc-nix

demo

ghc-nix is an attempt to augment GHC with the ability to use Nix as a caching layer. This has the potential to dramatically speed up "pure Nix" builds by being able to re-use build artefacts for files that haven't changed. It also has the potential to speed up developers collaborating on work, by allowing them to pull down pre-built artefacts from a shared build server, rather than building locally.

What's the Plan?

When you do cabal build, Cabal calls out to GHC with a single exec call, primarily to ghc --make. GHC actually has a build system built in, and GHC will form a dependency graph and start doing minimal recompilation where possible. However, this is only useful if you have some kind of reusable directory to keep the cache artefacts - and that doesn't exist on build machines like Hydra.

The plan then is to replace ghc --make with something that can use the Nix store. cabal build has the -w option to give it another compiler, but it will still call that executable with the same arguments. If we can impersonate GHC, then we can get the caching we desire - that's what this project does.

When ghc-nix is called with --make, we use the GHC API to form a dependency graph, as ghc --make would. However, rather than just doing the build, instead we transform this dependency graph into a series of Nix expressions, each of which will build a single .hs source file into .o and .hi files. Finally, we aggregate all the build artefacts and place the files where they need to go for Cabal to continue with the next phase (linking).

However, just doing Nix builds is not enough, we need one more experimental Nix feature to get something useful - content-addressable store paths. Usually, the objects in a Nix store are hashed based on all of the dependencies that are required to produce the store path. This ultimately comes down to an exact hash of all the source code. However, this means that if you make object-file-preserving changes, you still end up recompiling everything downstream. For example, a single comment change causes all dependents to rebuild, even though they can't observe a change.

Once we've built the .o and .hi files, we rewrite the resulting store path to one who's hash is based on the contents of these files, and not the the files that built them. This cuts the connection between an object file and its source code, allowing early cut off.

This is supported as an experimental Nix feature - nix make-content-addressable.

Trying this Out

This project is still in very early days, so it's not too easy to use... yet. Maybe you can help here! To try ghc-nix out you will need:

Nix changes

  • A version of Nix that is at least 4511f09b490fad4ce0dcfbcd7c4fd83b11e7df46.

  • nix-command and ca-references enabled as experimental features.

Heres's a snippet of my configuration.nix that should get you this.

{
  systemFeatures =
    [ "benchmark" "big-parallel" "kvm" "nixos-test" "recursive-nix" "nix-command" "ca-references" ];

  extraOptions =
    ''
    experimental-features = recursive-nix nix-command ca-references
    '';

  package =
    let
      src =
        pkgs.fetchFromGitHub
          { owner =
              "NixOS";

            repo =
              "nix";

            rev =
              "9f7b4d068cc106e5d902dc6f52bf46d4a057fd00";

            sha256 =
              "187pfanj0g49ng5smfi8rwkq1l3r43mf85yv390h0ars050fxfik";
          };

    in
    ( import "${src}/release.nix" { nix = src; officialRelease = true; } ).build.x86_64-linux;
}

Building ghc-nix

ghc-nix is built like a normal Haskell project, but you will have to actually use cabal install to have this work. I do this, in the ghc-nix directory:

cabal install --installdir=./bin --overwrite-policy=always

Using ghc-nix

You can now finally use ghc-nix by going to a Cabal project and running:

cabal build -w /path/to/ghc-nix/bin/ghc-nix

You will need a working ghc on your PATH, too (this will hopefully change in the future).

If you get problems with packages not being found and you're using Nix, you might need to run:

NIX_GHC=$(type -p ghc)
eval $(grep export "$NIX_GHC")

More Repositories

1

bauer

An Emacs+Nix IDE
Emacs Lisp
107
star
2

nixiosk

Declarative Kiosk systems built with NixOS
Nix
106
star
3

gametime-player

A super simple JS-based libretro frontend
CoffeeScript
40
star
4

unipkg

Create a package for different distros using a single PKGBUILD file (unmaintained)
Shell
31
star
5

undmg

extract a dmg file
C
26
star
6

nix-haskell-mode

Automatic Haskell setup in Emacs, for Nix users
Emacs Lisp
25
star
7

rivulet

a simple, fast, open source RSS reader
Go
21
star
8

reddwall

Change your background to a random wallpaper on Reddit.
Python
20
star
9

resume

my rΓ©sumΓ©
TeX
19
star
10

retrojs

Libretro emulator cores compiled into modular JavaScript.
JavaScript
17
star
11

eu4-parser

some tools to parse files from the clausewitz engine
Haskell
14
star
12

static-nix

Statically built Nix binaries
Nix
14
star
13

nixos-kiosk

NixOS kiosk using Cage
Nix
10
star
14

caldav-to-gtasks

A caldav server that writes directly to Google Tasks
Python
9
star
15

uzbl-os

A Uzbl based distro, similar to Google Chrome OS
Python
8
star
16

libmii

Wii homebrew library for Miis
C
7
star
17

macNixOS

a set of useful things for running Nix on macOS
Shell
4
star
18

x-retro

<x-retro></x-retro>
CoffeeScript
4
star
19

uzbl-extension-manager

A simple extension manager for Uzbl
Shell
4
star
20

metapkgs

A set of package sets
Nix
4
star
21

gpemu2

libretro frontend written in Haskell and Reflex
Haskell
3
star
22

make-package

Nix
3
star
23

counter

Counter app for the Pebble watch
C
3
star
24

xilium.cefglue

GitHub mirror of xilium.cefglue
C#
3
star
25

node-retro

THIS REPO IS DEPRECATED!!! Use retrojs instead.
C
2
star
26

matthewbauer.github.io

My personal site
Shell
2
star
27

auto-update-flake

GitHub Action to automatically update flake inputs if they pass CI
2
star
28

nur-packages

Nix
2
star
29

unipkg-pacman

C
2
star
30

denali

Denali Linux - NixOS for humans
Nix
2
star
31

todo.txt-gtk

Graphical User Interface for todo.txt in Python
Python
2
star
32

hbb-medawiki

Homebrew Browser Mediawiki extensions to allow repository editing from a wiki
PHP
1
star
33

65536

CSS
1
star
34

vector-algorithms

Haskell
1
star
35

skeleton

Haskell
1
star
36

xib2nib

C++
1
star
37

snes-info

command line tool to get information on the header of SNES .SMC files
Go
1
star
38

exdongle

provides 'AutoRun' capabilities to Linux systems (based on a script by Gogodidi)
1
star
39

hangman

Really hard hangman game in HTML and Javascript
JavaScript
1
star
40

dot-emacs

Emacs Lisp
1
star
41

jaycal

Automagically generate a calendar using your university account.
JavaScript
1
star