• Stars
    star
    172
  • Rank 211,855 (Top 5 %)
  • Language
    Go
  • License
    MIT License
  • Created over 1 year ago
  • Updated 7 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A simple and flexible build tool using Lua, similar to make/mk.

Knit 🧢

Test Workflow Go Reference Go Report Card MIT License

Knit is a build tool inspired by Make and Plan9 Mk. You define rules with a Make-like embedded syntax within a Lua program. Rules can be passed around as Lua objects, and generated by Lua code. You can use the Lua module system to make reusable modules for building any kind of source code. Knit combines the readability of a Make-style rules language will the power and expressiveness of Lua. If you are familiar with Make, you can learn Knit very quickly.

Knit tracks more of your build to give you better incremental builds. For example, Knit automatically adds an implicit dependency on a rule's recipe, so if you change a recipe (either directly or through a variable change), Knit will automatically re-run all rules that were affected.

Knit has support for namespaced sub-builds that execute relative to their directory, but Knit avoids build fragmentation because sub-builds don't rely on spawning build sub-processes. No more make -C to do sub-builds! Everything is tracked by the root Knitfile, but you can still make directory-specific rules.

Knit's rules language is heavily inspired by Plan9 Mk. In some ways, Knit can be considered a modern version of Mk with a Lua meta-programming system built on top of it (there are some differences compared to Mk).

Why make yet another build system? Because it's fun and useful to me! Maybe it will be useful to you too. Everyone hates something about their build system so if you have feedback or a request, let me know! The project is new enough that your feedback may be seriously taken into account.

I have written an article with more details about Knit here.

Features

  • Knit uses Lua for customization. This makes it possible to write reusable build libraries, and in general makes it easier to write powerful and expressive builds.
  • Knit has built-in syntax for a rules language inspired by Make and Plan9 Mk. This makes it very familiar to anyone who has used Make/Mk.
  • Knit has direct support for sub-builds (compared to Make, which usually involves spawning a separate make sub-process to perform a sub-build).
  • Knit can hash files to determine if they are out-of-date, rather than just relying on file modification times.
    • Knit additionally uses hashes for "dynamic task elision": if Knit can dynamically determine that a prerequisite that was rebuilt actually changed nothing, it won't re-run the dependent build step, allowing for even better incremental builds compared to timestamp-based approaches (Make, Ninja, etc.).
  • Knit tracks recipe changes, so if you update a variable (in the Knitfile or at the command-line), any dependent rules will be automatically rebuilt.
  • Knit supports % meta-rules and regular expression meta-rules.
  • Knit uses rule attributes instead of using targets such as .SECONDARY to indicate special processing.
  • Knit supports virtual rules that are independent of the file system.
  • Knit uses sane variable names like $input, $output, and $match instead of Make's $^, $@, and $*.
  • Knit supports rules with multiple outputs, and treats them like Make's group targets by default.
  • Knit supports sub-tools that implement various build utilities including:
    • Generating a graph visualization using graphviz (dot).
    • Showing build status information (whether targets are out-of-date and why).
    • Exporting a compile commands database for use with a language server.
    • Automatically cleaning all build outputs.
    • Converting your build into a shell script, Makefile, or Ninja file.
  • Knit will search up the directory hierarchy for a Knitfile, allowing you to run your build from anywhere in your project.
  • Knit supports parallel builds and uses all cores by default.
  • Cross-platform support (Windows support is still experimental).
    • Knit uses a shell to execute commands. By default, Knit searches for sh on your system and uses that. If it cannot find sh, it uses an internal (cross-platform) shell.

Example Knitfile

Here is a very basic Knitfile for building a simple C project.

return b{
    $ hello: hello.o
        cc -O2 $input -o $output
    $ %.o: %.c
        cc -O2 -c $input -o $output
}

The syntax for rules is nearly the same as Make, and Knit supports % meta-rules just like Make. However, rather than using a custom language to configure the build, Knit uses Lua.

Here is a more complex example Knitfile used for building a simple C project. This time the Knitfile supports various configurations (changing cc and enabling debug flags), and automatically detects the source files.

local knit = require("knit")

local conf = {
    cc = cli.cc or "gcc",
    debug = tobool(cli.debug) or false,
}

local cflags := -Wall

if conf.debug then
    cflags := $cflags -Og -g
else
    cflags := $cflags -O2
end

local src = knit.glob("*.c")
local obj = knit.extrepl(src, ".c", ".o")
local prog := hello

return b{
    $ build:VB: $prog

    $ $prog: $obj
        $(conf.cc) $cflags $input -o $output
    $ %.o:D[%.d]: %.c
        $(conf.cc) $cflags -MMD -c $input -o $output
}

Running knit hello would build all the necessary .o files and then link them together. Running knit hello debug=1 would change the flags and re-run the affected rules. Running knit build will build hello (effectively an alias for knit hello). The VB attributes on the build rule means that it is virtual (not referring to a file on the system), and should always be built (out-of-date).

Running knit -t clean will run a sub-tool that automatically removes all generated files.

Header dependencies are automatically handled by using the -MMD compiler flag with the D[%.d] attribute. To explicitly name the dependency file (e.g., to put it in a .dep folder), you could instead use:

$ %.o:D[.dep/%.dep]: %.c
    $(conf.cc) $cflags -MMD -MF $dep -c $input -o $output

Note that Knitfiles are Lua programs with some modified syntax: special syntax using $ for defining rules, and special syntax using := for defining raw strings (no quotes) with interpolation.

See the docs for more information.

See examples for a few examples, and see this repository's Knitfile and the tests for even more examples.

Installation

Prebuilt binaries are available from the release page.

You can install one automatically using eget.

eget zyedidia/knit

Or you can build from source (requires Go 1.19):

go install github.com/zyedidia/knit/cmd/knit@latest

Experimental or future possible features

  • Ninja to Knit converter (for compatibility with cmake, and for benchmarking). See knitja for the converter tool.
  • Performance optimizations.
    • Knit can already be used to build large projects, such as CVC5 (using the knitja converter). For massive builds though, like LLVM, Knit suffers from some performance problems that could be improved.
  • Better support for dynamic dependencies. Currently it is possible to handle dynamic dependencies by generating rules, but I would like to explore the possibility of a more clean and cohesive solution.
    • Ptrace enabled automatic dependency discovery (Linux-only feature). See the xkvt project for some experiments on this front.
  • Global build file cache (similar to ccache, but for every command that is executed).
  • A restrictive mode for build sandboxing.

Feedback

It is always useful useful to get feedback from others to improve Knit. If you have feedback, or questions about how to use it, please open a discussion. It would be great to discuss the good and bad parts of the current design, and how it can be improved.

Usage

Usage of knit:
  knit [TARGETS] [ARGS]

Options:
  -B, --always-build        unconditionally build all targets
      --cache string        directory for caching internal build information (default ".")
      --cpuprofile string   write cpu profile to 'file'
  -D, --debug               print debug information
  -C, --directory string    run command from directory
  -n, --dry-run             print commands without actually executing
  -f, --file string         knitfile to use (default "knitfile")
      --hash                hash files to determine if they are out-of-date (default true)
  -h, --help                show this help message
      --keep-going          keep going even if recipes fail
  -q, --quiet               don't print commands
      --shell string        shell to use when executing commands (default "sh")
  -s, --style string        printer style to use (basic, steps, progress) (default "basic")
  -j, --threads int         number of cores to use (default 8)
  -t, --tool string         subtool to invoke (use '-t list' to list subtools); further flags are passed to the subtool
  -u, --updated strings     treat files as updated
  -v, --version             show version information

Available sub-tools (knit -t list):

list - list all available tools
graph - print build graph in specified format: text, tree, dot, pdf
clean - remove all files produced by the build
targets - list all targets (pass 'virtual' for just virtual targets)
compdb - output a compile commands database
commands - output the build commands (formats: knit, json, make, ninja, shell)
status - output dependency status information

Contributing

If you find a bug or have a feature request please open an issue for discussion. I am sometimes prone to being unresponsive to pull requests, so I apologize in advance. Please ping me if I forget to respond. If you have a feature you would like to implement, please double check with me about the feature before investing lots of time into implementing it.

If you have a question or feedback about the current design, please open a discussion.

More Repositories

1

micro

A modern and intuitive terminal-based text editor
Go
22,579
star
2

generic

A collection of generic data structures written in Go.
Go
1,260
star
3

eget

Easily install prebuilt binaries from GitHub.
Go
708
star
4

Literate

A literate programming tool for any language
D
641
star
5

multiplix

An operating system kernel for RISC-V and AArch64 SBCs
D
120
star
6

SFML.jl

A binding of the game and multimedia library SFML for Julia
Julia
93
star
7

gpeg

A PEG parsing machine with support for incremental parsing.
Go
75
star
8

perforator

Record "perf" performance metrics for individual functions/regions of an ELF binary.
Go
58
star
9

sregx

A tool and library for using structural regular expressions.
Go
56
star
10

highlight

A Go package for syntax highlighting
Go
54
star
11

literate.vim

Vim plugin for the Literate programming tool
Vim Script
36
star
12

termbox-d

A D wrapper of the TUI library Termbox
D
23
star
13

dotvim

My Vim configuration
Vim Script
20
star
14

AnimatedPlots.jl

Fast animated (and static) plots for Julia
Julia
18
star
15

vim-snake

Snake game written in Vimscript
Vim Script
18
star
16

unionize

A tool for generating unions in Go
Go
16
star
17

flare

Syntax highlighting engine built using GPeg
Go
13
star
18

julialint.vim

Built in Vim linter for Julia
Vim Script
12
star
19

lfi

Lightweight Fault Isolation: Practical, Efficient, and Secure Software Sandboxing
C
12
star
20

mkinfo

Program to fix the terminal entry problem
Go
11
star
21

Chipmunk.jl

A binding of the physics engine Chipmunk for Julia
Julia
9
star
22

riscinator

A tiny 3-stage RISC-V core written in Chisel.
C
9
star
23

make-template

Makefile template for simple C/C++ programs
Makefile
9
star
24

crypt

A tool for making password-protected files
Go
9
star
25

cgc

C garbage collector
C
8
star
26

wasm2bin

Compiler from Webassembly to native binaries
C
8
star
27

SpaceShooter.jl

Julia
7
star
28

clipper

Cross-platform clipboard access in Go
Go
6
star
29

xkvt

A tool for tracing the inputs and outputs of a command
Go
5
star
30

ftdetect

A filetype detection library
Go
5
star
31

rvasm

A simple RISC-V RV32I assembler
Go
4
star
32

JuliaCon-SFML

SFML.jl examples from JuliaCon
Julia
4
star
33

rvsym

A small RISC-V symbolic execution engine
Go
4
star
34

blog-code

D
4
star
35

microsnippets

Lua
4
star
36

ballgame

Go
4
star
37

rope

A rope data structure implemented in Go
Go
4
star
38

ht

Simple hashtable written in C
C
3
star
39

elm-asteroids

Elm
3
star
40

midi

Go
3
star
41

asm

Go
3
star
42

knitja

A tool for converting Ninja build files to Knitfiles
C
3
star
43

julia-jump-game

Julia
3
star
44

homebrew-micro

Ruby
2
star
45

tzu

An experiment in unpredictability.
Go
2
star
46

dpi

D
2
star
47

glob

A Go package for glob matching
Go
2
star
48

install-utils

Shell
2
star
49

armgen

A tool for analyzing the Arm Machine Readable Specification
Go
2
star
50

gpeg-extra

C
2
star
51

riscv-gnu-toolchain-prebuilt

Pre-built Linux binaries for the RISC-V GNU Toolchain
Shell
2
star
52

pios

C
2
star
53

avalon

Go
2
star
54

rpi4

C
2
star
55

quark

QUark Assembly Rewriting Kit
C++
2
star
56

lfi-artifact

2
star
57

armvis

Visualizing the ARM64 instruction set
Go
2
star
58

CarGame

JavaScript
1
star
59

hifive

Simple bare-metal library and example programs for the HiFive 1 Rev B
C
1
star
60

setup-knit

GitHub Action for installing Knit
1
star
61

tanks

Go
1
star
62

interval

Go
1
star
63

sfml

Go
1
star
64

kbd

Go
1
star
65

rustalyzer

Rust
1
star
66

libmemstk

C
1
star
67

literate-micro

Lua
1
star
68

TankGame

JavaScript
1
star
69

axum

A small SoC built around the Ibex processor
C
1
star
70

comp

A tool for finding and executing commands from compile_commands.json
Go
1
star
71

verilator-example

C++
1
star
72

revisor

D
1
star
73

CircuitEvolver

D
1
star
74

bzlatest

C++
1
star
75

CoDI

JavaScript
1
star