ZKP (circuit) compiler shootout
Evaluating & benchmarking ZKP compilation strategies.
Currently we are testing the following Zero Knowledge machines
If you would like your machine / framework / compilation strategy to be benchmarked against the standard suite, please submit a PR! You can find instructions below.
Results can be seen in the following files:
- BENCHMARKS-x86.md
- The results rendered in a mark down table
- BENCHMARKS-arm64.md
- The results rendered in a mark down table
- anoma.github.io/zkp-compiler-shootout/
- The results with graphs and standard deviations, means, etc.
The results were collected on a AMD Ryzen 7 5700X 8-Core @ 16x 3.4GHz
CPU.
To see very rough notes about the languages in the benchmark and potential improvement points please read my notes file.
How to get benchmark results
There are two commands for producing results
make table arg="desired_backend"
make bench arg="desired_backend
It is recommended to run make table
. However it requires two
external dependencies
cargo-criterion
criterion-table
these can both be installed with the following command.
cargo install cargo-criterion
cargo install criterion-table
Make sure cargo packages are on your path.
Either way, the results can be seen
- run
make table arg="desired_flags"
- or
make bench arg="desired_flags
, if one does not wish to install the cargo packages
- or
- sit back and watch your CPU spin
- The HTML results should be in
./shootout/target/criterion/reports/index.html
- If
make table
was run, the updated benchmark results should be seen in./BENCHMARKS.md
Benchmark Flags
In order to be effective for developers and users wishing to verify results, we do not enable all benchmarks by default.
The following class of flags are had
- Compiler flags
- Test flags
- Step Flags
All Step flags
and Test flags
are enabled by default. ALl Step flags
on may be too slow for your workflow.
Parent Flag | Flag | default status | Notes |
---|---|---|---|
all | all | off | |
all_compilers | miden | off | |
all_compilers | triton | off | |
all_compilers | halo2 | off | |
all_compilers | plonk | off | |
all_compilers | risc | off | Fails to compile |
all_compilers | vampir_p | off | |
all_compilers | vampir_halo2 | off | |
all_tests | sudoku | on | |
all_tests | fib | on | |
all_tests | blake | on | |
all_steps | compile | on | |
all_steps | prove | on | |
all_steps | verify | on | |
all_steps | prove_and_verify | on |
- Any compilers that have
Fails to compile
, have to be enabled by hand in arg, as they are excluded from their parent
So if one wants to run everything one can run
make table arg="all"
In fact if one is fine with the default on
features then one just
needs to run
make table arg="miden"
make bench arg="miden"
where you can replace miden with any compiler_flag, such as risc
.
However if one wishes to turn off a feature like various steps
or
tests
then one has to run a command like
cd shootout
cargo bench --no-default-features --features "all_tests prove_and_verify miden"
Here, we turn off all the steps, running only the prove_and_verify
part of the benchmark.
Contributing
This repository serves as a base for contributions and so contributions should hopefully be easy, and deficiencies on the implementations well known.
Hopefully the process is rather straight forward, however we give a guide below to more quickly get familiar with the structure of the project.
To get a quick TL;DR
feeling of this you could alternatively read
how the miden
backend plugs in both in the miden.rs
, main.rs
,
bench.rs
, and the miden
sub directory for project layout. That
should hopefully be readable enough without having to read the full
guide below.
Layout Structure: SRC
The project has two components of organization, the shootout
directory has an src
folder that hosts a file per backend. These
files (miden.rs
, risc.rs
, etc.) are rather basic, just hosting an
implementation of the given backend struct
that was created.
.
├── Cargo.lock
├── Cargo.toml
├── src
│  ├── bench.rs
│  ├── halo.rs
│  ├── main.rs
│  ├── miden.rs
│  ├── plonk.rs
│  └── risc.rs
We can see other files as well such as bench.rs
and main.rs
For creating a new backend to test, one will have to touch
bench.rs
to have their new backend be
tested. Since the code added is just a way to get around Rust's lack
of typing of existentials this should be an easy task.
One should also add a feature flag for one's compiler in Cargo.toml
.
Finally, in main.rs
we hook up our tests to
a certain benchmark we care about.
pub fn bench_sudoku(c: &mut Criterion) {
let to_bench = vec![
#[cfg(feature = "miden")]
ZKP::Miden(miden::sudoku()),
#[cfg(feature = "plonk")]
ZKP::Plonk(plonk::sudoku()),
#[cfg(feature = "risc")]
ZKP::Risc0(risc::sudoku()),
#[cfg(feature = "halo2")]
ZKP::Halo2(halo::sudoku()),
];
bench_zkp(c, String::from("Sudoku"), to_bench)
}
As we can see it's just adding the backend's struct to a vector list
and wrapped with the enum found in bench.rs
.
This structure should also make it easy to add new benchmark programs, please feel free to add new kinds of programs we wish to test here with whatever backend you are interested in! Hopefully others, including myself, can contribute to other backends for your test!
Layout Structure: sub directories
The layout structures of the other directories are as follows:
.
├── Cargo.lock
├── Cargo.toml
├── miden
├── notes.org
├── risc
├── sudoku-halo2
├── sudoku-plonk
└── zero-knowledge
├── Cargo.toml
└── src
The most important folder is zero-knowledge
as this has the trait
needed to implement to have the benchmarker work.
pub trait ZeroKnowledge {
type C;
type R;
fn name(&self) -> String;
fn compile(&self) -> Self::C;
fn prove(&self, setup: &mut Self::C) -> Self::R;
fn verify(&self, receipt: Self::R, program: &mut Self::C) -> ();
fn prove_and_verify(&self) -> () {
let circuit = self.compile();
let receipt = self.prove(&circuit);
self.verify(receipt, &circuit);
}
}
Here for a custom backend one just needs to implement compile
,
prove
, verify
, and a name
, and your new zero knowledge backend
can be tested!
The other folders like miden
and risc
, represent stand alone
backends. Adding a new program to benchmark or improve a current
benchmark should be quite easy.
the sudoku-halo2
and sudoku-plonk
are similar, but represent a
single program/project style structure (In the future we will likely
transform these to fit the mold of what risc
and miden
have).
Lastly, the notes.org
file lays out notes about the backends and
deficiencies in any particular implementation, hopefully over time all
the programs will be optimal for their specific backend to have a more
accurate and fair results.
Project Structure
For adding a new backend or adding new programs for backends like
halo2
or plonk
. One may have to create a new sub directory.
this can be done with cargo new
, or if you already have working
code, just copy and paste your code into the directory! each of the
sub-directories are standalone independent projects with their own
workspace
. The only new dependency you'll have to add is:
[dependencies]
zero-knowledge = { path = "../zero-knowledge" }
and make sure the top level Cargo.toml
excludes the project and
brings it in as a dependency.
If you are importing existing code for a new backend, make sure you
implement the ZeroKnowledge
trait, and you're all set!
We will now talk about backend specific considerations, feel free to skip these if they aren't a backend you care about contributing to
Project Structure: Miden
The miden backend can be found in the miden
sub directory. Really
one does not have to alter this code at all, just add your program to
src/miden.rs
to mainly fill in the starting advice tape or the
starting stack, along with the file path where the miden program is
stored.
Ι personally keep my miden code in miden-assembler, as Ι generate out miden code from my Common lisp DSL in programs sub directory. All one needs to run this code is quick-lisp and just write
;; if you need to load the file by hand
;; I will assume the repl is in the miden-assembler directory
(load "miden-assembler.asd")
;; load the project
(ql:quickload :miden-assembler)
;; switch to the project
(in-package :miden)
;; dump the code changes
(dump)
However you may keep it wherever you feel, just make sure path is
properly given in the miden.rs
file.
the rest is just hooking up your new program to main.rs
which should
be straight forward.
Project Structure: Risc0
Risc0 can compile straight rust code, which is nice, however this
means the structure is a bit different from miden
or other ZKVMs.
.
├── Cargo.toml
├── methods
│  ├── build.rs
│  ├── Cargo.toml
│  ├── guest
│  └── src
├── README.md
├── src
│  ├── lib.rs
│  └── main.rs
├── sudoku-core
│  ├── Cargo.toml
│  └── src
└── target
the src
directory contains lib.rs
which you will only need to
touch the imports from the methods
directory. the Risc
struct
should be general enough to handle any program you wish to compile.
.
├── build.rs
├── Cargo.toml
├── guest
│  ├── build.rs
│  ├── Cargo.lock
│  ├── Cargo.toml
│  ├── src
│  │  └── bin
│  │  ├── fib_fifty.rs
│  │  ├── fib_ninty_two.rs
│  │  ├── fib.rs
│  │  └── sudoku.rs
└── src
└── lib.rs
inside methods
, you should place your code in the bin
of
guest
. As you can see the names of the current programs, these
correspond to the imports in lib.rs
pub use methods_fib::{FIB_ID, FIB_PATH,
FIB_FIFTY_ID, FIB_FIFTY_PATH,
FIB_NINTY_TWO_ID, FIB_NINTY_TWO_PATH,
SUDOKU_ID, SUDOKU_PATH};
Note that any dependency you want to be compiled with these should be
placed in the Cargo.toml
in the guest directory. This includes
shared code like we have for sudoku-core
, which is linked both to
the guest
and to the native rust code in src
.
Much like miden, all that is now required is to make the Risc
struct
in shootout/src/risc.rs
for your new program and tell main.rs
to
benchmark it!
Project Structure: Stand alone.
Hopefully miden
shows a good way to create standalone projects. So
if one has existing halo2
or plonk
code, one could just copy and
paste it into a directory and implement the ZeroKnowledge
trait. for
their program.
Alucard/VAMP-IR
Please see the official Alucard repository.
Geb
See the Geb repository.