• Stars
    star
    118
  • Rank 299,923 (Top 6 %)
  • Language
    Rust
  • License
    GNU General Publi...
  • Created over 2 years ago
  • Updated 11 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Evaluating & benchmarking ZKP compilation strategies.

ZKP (circuit) compiler shootout

Evaluating & benchmarking ZKP compilation strategies.

Currently we are testing the following Zero Knowledge machines

  1. RISC0
  2. Miden
  3. Plonk
  4. Halo2

If you would like your machine / framework / compilation strategy to be benchmarked against the standard suite, please submit a PR! You can find instructions below.

Results can be seen in the following files:

The results were collected on a AMD Ryzen 7 5700X 8-Core @ 16x 3.4GHz CPU.

To see very rough notes about the languages in the benchmark and potential improvement points please read my notes file.

How to get benchmark results

There are two commands for producing results

  1. make table arg="desired_backend"
  2. make bench arg="desired_backend

It is recommended to run make table. However it requires two external dependencies

  1. cargo-criterion
  2. criterion-table

these can both be installed with the following command.

cargo install cargo-criterion
cargo install criterion-table

Make sure cargo packages are on your path.

Either way, the results can be seen

  1. run make table arg="desired_flags"
    • or make bench arg="desired_flags, if one does not wish to install the cargo packages
  2. sit back and watch your CPU spin
  3. The HTML results should be in ./shootout/target/criterion/reports/index.html
  4. If make table was run, the updated benchmark results should be seen in ./BENCHMARKS.md

Benchmark Flags

In order to be effective for developers and users wishing to verify results, we do not enable all benchmarks by default.

The following class of flags are had

  1. Compiler flags
  2. Test flags
  3. Step Flags

All Step flags and Test flags are enabled by default. ALl Step flags on may be too slow for your workflow.

Parent Flag Flag default status Notes
all all off
all_compilers miden off
all_compilers triton off
all_compilers halo2 off
all_compilers plonk off
all_compilers risc off Fails to compile
all_compilers vampir_p off
all_compilers vampir_halo2 off
all_tests sudoku on
all_tests fib on
all_tests blake on
all_steps compile on
all_steps prove on
all_steps verify on
all_steps prove_and_verify on
  • Any compilers that have Fails to compile, have to be enabled by hand in arg, as they are excluded from their parent

So if one wants to run everything one can run

make table arg="all"

In fact if one is fine with the default on features then one just needs to run

make table arg="miden"
make bench arg="miden"

where you can replace miden with any compiler_flag, such as risc.

However if one wishes to turn off a feature like various steps or tests

then one has to run a command like

cd shootout

cargo bench --no-default-features --features "all_tests prove_and_verify miden"

Here, we turn off all the steps, running only the prove_and_verify part of the benchmark.

Contributing

This repository serves as a base for contributions and so contributions should hopefully be easy, and deficiencies on the implementations well known.

Hopefully the process is rather straight forward, however we give a guide below to more quickly get familiar with the structure of the project.

To get a quick TL;DR feeling of this you could alternatively read how the miden backend plugs in both in the miden.rs, main.rs, bench.rs, and the miden sub directory for project layout. That should hopefully be readable enough without having to read the full guide below.

Layout Structure: SRC

The project has two components of organization, the shootout directory has an src folder that hosts a file per backend. These files (miden.rs, risc.rs, etc.) are rather basic, just hosting an implementation of the given backend struct that was created.

.
├── Cargo.lock
├── Cargo.toml
├── src
│   ├── bench.rs
│   ├── halo.rs
│   ├── main.rs
│   ├── miden.rs
│   ├── plonk.rs
│   └── risc.rs

We can see other files as well such as bench.rs and main.rs

For creating a new backend to test, one will have to touch bench.rs to have their new backend be tested. Since the code added is just a way to get around Rust's lack of typing of existentials this should be an easy task.

One should also add a feature flag for one's compiler in Cargo.toml.

Finally, in main.rs we hook up our tests to a certain benchmark we care about.

pub fn bench_sudoku(c: &mut Criterion) {
    let to_bench = vec![
        #[cfg(feature = "miden")]
        ZKP::Miden(miden::sudoku()),
        #[cfg(feature = "plonk")]
        ZKP::Plonk(plonk::sudoku()),
        #[cfg(feature = "risc")]
        ZKP::Risc0(risc::sudoku()),
        #[cfg(feature = "halo2")]
        ZKP::Halo2(halo::sudoku()),
    ];
    bench_zkp(c, String::from("Sudoku"), to_bench)
}

As we can see it's just adding the backend's struct to a vector list and wrapped with the enum found in bench.rs.

This structure should also make it easy to add new benchmark programs, please feel free to add new kinds of programs we wish to test here with whatever backend you are interested in! Hopefully others, including myself, can contribute to other backends for your test!

Layout Structure: sub directories

The layout structures of the other directories are as follows:

.
├── Cargo.lock
├── Cargo.toml
├── miden
├── notes.org
├── risc
├── sudoku-halo2
├── sudoku-plonk
└── zero-knowledge
    ├── Cargo.toml
    └── src

The most important folder is zero-knowledge as this has the trait needed to implement to have the benchmarker work.

pub trait ZeroKnowledge {
    type C;
    type R;
    fn name(&self) -> String;
    fn compile(&self) -> Self::C;
    fn prove(&self, setup: &mut Self::C) -> Self::R;
    fn verify(&self, receipt: Self::R, program: &mut Self::C) -> ();
    fn prove_and_verify(&self) -> () {
        let circuit = self.compile();
        let receipt = self.prove(&circuit);
        self.verify(receipt, &circuit);
    }
}

Here for a custom backend one just needs to implement compile, prove, verify, and a name, and your new zero knowledge backend can be tested!

The other folders like miden and risc, represent stand alone backends. Adding a new program to benchmark or improve a current benchmark should be quite easy.

the sudoku-halo2 and sudoku-plonk are similar, but represent a single program/project style structure (In the future we will likely transform these to fit the mold of what risc and miden have).

Lastly, the notes.org file lays out notes about the backends and deficiencies in any particular implementation, hopefully over time all the programs will be optimal for their specific backend to have a more accurate and fair results.

Project Structure

For adding a new backend or adding new programs for backends like halo2 or plonk. One may have to create a new sub directory.

this can be done with cargo new, or if you already have working code, just copy and paste your code into the directory! each of the sub-directories are standalone independent projects with their own workspace. The only new dependency you'll have to add is:

[dependencies]
zero-knowledge = { path = "../zero-knowledge" }

and make sure the top level Cargo.toml excludes the project and brings it in as a dependency.

If you are importing existing code for a new backend, make sure you implement the ZeroKnowledge trait, and you're all set!

We will now talk about backend specific considerations, feel free to skip these if they aren't a backend you care about contributing to

Project Structure: Miden

The miden backend can be found in the miden sub directory. Really one does not have to alter this code at all, just add your program to src/miden.rs to mainly fill in the starting advice tape or the starting stack, along with the file path where the miden program is stored.

Ι personally keep my miden code in miden-assembler, as Ι generate out miden code from my Common lisp DSL in programs sub directory. All one needs to run this code is quick-lisp and just write

;; if you need to load the file by hand
;; I will assume the repl is in the miden-assembler directory
(load "miden-assembler.asd")
;; load the project
(ql:quickload :miden-assembler)
;; switch to the project
(in-package :miden)
;; dump the code changes
(dump)

However you may keep it wherever you feel, just make sure path is properly given in the miden.rs file.

the rest is just hooking up your new program to main.rs which should be straight forward.

Project Structure: Risc0

Risc0 can compile straight rust code, which is nice, however this means the structure is a bit different from miden or other ZKVMs.

.
├── Cargo.toml
├── methods
│   ├── build.rs
│   ├── Cargo.toml
│   ├── guest
│   └── src
├── README.md
├── src
│   ├── lib.rs
│   └── main.rs
├── sudoku-core
│   ├── Cargo.toml
│   └── src
└── target

the src directory contains lib.rs which you will only need to touch the imports from the methods directory. the Risc struct should be general enough to handle any program you wish to compile.

.
├── build.rs
├── Cargo.toml
├── guest
│   ├── build.rs
│   ├── Cargo.lock
│   ├── Cargo.toml
│   ├── src
│   │   └── bin
│   │       ├── fib_fifty.rs
│   │       ├── fib_ninty_two.rs
│   │       ├── fib.rs
│   │       └── sudoku.rs
└── src
    └── lib.rs

inside methods, you should place your code in the bin of guest. As you can see the names of the current programs, these correspond to the imports in lib.rs

pub use methods_fib::{FIB_ID,           FIB_PATH,
                      FIB_FIFTY_ID,     FIB_FIFTY_PATH,
                      FIB_NINTY_TWO_ID, FIB_NINTY_TWO_PATH,
                      SUDOKU_ID,        SUDOKU_PATH};

Note that any dependency you want to be compiled with these should be placed in the Cargo.toml in the guest directory. This includes shared code like we have for sudoku-core, which is linked both to the guest and to the native rust code in src.

Much like miden, all that is now required is to make the Risc struct in shootout/src/risc.rs for your new program and tell main.rs to benchmark it!

Project Structure: Stand alone.

Hopefully miden shows a good way to create standalone projects. So if one has existing halo2 or plonk code, one could just copy and paste it into a directory and implement the ZeroKnowledge trait. for their program.

Alucard/VAMP-IR

Please see the official Alucard repository.

Geb

See the Geb repository.

More Repositories

1

namada

Rust implementation of Namada, a Proof-of-Stake L1 for interchain asset-agnostic privacy
Rust
2,394
star
2

juvix

A language for intent-centric and declarative decentralised applications
Haskell
447
star
3

anoma-archive

Reference implementation of the Anoma protocols in Rust.
Rust
426
star
4

namada-testnets

Testnet configurations and coordination for the Namada network.
327
star
5

namada-trusted-setup-claimer

Sign arbitrary messages with keys obtained from Namada's trusted setup
Rust
320
star
6

vamp-ir

Vamp-IR is a proof-system-agnostic language for writing arithmetic circuits
Rust
155
star
7

taiga

A framework for generalized shielded state transitions
Rust
134
star
8

masp

The multi-asset shielded pool (MASP) provides a unified privacy set for all assets on Namada.
Rust
88
star
9

ferveo

An implementation of a DKG protocol for front-running protection on Anoma.
Rust
78
star
10

namada-interface

TypeScript
77
star
11

whitepaper

TeX
59
star
12

namada-docs

MDX
48
star
13

namada-shielded-expedition

Python
42
star
14

typhon

Formalization of Typhon protocol
TLA
28
star
15

geb

A Categorical View of Computation
Idris
28
star
16

research

Early-stage research related to the Anoma protocols.
Idris
23
star
17

specs

Specifications for the Anoma protocols.
TeX
23
star
18

alucard

A common lisp DSL for writing zero knowledge circuits
Common Lisp
18
star
19

namada-indexer

Namada indexer to be used in conjunction with the namada interface
Rust
16
star
20

ethereum-bridge

A generalized fully trustless Ethereum bridge for sovereign chains.
Solidity
12
star
21

minijuvix

Haskell
11
star
22

juvix-stdlib

The Juvix standard library
Makefile
9
star
23

VampIR-Book

A book which can act as an introduction and documentation for VampIR
CSS
9
star
24

masp-mpc

MASP MPC code
Rust
8
star
25

namada-mainnet-genesis

Python
7
star
26

namada-sdk-starter

Rust
6
star
27

juvix-nightly-builds

6
star
28

abstract-resource-machine-simulator

Makefile
6
star
29

anoma-alpha

An intent-centric, privacy-preserving protocol for decentralized counterparty discovery, solving, and multi-chain atomic settlement.
Rust
6
star
30

juvix-e2e-demo

Juvix compiler stack - Integration demo
Dockerfile
6
star
31

exhibit_plonkup

Rust
5
star
32

devtool

Rust
4
star
33

anoma-blog

Jupyter Notebook
4
star
34

verify-beacon

Rust
3
star
35

vscode-juvix

VSCode extension for Juvix
TypeScript
3
star
36

dev-utils

Shell
3
star
37

namada-whitepaper

Namada whitepaper
3
star
38

juvix-docs

Juvix Official Documentation
Makefile
3
star
39

devchain-container

Shell
2
star
40

intents-snippets

A repo to put code snippets related to intent research
Agda
2
star
41

Isabelle-HPaxos

A formalization of HPaxos in Isabelle
Isabelle
2
star
42

typhon-sml

Typhon reference implementation in Standard ML
Standard ML
2
star
43

namada-genesis

Python
2
star
44

namada-masp-indexer

The Namada Masp Indexer reference implementation.
Rust
2
star
45

anoma-wasm-multitoken

Validity predicate and transactions for a variable supply multitoken account
Rust
2
star
46

wasm-workspace

Rust
2
star
47

namada-shielded-airdrop

Rust
2
star
48

anoma-blog-writing-guide

Comprehensive guide to writing blogposts for the anoma blog.
2
star
49

juvix-quickcheck

Property-based testing for Juvix
Makefile
2
star
50

plonkup-hash

A collection of hash gadgets written for Plonkup
Rust
1
star
51

juvix-mode

The Juvix Emacs mode
Emacs Lisp
1
star
52

vscode-vampir

1
star
53

anoma-app-lib

A library for intent-centric application development.
1
star
54

juvix-test

A testing framework for Juvix
Shell
1
star
55

juvix-anoma-test

A library for testing Anoma applications
1
star
56

juvix-containers

Immutable container types for Juvix
Makefile
1
star
57

highlightjs-juvix

Juvix language grammars for highlightjs (low maintenance)
JavaScript
1
star
58

art-template

Anoma Research Topics Template
TeX
1
star
59

juvix-anoma-stdlib

A Juvix library for writing Anoma applications
Makefile
1
star
60

namada-light-sdk-starter

Namada light sdk starter kit
Dockerfile
1
star
61

ART-2023-Gureev-Cubides-Geb-Pipeline

TeX
1
star