• Stars
    star
    342
  • Rank 123,731 (Top 3 %)
  • Language
    Rust
  • License
    The Unlicense
  • Created over 8 years ago
  • Updated 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A small utility to compare Rust micro-benchmarks.

cargo benchcmp

A small utility for comparing micro-benchmarks produced by cargo bench. The utility takes as input two sets of micro-benchmarks (one "old" and the other "new") and shows as output a comparison between each benchmark.

Linux build status Windows build status crates.io page

Dual-licensed under MIT or the UNLICENSE.

Installation

cargo benchcmp can be installed with cargo install:

$ cargo install cargo-benchcmp

The resulting binary should then be in $HOME/.cargo/bin.

Criterion support

This tool only supports the standard benchmark output emitted by cargo bench. For Criterion support, a different tool was developed called critcmp.

Example output

Coloured example output on aho-corasick benchmarks

Usage

First, run your benchmarks and save them to a file:

$ cargo bench > control

Next, apply the changes you'd like to test out, and then run the benchmarks and save them to a file again.

$ cargo bench > variable

Finally, use cargo benchcmp to compare the benchmark results!

$ cargo benchcmp control variable
 name                                bench_output_3.txt ns/iter  bench_output_2.txt ns/iter  diff ns/iter   diff %  speedup
 ac_one_byte                         354 (28248 MB/s)            349 (28653 MB/s)                      -5   -1.41%   x 1.01
 ac_one_prefix_byte_every_match      150,581 (66 MB/s)           112,957 (88 MB/s)                -37,624  -24.99%   x 1.33
 ac_one_prefix_byte_no_match         354 (28248 MB/s)            350 (28571 MB/s)                      -4   -1.13%   x 1.01
 ac_one_prefix_byte_random           20,273 (493 MB/s)           16,096 (621 MB/s)                 -4,177  -20.60%   x 1.26
 ac_ten_bytes                        108,092 (92 MB/s)           58,588 (170 MB/s)                -49,504  -45.80%   x 1.84
 ac_ten_diff_prefix                  108,082 (92 MB/s)           58,601 (170 MB/s)                -49,481  -45.78%   x 1.84
...

If you want to compare the same benchmark run in multiple ways, reuse the names in different modules. Then your benchmark output will look like:

module1::ac_two_one_prefix_byte_random   ...
module2::ac_two_one_prefix_byte_random   ...

You can then instruct benchcmp to compare the two modules by providing the two prefixes, followed by the file containing the output:

$ cargo benchcmp module1:: module2:: benchmark-output
 name                                dense_boxed:: ns/iter  dense:: ns/iter    diff ns/iter   diff %  speedup
 ac_one_byte                         354 (28248 MB/s)       349 (28653 MB/s)             -5   -1.41%   x 1.01
 ac_one_prefix_byte_every_match      150,581 (66 MB/s)      112,957 (88 MB/s)       -37,624  -24.99%   x 1.33
 ac_one_prefix_byte_no_match         354 (28248 MB/s)       350 (28571 MB/s)             -4   -1.13%   x 1.01
 ac_one_prefix_byte_random           20,273 (493 MB/s)      16,096 (621 MB/s)        -4,177  -20.60%   x 1.26
 ac_ten_bytes                        108,092 (92 MB/s)      58,588 (170 MB/s)       -49,504  -45.80%   x 1.84
 ac_ten_diff_prefix                  108,082 (92 MB/s)      58,601 (170 MB/s)       -49,481  -45.78%   x 1.84
 ac_ten_one_prefix_byte_every_match  150,561 (66 MB/s)      112,920 (88 MB/s)       -37,641  -25.00%   x 1.33
 ac_ten_one_prefix_byte_no_match     354 (28248 MB/s)       350 (28571 MB/s)             -4   -1.13%   x 1.01
 ac_ten_one_prefix_byte_random       23,684 (422 MB/s)      19,181 (521 MB/s)        -4,503  -19.01%   x 1.23
 ac_two_bytes                        3,138 (3186 MB/s)      3,125 (3200 MB/s)           -13   -0.41%   x 1.00
 ac_two_diff_prefix                  3,138 (3186 MB/s)      3,124 (3201 MB/s)           -14   -0.45%   x 1.00
 ac_two_one_prefix_byte_every_match  150,571 (66 MB/s)      112,934 (88 MB/s)       -37,637  -25.00%   x 1.33
 ac_two_one_prefix_byte_no_match     354 (28248 MB/s)       350 (28571 MB/s)             -4   -1.13%   x 1.01
 ac_two_one_prefix_byte_random       21,009 (476 MB/s)      16,511 (605 MB/s)        -4,498  -21.41%   x 1.27

The tool supports basic filtering. For example, it's easy to see only improvements:

$ cargo benchcmp old new --improvements
 name                             full:: ns/iter     full_overlap:: ns/iter  diff ns/iter  diff %  speedup
 ac_one_byte                      367 (27247 MB/s)   367 (27247 MB/s)                   0   0.00%   x 1.00
 ac_two_one_prefix_byte_no_match  371 (26954 MB/s)   368 (27173 MB/s)                  -3  -0.81%   x 1.01
 ac_two_one_prefix_byte_random    11,530 (867 MB/s)  11,514 (868 MB/s)                -16  -0.14%   x 1.00

Or only see regressions:

$ cargo benchcmp old new --regressions
 name                                full:: ns/iter     full_overlap:: ns/iter  diff ns/iter  diff %  speedup
 ac_one_prefix_byte_every_match      27,425 (364 MB/s)  27,972 (357 MB/s)                547   1.99%   x 0.98
 ac_one_prefix_byte_no_match         367 (27247 MB/s)   373 (26809 MB/s)                   6   1.63%   x 0.98
 ac_one_prefix_byte_random           11,076 (902 MB/s)  11,243 (889 MB/s)                167   1.51%   x 0.99
 ac_ten_bytes                        25,474 (392 MB/s)  25,754 (388 MB/s)                280   1.10%   x 0.99
 ac_ten_diff_prefix                  25,466 (392 MB/s)  25,800 (387 MB/s)                334   1.31%   x 0.99
 ac_ten_one_prefix_byte_every_match  27,424 (364 MB/s)  28,046 (356 MB/s)                622   2.27%   x 0.98
 ac_ten_one_prefix_byte_no_match     367 (27247 MB/s)   369 (27100 MB/s)                   2   0.54%   x 0.99
 ac_ten_one_prefix_byte_random       13,661 (732 MB/s)  13,742 (727 MB/s)                 81   0.59%   x 0.99
 ac_two_bytes                        3,141 (3183 MB/s)  3,164 (3160 MB/s)                 23   0.73%   x 0.99
 ac_two_diff_prefix                  3,141 (3183 MB/s)  3,174 (3150 MB/s)                 33   1.05%   x 0.99
 ac_two_one_prefix_byte_every_match  27,638 (361 MB/s)  27,953 (357 MB/s)                315   1.14%   x 0.99

Many times, the difference in micro-benchmarks is just noise, so you can filter by percent difference:

$ cargo benchcmp old new --regressions --threshold 2
 name                                full:: ns/iter     full_overlap:: ns/iter  diff ns/iter  diff %  speedup
 ac_ten_one_prefix_byte_every_match  27,424 (364 MB/s)  28,046 (356 MB/s)                622   2.27%   x 0.98

More Repositories

1

ripgrep

ripgrep recursively searches directories for a regex pattern while respecting your gitignore
Rust
48,517
star
2

xsv

A fast CSV command line toolkit written in Rust.
Rust
10,370
star
3

toml

TOML parser for Golang with reflection.
Go
4,464
star
4

quickcheck

Automated property based testing for Rust (with shrinking).
Rust
2,408
star
5

erd

Translates a plain text description of a relational database schema to a graphical entity-relationship diagram.
Haskell
1,805
star
6

fst

Represent large sets and maps compactly with finite state transducers.
Rust
1,771
star
7

jiff

A date-time library for Rust that encourages you to jump into the pit of success.
Rust
1,736
star
8

rust-csv

A CSV parser for Rust, with Serde support.
Rust
1,706
star
9

walkdir

Rust library for walking directories recursively.
Rust
1,283
star
10

nflgame

An API to retrieve and read NFL Game Center JSON data. It can work with real-time data, which can be used for fantasy football.
Python
1,257
star
11

nfldb

A library to manage and update NFL data in a relational database.
Python
1,079
star
12

aho-corasick

A fast implementation of Aho-Corasick in Rust.
Rust
1,027
star
13

byteorder

Rust library for reading/writing numbers in big-endian and little-endian.
Rust
980
star
14

wingo

A fully-featured window manager written in Go.
Go
958
star
15

memchr

Optimized string search routines for Rust.
Rust
888
star
16

bstr

A string type for Rust that is not required to be valid UTF-8.
Rust
795
star
17

advent-of-code

Rust solutions to AoC 2018
Rust
479
star
18

xgb

The X Go Binding is a low-level API to communicate with the X server. It is modeled on XCB and supports many X extensions.
Go
472
star
19

termcolor

Cross platform terminal colors for Rust.
Rust
462
star
20

rust-snappy

Snappy compression implemented in Rust (including the Snappy frame format).
Rust
449
star
21

go-sumtype

A simple utility for running exhaustiveness checks on Go "sum types."
Go
421
star
22

chan

Multi-producer, multi-consumer concurrent channel for Rust.
Rust
392
star
23

regex-automata

A low level regular expression library that uses deterministic finite automata.
Rust
352
star
24

suffix

Fast suffix arrays for Rust (with Unicode support).
Rust
261
star
25

rure-go

Go bindings to Rust's regex engine.
Go
250
star
26

tabwriter

Elastic tabstops for Rust.
Rust
247
star
27

imdb-rename

A command line tool to rename media files based on titles from IMDb.
Rust
226
star
28

rebar

A biased barometer for gauging the relative speed of some regex engines on a curated set of tasks.
Python
226
star
29

critcmp

A command line tool for comparing benchmarks run by Criterion.
Rust
216
star
30

ty

Easy parametric polymorphism at run time using completely unidiomatic Go.
Go
198
star
31

xgbutil

A utility library to make use of the X Go Binding easier. (Implements EWMH and ICCCM specs, key binding support, etc.)
Go
191
star
32

pytyle3

An updated (and much faster) version of pytyle that uses xpybutil and is compatible with Openbox Multihead.
Python
181
star
33

dotfiles

My configuration files and personal collection of scripts.
Vim Script
154
star
34

rsc-regexp

Translations of a simple C program to Rust.
Rust
137
star
35

rust-cbor

CBOR (binary JSON) for Rust with automatic type based decoding and encoding.
Rust
129
star
36

chan-signal

Respond to OS signals with channels.
Rust
125
star
37

goim

Goim is a robust command line utility to maintain and query the Internet Movie Database (IMDb).
Go
117
star
38

same-file

Cross platform Rust library for checking whether two file paths are the same file.
Rust
101
star
39

clibs

A smattering of miscellaneous C libraries. Includes sane argument parsing, a thread-safe multi-producer/multi-consumer queue, and implementation of common data structures (hashmaps, vectors and linked lists).
C
98
star
40

ucd-generate

A command line tool to generate Unicode tables as source code.
Rust
95
star
41

nflvid

An experimental library to map play meta data to footage of that play.
Python
91
star
42

rust-stats

Basic statistical functions on streams for Rust.
Rust
87
star
43

migration

Package migration for Golang automatically handles versioning of a database schema by applying a series of migrations supplied by the client.
Go
81
star
44

winapi-util

Safe wrappers for various Windows specific APIs.
Rust
64
star
45

xpybutil

An incomplete xcb-util port plus some extras
Python
62
star
46

graphics-go

Automatically exported from code.google.com/p/graphics-go
Go
60
star
47

rust-pcre2

High level Rust bindings to PCRE2.
C
56
star
48

blog

My blog.
Rust
52
star
49

rust-sorts

Implementations of common sorting algorithms in Rust with comprehensive tests and benchmarks.
Rust
51
star
50

openbox-multihead

Openbox with patches for enhanced multihead support.
C
46
star
51

nakala

A low level embedded information retrieval system.
Rust
45
star
52

nflfan

View your fantasy teams with nfldb using a web interface.
JavaScript
43
star
53

globset

A globbing library for Rust.
Rust
42
star
54

utf8-ranges

Convert contiguous ranges of Unicode codepoints to UTF-8 byte ranges.
Rust
42
star
55

rtmpdump-ksv

rtmpdump with ksv's patch. Intended to track upstream git://git.ffmpeg.org/rtmpdump as well.
C
40
star
56

regexp

A regular expression library implemented in Rust.
Rust
37
star
57

xdg

A Go package for reading config and data files according to the XDG Base Directory specification.
Go
35
star
58

locker

A simple Golang package for conveniently using named read/write locks. Useful for synchronizing access to session based storage in web applications.
Go
34
star
59

nflcmd

A collection of command line utilities for viewing NFL statistics and rankings with nfldb.
Python
32
star
60

notes

A collection of small notes that aren't appropriate for my blog.
31
star
61

mempool

A fast thread safe memory pool for reusing allocations.
Rust
29
star
62

gribble

A command oriented language whose environment is defined through Go struct types by reflection.
Go
28
star
63

vcr

A simple wrapper tool around ffmpeg to capture video from a VCR.
Rust
27
star
64

encoding_rs_io

Streaming I/O adapters for the encoding_rs crate.
Rust
25
star
65

rust-cmail

A simple command line utility for periodically sending email containing the output of long-running commands.
Rust
21
star
66

cluster

A simple API for managing a network cluster with smart peer discovery.
Go
19
star
67

pager-multihead

A pager that supports per-monitor desktops (compatible with Openbox Multihead and Wingo)
Python
15
star
68

cablastp

Performs BLAST on compressed proteomic data.
Go
15
star
69

rust-error-handling-case-study

Code for the case study in my blog post: http://blog.burntsushi.net/rust-error-handling
Rust
15
star
70

rg-cratesio-typosquat

The source code of the 'rg' crate. It is an intentional typo-squat that redirects folks to 'ripgrep'.
Rust
15
star
71

imgv

An image viewer for Linux written in Go.
Go
14
star
72

cmd

A convenience library for executing commands in Go, including executing commands in parallel with a pool.
Go
14
star
73

fanfoot

View your fantasy football leagues and get text alerts when one of your players scores.
Python
12
star
74

cmail

cmail runs a command and sends the output to your email address at certain intervals.
Go
12
star
75

gohead

An xrandr wrapper script to manage multi-monitor configurations. With hooks.
Go
12
star
76

burntsushi-blog

A small Go application for my old blog.
CSS
12
star
77

intern

A simple package for interning strings, with a focus on efficiently representing dense pairwise data.
Go
11
star
78

crev-proofs

My crev reviews.
10
star
79

pytyle1

A lightweight X11 tool for simulating tiling in a stacking window manager.
Python
9
star
80

cif

A golang package for reading and writing data in the Crystallographic Information File (CIF) format. It mostly conforms to the CIF 1.1 specification.
Go
9
star
81

rucd

WIP
Rust
8
star
82

qcsv

An API to read and analyze CSV files by inferring types for each column of data.
Python
8
star
83

pyndow

A window manager written in Python
Python
8
star
84

csql

Package csql provides convenience functions for use with the types and functions defined in the standard library `database/sql` package.
Go
6
star
85

freetype-go

A fork of freetype-go with bounding box calculations.
Go
6
star
86

sqlsess

Simple database backed session management. Integrates with Gorilla's sessions package.
Go
6
star
87

go-wayland-simple-shm

C
5
star
88

sqlauth

A simple Golang package that provides database backed user authentication with bcrypt.
Vim Script
4
star
89

lcmweb

A Go web application for coding documents with the Linguistic Category Model.
JavaScript
4
star
90

bcbgo

Computational biology tools for the BCB group at Tufts University.
Go
4
star
91

fex

A framework for specifying and executing experiments.
Haskell
3
star
92

present

My presentations.
HTML
3
star
93

memchr-2.6-mov-regression

Rust
3
star
94

genecentric

A tool to generate between-pathway modules and perform GO enrichment on them.
Python
3
star
95

rust-docs

A silly repo for managing my Rust crate documentation.
Python
3
star
96

pcre2-mirror

A git mirror for PCRE2's SVN repository at svn://vcs.exim.org/pcre2/code
2
star
97

xpyb

A clone of xorg-xpyb.
C
2
star
98

burntsushi-homepage

A small PHP web application for my old homepage.
PHP
2
star
99

window-marker

Use vim-like marks on windows.
Python
2
star
100

sudoku

An attempt at a sudoku solver in Haskell.
Haskell
1
star