• Stars
    star
    1,718
  • Rank 27,170 (Top 0.6 %)
  • Language
    Ruby
  • License
    MIT License
  • Created over 12 years ago
  • Updated about 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Provides iteration per second benchmarking for Ruby

benchmark-ips

Gem Version Build Status Inline docs

DESCRIPTION:

An iterations per second enhancement to Benchmark.

FEATURES/PROBLEMS:

  • benchmark/ips - benchmarks a blocks iterations/second. For short snippits of code, ips automatically figures out how many times to run the code to get interesting data. No more guessing at random iteration counts!

SYNOPSIS:

require 'benchmark/ips'

Benchmark.ips do |x|
  # Configure the number of seconds used during
  # the warmup phase (default 2) and calculation phase (default 5)
  x.config(:time => 5, :warmup => 2)

  # These parameters can also be configured this way
  x.time = 5
  x.warmup = 2

  # Typical mode, runs the block as many times as it can
  x.report("addition") { 1 + 2 }

  # To reduce overhead, the number of iterations is passed in
  # and the block must run the code the specific number of times.
  # Used for when the workload is very small and any overhead
  # introduces incorrectable errors.
  x.report("addition2") do |times|
    i = 0
    while i < times
      1 + 2
      i += 1
    end
  end

  # To reduce overhead even more, grafts the code given into
  # the loop that performs the iterations internally to reduce
  # overhead. Typically not needed, use the |times| form instead.
  x.report("addition3", "1 + 2")

  # Really long labels should be formatted correctly
  x.report("addition-test-long-label") { 1 + 2 }

  # Compare the iterations per second of the various reports!
  x.compare!
end

This will generate the following report:

Warming up --------------------------------------
            addition     3.572M i/100ms
           addition2     3.672M i/100ms
           addition3     3.677M i/100ms
addition-test-long-label
                         3.511M i/100ms
Calculating -------------------------------------
            addition     36.209M (Β± 2.8%) i/s   (27.62 ns/i) -    182.253M in   5.037433s
           addition2     36.552M (Β± 7.8%) i/s   (27.36 ns/i) -    183.541M in   5.069987s
           addition3     36.639M (Β± 4.8%) i/s   (27.29 ns/i) -    182.994M in   5.009234s
addition-test-long-label
                         36.164M (Β± 5.8%) i/s   (27.65 ns/i) -    181.312M in   5.038364s

Comparison:
           addition2: 36558904.5 i/s
           addition3: 36359284.0 i/s - same-ish: difference falls within error
addition-test-long-label: 36135428.8 i/s - same-ish: difference falls within error
            addition: 34666931.3 i/s - same-ish: difference falls within error

Benchmark/ips will report the number of iterations per second for a given block of code. When analyzing the results, notice the percent of standard deviation which tells us how spread out our measurements are from the average. A high standard deviation could indicate the results having too much variability.

One benefit to using this method is benchmark-ips automatically determines the data points for testing our code, so we can focus on the results instead of guessing iteration counts as we do with the traditional Benchmark library.

Custom Suite

Pass a custom suite to disable garbage collection during benchmark:

require 'benchmark/ips'

# Enable and start GC before each job run. Disable GC afterwards.
#
# Inspired by https://www.omniref.com/ruby/2.2.1/symbols/Benchmark/bm?#annotation=4095926&line=182
class GCSuite
  def warming(*)
    run_gc
  end

  def running(*)
    run_gc
  end

  def warmup_stats(*)
  end

  def add_report(*)
  end

  private

  def run_gc
    GC.enable
    GC.start
    GC.disable
  end
end

suite = GCSuite.new

Benchmark.ips do |x|
  x.config(:suite => suite)
  x.report("job1") { ... }
  x.report("job2") { ... }
end

Independent benchmarking

If you are comparing multiple implementations of a piece of code you may want to benchmark them in separate invocations of Ruby so that the measurements are independent of each other. You can do this with the hold! command.

Benchmark.ips do |x|

  # Hold results between multiple invocations of Ruby
  x.hold! 'filename'

end

This will run only one benchmarks each time you run the command, storing results in the specified file. The file is deleted when all results have been gathered and the report is shown.

Alternatively, if you prefer a different approach, the save! command is available. Examples for hold! and save! are available in the examples/ directory.

Multiple iterations

In some cases you may want to run multiple iterations of the warmup and calculation stages and take only the last result for comparison. This is useful if you are benchmarking with an implementation of Ruby that optimizes using tracing or on-stack-replacement, because to those implementations the calculation phase may appear as new, unoptimized code.

You can do this with the iterations option, which by default is 1. The total time spent will then be iterations * warmup + iterations * time seconds.

Benchmark.ips do |x|

  x.config(:iterations => 3)

    # or

  x.iterations = 3

end

Online sharing

If you want to quickly share your benchmark result with others, run you benchmark with SHARE=1 argument. For example: SHARE=1 ruby my_benchmark.rb.

Result will be sent to benchmark.fyi and benchmark-ips will display the link to share the benchmark's result.

If you want to run your own instance of benchmark.fyi and share it to that instance, you can do this: SHARE_URL=https://ips.example.com ruby my_benchmark.rb

Advanced Statistics

By default, the margin of error shown is plus-minus one standard deviation. If a more advanced statistical test is wanted, a bootstrap confidence interval can be calculated instead. A bootstrap confidence interval has the advantages of arguably being more mathematically sound for this application than a standard deviation, it additionally produces an error for relative slowdowns, which the standard deviation does not, and it is arguably more intuitive and actionable.

When a bootstrap confidence interval is used, a median of the interval is used rather than the mean of the samples, which is what you get with the default standard deviation.

The bootstrap confidence interval used is the one described by Tomas Kalibera. Note that for this technique to be valid your benchmark should have reached a non-periodic steady state with statistically independent samples (it should have warmed up) by the time measurements start.

Using a bootstrap confidence internal requires that the 'kalibera' gem is installed separately. This gem is not a formal dependency, as by default it is not needed.

gem install kalibera
Benchmark.ips do |x|

  # The default is :stats => :sd, which doesn't have a configurable confidence
  x.config(:stats => :bootstrap, :confidence => 95)

    # or

  x.stats = :bootstrap
  x.confidence = 95

  # confidence is 95% by default, so it can be omitted

end

Output as JSON

You can generate output in JSON. If you want to write JSON to a file, pass filename to json! method:

Benchmark.ips do |x|
  x.report("some report") {  }
  x.json! 'filename.json'
end

If you want to write JSON to STDOUT, pass STDOUT to json! method and set quiet = true before json!:

Benchmark.ips do |x|
  x.report("some report") {  }
  x.quiet = true
  x.json! STDOUT
end

This is useful when the output from benchmark-ips becomes an input of other tools via stdin.

REQUIREMENTS:

  • None!

INSTALL:

$ gem install benchmark-ips

DEVELOPERS:

After checking out the source, run:

$ rake newb

This task will install any missing dependencies, run the tests/specs, and generate the RDoc.

More Repositories

1

json-patch

A Go library to apply RFC6902 patches and create and apply RFC7386 patches
Go
1,043
star
2

kpeg

A simple PEG library for ruby
Ruby
170
star
3

gx

A set of git tools
Ruby
129
star
4

alexa

Golang interface to the Amazon Alexa Voice service
Go
84
star
5

newrelic-redis

NewRelic instrumentation for redis
Ruby
77
star
6

wildcat

A golang zero-allocation HTTP parser (and eventually http server)
Go
68
star
7

eventd-rfc

A RFC of a syslog replacement
Protocol Buffer
54
star
8

stark

Optimized thrift bindings for ruby
Ruby
52
star
9

benchmark_suite

A set of enhancements to benchmark.rb
Ruby
44
star
10

gemjour

Serve and install gems over Bonjour
Ruby
32
star
11

lost

A ruby wrapper for CoreLocation
Objective-C
29
star
12

ulysses

A thin OS for application goodness
C
28
star
13

distance_between

A RubyMotion App that uses calculates the distance between 2 locations
Ruby
24
star
14

Gauge

A live status viewer for Rubinius
Ruby
22
star
15

prattle

A simple smalltalk frontend to Rubinius
Ruby
21
star
16

orthrus-ssh

A user authentication system built on SSH's key
Ruby
21
star
17

talon

A syntax engine for a series of languages
Ruby
19
star
18

hear

A PortAudio + GCP Speech2text golang library
Go
16
star
19

marius

A dynamic language experiment with a fun VM
C++
14
star
20

heap_dump

Code to read Rubinius HeapDump format
Ruby
13
star
21

ssh

Fork of go's ssh lib
Go
12
star
22

remoteenv

A POC for patching getenv to fetch values from Consul
C
11
star
23

wal

A WAL primitive for Golang
Go
10
star
24

yoke

A VirtualBox fork for the modern world
C
10
star
25

mesh

A Peer to Peer networking package for Go
Go
9
star
26

columbia

WebAssembly based Linux compatible Runtime
WebAssembly
9
star
27

irccat

irccat is like `cat`, but here, the STDOUT is an IRC channel.
Ruby
9
star
28

harq

A simple, high speed message queue with optional message durability
C++
9
star
29

webui

A platform independent wrapper for creating applications using webviews
C
9
star
30

benchmark.fyi

A place to share benchmarking results
Ruby
9
star
31

schubert

A simple systems/configuration management idea
Ruby
8
star
32

go-secretly

A package for storing and retrieving secrets from files, Vault, AWS Parameter Storage, (etc?)
Go
7
star
33

puma-heroku

Puma plugin for easy integration with Heroku
Ruby
7
star
34

marlowe

A language experiment
Ruby
7
star
35

schain

An alternative to envchain that is cross platform
Go
6
star
36

osx-notify

A tiny, ruby like, wrapper for OS X's notifications
Ruby
6
star
37

go-hclog-slog

An adapter from hclog to log/slog
Go
6
star
38

mesh-vpn

Go
5
star
39

redsun

Ruby
5
star
40

inspeqtor

A older checkout of @mperham's inspeqtor to use for Monit's DMCA comparison.
Go
4
star
41

zodiac-prime

A RAFT consensus implementation
Ruby
4
star
42

m13

A dynamic language experiment
Go
4
star
43

rubygems_fp

Rubygems Future Proof APIs
Ruby
4
star
44

lights

Control Phillips Hue lights in Go
Go
4
star
45

ulysses-libc

The libc to go along with the ulysses kernel
C
4
star
46

rivetdb

A simple key/value database backed by log merging
Go
4
star
47

evanphx.github.com

My page
3
star
48

hclogr

Adapter for hclog to the logr protocol
Go
3
star
49

stark-rack

A rack middleware for thrift services
Ruby
3
star
50

party.to

Ruby
3
star
51

dotfiles

Various configuration files
Vim Script
3
star
52

sync

Docker image to sync between 2 directories
Go
3
star
53

go-crypto-dh

Diffie-Hellman algorithm for Go
Go
3
star
54

on_fork

Manager of code to run when a Ruby process forks
Ruby
2
star
55

w3c-css

A CSS spec compliant parser
Ruby
2
star
56

securetunnel

Go
2
star
57

tfe-emp-dev

Terraform Enterprise setup for AWS
HCL
2
star
58

opaqueany

Go
2
star
59

callbox

A twilio app to control my callbox
2
star
60

stark-http

Thrift client protocol for accessing thrift APIs over HTTP
Ruby
2
star
61

evanphx.github.io

blog!
HTML
1
star
62

ficus

LLVM + Lisp!
C++
1
star
63

party.to-website

The party.to website
1
star
64

mesh-shell

Go
1
star
65

pegdown

A markdown parser
Ruby
1
star
66

xlr8r

A turbocharger for ruby 1.8
C
1
star
67

rubinius-website

Rubinius Website
JavaScript
1
star
68

kids-pics

A Toshiba Flashair app and go server
Lua
1
star
69

yfs

A transactional deduping, compressing, encrypting filesystem-like package
Go
1
star
70

pubuser

Go library for fetching basic user and ssh key information from services
Go
1
star
71

blog

The source for my blog
HTML
1
star
72

tree-sitter-hardlight

C
1
star
73

rubygems-lazymirror

A rack app to lazily mirror rubygems infrastructure
Ruby
1
star