• Stars
    star
    783
  • Rank 56,918 (Top 2 %)
  • Language
    Scala
  • License
    Apache License 2.0
  • Created about 10 years ago
  • Updated 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

"Trust no one, bench everything." - sbt plugin for JMH (Java Microbenchmark Harness)

sbt-jmh

Join the chat at https://gitter.im/ktoso/sbt-jmh

SBT plugin for running OpenJDK JMH benchmarks.

JMH about itself:

JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM.

Please read nanotrusting nanotime and other blog posts on micro-benchmarking (or why most benchmarks are wrong) and make sure your benchmark is valid, before you set out to implement your benchmarks.

Versions

Plugin version Default JMH version Notes
0.4.7 (sbt 1.3.0+) 1.37
0.4.6 (sbt 1.3.0+) 1.37
0.4.5 (sbt 1.3.0+) 1.36
0.4.4 (sbt 1.3.0+) 1.36
0.4.3 (sbt 1.3.0+) 1.32
0.4.2 (sbt 1.3.0+) 1.31 JMH -prof async supports 2.x
0.4.1 (sbt 1.3.0+) 1.30
0.4.0 (sbt 1.3.0+) 1.25 profilers now in JMH core
0.3.7 (sbt 0.13.17 / sbt 1.1.4) 1.21 support JDK 11
0.3.6 (sbt 0.13.17 / sbt 1.1.4) 1.21 support JDK 11
0.3.4 (sbt 0.13.17 / sbt 1.1.4) 1.21 support of GraalVM
0.3.3 (sbt 0.13.17 / sbt 1.1.1) 1.20 JMH bugfix release
0.3.2 (sbt 0.13.16 / sbt 1.0) 1.19 minor bugfix release
0.3.1 (sbt 0.13.16 / sbt 1.0) 1.19 minor bugfix release
0.3.0 (sbt 0.13.16 / sbt 1.0) 1.19 async profiler, flame-graphs
... ...

Not interesting versions are skipped in the above listing. Always use the newest which has the JMH version you need. You should stick to the latest version at all times anyway of course.

Adding to your project

Since sbt-jmh is an AutoPlugin all you need to do in order to activate it in your project is to add the below line to your project/plugins.sbt file:

// project/plugins.sbt
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.4.3")

and enable it in the projects where you want to (useful in multi-project builds, as you can enable it only where you need it):

// build.sbt
enablePlugins(JmhPlugin)

If you define your project in a Build.scala, you also need the following import:

import pl.project13.scala.sbt.JmhPlugin

You can read more about auto plugins in sbt on it's documentation page.

Write your benchmarks in src/main/scala. They will be picked up and instrumented by the plugin.

JMH has a very specific way of working (it generates loads of code), so you should prepare a separate project for your benchmarks. In it, just type Jmh/run in order to run your benchmarks. All JMH options work as expected. For help type Jmh/run -h. Another example of running it is:

Jmh/run -i 3 -wi 3 -f1 -t1 .*FalseSharing.*

Which means "3 iterations" "3 warmup iterations" "1 fork" "1 thread". Please note that benchmarks should be usually executed at least in 10 iterations (as a rule of thumb), but more is better.

For "real" results we recommend to at least warm up 10 to 20 iterations, and then measure 10 to 20 iterations again. Forking the JVM is required to avoid falling into specific optimisations (no JVM optimisation is really "completely" predictable)

If your benchmark should be a module in a multimodule project and needs access to another modules test classes then you might want to define your benchmarks in src/test as well (because Intellij does not support "compile->test" dependencies). While this is not directly supported it can be achieved with some tweaks. Assuming the benchmarks live in a module bench and need access to test classes from anotherModule, you have to define this dependency in your build.sbt:

lazy val bench = project
  .dependsOn(anotherModule % "test->test")
  .enablePlugins(JmhPlugin)
  .settings(
     Jmh / sourceDirectory := (Test / sourceDirectory).value,
     Jmh / classDirectory := (Test / classDirectory).value,
     Jmh / dependencyClasspath := (Test / dependencyClasspath).value,
     // rewire tasks, so that 'bench/Jmh/run' automatically invokes 'bench/Jmh/compile' (otherwise a clean 'bench/Jmh/run' would fail)
     Jmh / compile := (Jmh / compile).dependsOn(Test / compile).value,
     Jmh / run := (Jmh / run).dependsOn(Jmh / compile).evaluated
  )

Options

Please invoke Jmh/run -h to get a full list of run as well as output format options.

Useful hint: If you plan to aggregate the collected data you should have a look at the available output formats (-lrf). For example it's possible to keep the benchmark's results as csv or json files for later regression analysis.

Using Java Flight Recorder / async-profiler.

NOTE: sbt-jmh-s integration with async-profiler and Java Flight Recorder has been contributed to the JMH project as of JMH 1.25 and removed from this project. Please migrate to using -prof jfr / -prof async. Use -prof jfr:help / -prof async:help to list available options.

Examples

The examples are scala-fied examples from the original JMH repo, check them out, and run them!

The results will look somewhat like this:

...

[info] # Run progress: 92.86% complete, ETA 00:00:15
[info] # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre/bin/java
[info] # VM options: <none>
[info] # Fork: 1 of 1
[info] # Warmup: 2 iterations, single-shot each
[info] # Measurement: 3 iterations, single-shot each
[info] # Threads: 1 thread, will synchronize iterations
[info] # Benchmark mode: Single shot invocation time
[info] # Benchmark: org.openjdk.jmh.samples.JMHSample_02_BenchmarkModes.measureSingleShot
[info] # Warmup Iteration   1: 100322.000 us
[info] # Warmup Iteration   2: 100556.000 us
[info] Iteration   1: 100162.000 us
[info] Iteration   2: 100468.000 us
[info] Iteration   3: 100706.000 us
[info]
[info] Result : 100445.333 Β±(99.9%) 4975.198 us
[info]   Statistics: (min, avg, max) = (100162.000, 100445.333, 100706.000), stdev = 272.707
[info]   Confidence interval (99.9%): [95470.135, 105420.532]
[info]
[info]
[info] # Run progress: 96.43% complete, ETA 00:00:07
[info] # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre/bin/java
[info] # VM options: <none>
[info] # Fork: 1 of 1
[info] # Warmup: 2 iterations, single-shot each, 5000 calls per batch
[info] # Measurement: 3 iterations, single-shot each, 5000 calls per batch
[info] # Threads: 1 thread, will synchronize iterations
[info] # Benchmark mode: Single shot invocation time
[info] # Benchmark: org.openjdk.jmh.samples.JMHSample_26_BatchSize.measureRight
[info] # Warmup Iteration   1: 15.344 ms
[info] # Warmup Iteration   2: 13.499 ms
[info] Iteration   1: 2.305 ms
[info] Iteration   2: 0.716 ms
[info] Iteration   3: 0.473 ms
[info]
[info] Result : 1.165 Β±(99.9%) 18.153 ms
[info]   Statistics: (min, avg, max) = (0.473, 1.165, 2.305), stdev = 0.995
[info]   Confidence interval (99.9%): [-16.988, 19.317]
[info]
[info]
[info] Benchmark                                                 Mode   Samples         Mean   Mean error    Units
[info] o.o.j.s.JMHSample_22_FalseSharing.baseline               thrpt         3      692.034      179.561   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.baseline:reader        thrpt         3      199.185      185.188   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.baseline:writer        thrpt         3      492.850        7.307   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.contended              thrpt         3      706.532      293.880   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.contended:reader       thrpt         3      210.202      277.801   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.contended:writer       thrpt         3      496.330       78.508   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.hierarchy              thrpt         3     1751.941      222.535   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.hierarchy:reader       thrpt         3     1289.003      277.126   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.hierarchy:writer       thrpt         3      462.938       55.329   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.padded                 thrpt         3     1745.650       83.783   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.padded:reader          thrpt         3     1281.877       47.922   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.padded:writer          thrpt         3      463.773      104.223   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.sparse                 thrpt         3     1362.515      461.782   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.sparse:reader          thrpt         3      898.282      415.388   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.sparse:writer          thrpt         3      464.233       49.958   ops/us

Advanced: Using custom Runners

It is possible to hand over the running of JMH to an App implemented by you, which allows you to programmatically access all test results and modify JMH arguments before you actually invoke it.

To use a custom runner class with runMain, simply use it: Jmh/runMain com.example.MyRunner -i 10 .* – an example for this is available in plugin/src/sbt-test/sbt-jmh/runMain (open the test file).

To replace the runner class which is used when you type Jmh/run, you can set the class in your build file – an example for this is available in plugin/src/sbt-test/sbt-jmh/custom-runner (open the build.sbt file).

Contributing

Yes, pull requests and opening issues is very welcome!

The plugin is maintained at an best-effort basis -- submitting a PR is the best way of getting something done :-)

You can locally publish the plugin with:

sbt 'project plugin; ^publishLocal'

Please test your changes by adding to the [scripted test suite][sbt-jmh/plugin/src/sbt-test/sbt-jmh/] which can be run with:

 sbt 'project plugin; ^scripted'

Special thanks

Special thanks for contributing async-profiler and flame-graphs support and other improvements go to @retronym of Lightbend's Scala team.

More Repositories

1

sbt

sbt, the interactive build tool
Scala
4,685
star
2

sbt-native-packager

sbt Native Packager
Scala
1,590
star
3

sbt-dependency-graph

sbt plugin to create a dependency graph for your project
Scala
1,241
star
4

sbt-eclipse

Plugin for sbt to create Eclipse project definitions
Scala
719
star
5

sbt-release

A release plugin for sbt
Scala
640
star
6

sbt-buildinfo

I know this because build.sbt knows this.
Scala
546
star
7

sbt-web

Library for building sbt plugins for the web
Scala
366
star
8

sbt-git

A git plugin for sbt
Scala
343
star
9

zinc

Scala incremental compiler library, used by sbt and other build tools
Scala
328
star
10

docker-sbt

Official sbt docker images
Dockerfile
313
star
11

sbt-dynver

An sbt plugin to dynamically set your version from git
Scala
297
star
12

sbt-ci-release

sbt plugin to automate Sonatype releases from GitHub Actions
Scala
278
star
13

sbt-onejar

Packages your project using One-JARβ„’
Scala
267
star
14

sbt-scalariform

sbt plugin adding support for source code formatting using Scalariform
Scala
258
star
15

sbt-fresh

sbt-plugin to create an opinionated fresh sbt project
Scala
233
star
16

sbt-github-actions

An sbt plugin which makes it easier to build with GitHub Actions
Scala
194
star
17

sbt-header

sbt-header is an sbt plugin for creating file headers, e.g. copyright headers
Scala
186
star
18

sbt-bintray

fresh packages delivered from your sbt console
Scala
179
star
19

sbt-protobuf

sbt plugin for compiling protobuf files
Scala
174
star
20

sbt-site

Site generation for sbt
Scala
173
star
21

sbt-start-script

SBT Plugin to create a "start" script to run the program
Scala
144
star
22

sbt-pgp

PGP plugin for sbt
Scala
141
star
23

sbt-groll

sbt plugin to roll the Git history
Scala
133
star
24

junit-interface

Implementation of sbt's test interface for JUnit
Java
132
star
25

sbt-unidoc

sbt plugin to create a unified Scaladoc or Javadoc API document across multiple subprojects.
Scala
124
star
26

sbt-jni

SBT Plugin to ease working with JNI
Scala
123
star
27

sbt-jacoco

an sbt plugin for JaCoCo Code Coverage
Scala
123
star
28

sbt-projectmatrix

Scala
118
star
29

sbt-boilerplate

sbt plugin for generating scala.Tuple/Function related boilerplate code
Scala
109
star
30

sbt-proguard

Proguard sbt plugin
Scala
98
star
31

sbt-atmos

sbt plugin for running Typesafe Console in development
Scala
98
star
32

sbt-launcher-package

Packaging for sbt so you can run it.
Scala
89
star
33

sbt-dirty-money

clean Ivy2 cache
Scala
88
star
34

sbt-license-report

Report on licenses used in an sbt project.
Scala
86
star
35

website

The source for scala-sbt.org
Scala
77
star
36

sbt-doge

sbt plugin to aggregate tasks across subprojects and their crossScalaVersions
Scala
77
star
37

sbt-pom-reader

Translates xml -> awesome. Maven-ish support for sbt.
Scala
76
star
38

sbt-remote-control

Create and manage sbt process using unicorns and forks
Scala
74
star
39

sbt-aspectj

AspectJ sbt plugin
Scala
72
star
40

sbt-scalabuff

SBT plugin which generate case classes and support for serialization from Google Protocol Buffer definitions using ScalaBuff
Scala
72
star
41

contraband

http://www.scala-sbt.org/contraband/
Scala
67
star
42

sbt-s3

sbt-s3 is a simple sbt plugin to manipulate objects on Amazon S3
Scala
63
star
43

sbt-multi-jvm

Multi-JVM testing in sbt
Scala
56
star
44

sbt-javaagent

sbt plugin for adding java agents to projects
Scala
53
star
45

sbt-paradox-material-theme

Material Design theme for Paradox
StringTemplate
51
star
46

sbt-cpd

Copy & Paste Detector plugin using PMD for sbt.
Scala
48
star
47

sbt-osgi

sbt plugin for creating OSGi bundles
Scala
47
star
48

sbt-man

Looks up scaladoc.
Scala
46
star
49

sbt-findbugs

FindBugs static analysis plugin for sbt.
Scala
46
star
50

librarymanagement

librarymanagement module for sbt
Scala
44
star
51

sbt-less

Scala
42
star
52

ipcsocket

IPC: Unix Domain Socket and Windows Named Pipes for Java
Java
42
star
53

launcher

The sbt launcher as its own project. Can launch any ivy/maven published project with a main class, with some fancy features.
Scala
41
star
54

io

IO module for sbt
Scala
40
star
55

sbt-js-engine

Support for sbt plugins that use JavaScript
Scala
40
star
56

sbt-autoversion

Scala
36
star
57

sbt-avro

sbt plugin for compiling Avro schemas, similar to sbt-protobuf
Scala
32
star
58

sbt-digest

sbt-web plugin for checksum files
Scala
32
star
59

sbt-jupiter-interface

Implementation of SBT's test interface for JUnit Jupiter
Java
31
star
60

sbt-slash

unified slash syntax for both shell and build.sbt
Scala
29
star
61

sbt-java-formatter

An sbt plugin for formating Java code
Scala
28
star
62

sbt-gzip

sbt-web plugin for gzipping assets
Scala
25
star
63

sbt-unique-version

emulates Maven's uniqueVersion snapshots
Scala
24
star
64

sbt-pull-request-validator

Plugin that optimizes pull request validation to only validate sub projects that have changed
Scala
24
star
65

sbt.github.com

See https://github.com/sbt/website for the source
HTML
22
star
66

sbt-duplicates-finder

Find classes and resources conflicts in your build
Scala
22
star
67

sbt-autoplugin.g8

giter8 template for sbt 0.13.5+ AutoPlugin
Scala
20
star
68

sbt-cucumber

Cucumber plugin for SBT.
Scala
20
star
69

sbt-jcstress

Trust no-one, and especially not memory visibility.
HTML
19
star
70

adept

adept helps you find, declare, and download dependencies. http://groups.google.com/group/adept-dev/
18
star
71

sbt-sriracha

Scala
17
star
72

sbt-mocha

SBT plugin for running mocha JavaScript unit tests on node
Scala
17
star
73

sbt-multi-release-jar

Support for JDK9's Multi Release JAR Files (JEP 238)
Scala
17
star
74

sbt-xjc

SBT plugin to compile an XML Schema with XJC
Scala
15
star
75

util

util modules for sbt
Scala
15
star
76

sbt-export-repo

exports your dependency graph to a preloaded local repository
Scala
15
star
77

sbt-nocomma

sbt-nocomma reduces commas from your build.sbt.
Scala
13
star
78

serialization

serialization facility for sbt
Scala
13
star
79

sbt-maven-resolver

An sbt plugin to resolve dependencies using Aether
Scala
12
star
80

sbt-core-next

sbt APIs targeted for eventual inclusion in sbt core
Scala
12
star
81

sbt-houserules

House rules for sbt modules.
Scala
12
star
82

sbt-pamflet

sbt plugin to run Pamflet (and Pamflet plugin to run sbt)
Scala
11
star
83

sbt-sdlc

Scaladoc link checker for sbt
Scala
11
star
84

sbt-appbundle

A plugin for the simple-build-tool to create an OS X application bundle.
Scala
10
star
85

bintry

your packages, delivered fresh
Scala
10
star
86

sbt-fmpp

FreeMarker Scala/Java Templating Plugin for SBT
Scala
9
star
87

sbt-ynolub

Scala
9
star
88

sbt-testng

Implementation of the sbt testing interface for TestNG, bundled with an sbt plug-in for convenience.
Scala
9
star
89

sbt-concat

sbt-web plugin for concatenating web assets
Scala
8
star
90

sbt-ant

SBT plug-in to call Ant targets from within SBT builds
Scala
7
star
91

sbtn-dist

Shell
6
star
92

sbt-community-plugins

All community plugins that opt into an uber-build
Scala
6
star
93

sbt-vimquit

an sbt plugin that adds :q command.
Scala
5
star
94

helloworld-one

An example build for sbt 1.0.0.
Scala
5
star
95

sbt-giter8-resolver

Scala
5
star
96

sbt-sequential

adds sequential tasks to sbt
Scala
4
star
97

sbt-scalashim

generates sys.error.
Scala
4
star
98

sbt-experimental

Experimental APIs to fix rough edges in sbt
Scala
3
star
99

sbt-web-build-base

Scala
3
star
100

sbt-validator

Builds sbt 1.0.x against recent versions of the sbt modules
Shell
3
star