• Stars
    star
    242
  • Rank 162,901 (Top 4 %)
  • Language
    R
  • License
    Other
  • Created about 6 years ago
  • Updated 23 days ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

High Precision Timing of R Expressions

bench

CRAN status R-CMD-check Codecov test coverage

The goal of bench is to benchmark code, tracking execution time, memory allocations and garbage collections.

Installation

You can install the release version from CRAN with:

install.packages("bench")

Or you can install the development version from GitHub with:

# install.packages("pak")
pak::pak("r-lib/bench")

Features

bench::mark() is used to benchmark one or a series of expressions, we feel it has a number of advantages over alternatives.

  • Always uses the highest precision APIs available for each operating system (often nanoseconds).
  • Tracks memory allocations for each expression.
  • Tracks the number and type of R garbage collections per expression iteration.
  • Verifies equality of expression results by default, to avoid accidentally benchmarking inequivalent code.
  • Has bench::press(), which allows you to easily perform and combine benchmarks across a large grid of values.
  • Uses adaptive stopping by default, running each expression for a set amount of time rather than for a specific number of iterations.
  • Expressions are run in batches and summary statistics are calculated after filtering out iterations with garbage collections. This allows you to isolate the performance and effects of garbage collection on running time (for more details see Neal 2014).

The times and memory usage are returned as custom objects which have human readable formatting for display (e.g. 104ns) and comparisons (e.g. x$mem_alloc > "10MB").

There is also full support for plotting with ggplot2 including custom scales and formatting.

Usage

bench::mark()

Benchmarks can be run with bench::mark(), which takes one or more expressions to benchmark against each other.

library(bench)
set.seed(42)

dat <- data.frame(
  x = runif(10000, 1, 1000), 
  y = runif(10000, 1, 1000)
)

bench::mark() will throw an error if the results are not equivalent, so you don’t accidentally benchmark inequivalent code.

bench::mark(
  dat[dat$x > 500, ],
  dat[which(dat$x > 499), ],
  subset(dat, x > 500)
)
#> Error: Each result must equal the first result:
#> `dat[dat$x > 500, ]` does not equal `dat[which(dat$x > 499), ]`

Results are easy to interpret, with human readable units.

bnch <- bench::mark(
  dat[dat$x > 500, ],
  dat[which(dat$x > 500), ],
  subset(dat, x > 500)
)
bnch
#> # A tibble: 3 × 6
#>   expression                     min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr>                <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 dat[dat$x > 500, ]           277µs    383µs     2485.     377KB     16.3
#> 2 dat[which(dat$x > 500), ]    203µs    276µs     3635.     260KB     16.9
#> 3 subset(dat, x > 500)         361µs    487µs     1981.     510KB     16.8

By default the summary uses absolute measures, however relative results can be obtained by using relative = TRUE in your call to bench::mark() or calling summary(relative = TRUE) on the results.

summary(bnch, relative = TRUE)
#> # A tibble: 3 × 6
#>   expression                  min median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr>                <dbl>  <dbl>     <dbl>     <dbl>    <dbl>
#> 1 dat[dat$x > 500, ]         1.36   1.39      1.25      1.45     1   
#> 2 dat[which(dat$x > 500), ]  1      1         1.84      1        1.03
#> 3 subset(dat, x > 500)       1.78   1.77      1         1.96     1.03

bench::press()

bench::press() is used to run benchmarks against a grid of parameters. Provide setup and benchmarking code as a single unnamed argument then define sets of values as named arguments. The full combination of values will be expanded and the benchmarks are then pressed together in the result. This allows you to benchmark a set of expressions across a wide variety of input sizes, perform replications and other useful tasks.

set.seed(42)

create_df <- function(rows, cols) {
  out <- replicate(cols, runif(rows, 1, 100), simplify = FALSE)
  out <- setNames(out, rep_len(c("x", letters), cols))
  as.data.frame(out)
}

results <- bench::press(
  rows = c(1000, 10000),
  cols = c(2, 10),
  {
    dat <- create_df(rows, cols)
    bench::mark(
      min_iterations = 100,
      bracket = dat[dat$x > 500, ],
      which = dat[which(dat$x > 500), ],
      subset = subset(dat, x > 500)
    )
  }
)
#> Running with:
#>    rows  cols
#> 1  1000     2
#> 2 10000     2
#> 3  1000    10
#> 4 10000    10

results
#> # A tibble: 12 × 8
#>    expression  rows  cols      min   median `itr/sec` mem_alloc `gc/sec`
#>    <bch:expr> <dbl> <dbl> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#>  1 bracket     1000     2     27µs     34µs    27964.   15.84KB     19.6
#>  2 which       1000     2   25.7µs   33.4µs    29553.    7.91KB     17.7
#>  3 subset      1000     2   45.9µs   58.2µs    16793.    27.7KB     17.1
#>  4 bracket    10000     2   64.1µs   70.8µs    13447.  156.46KB     40.5
#>  5 which      10000     2   46.7µs   54.7µs    17586.   78.23KB     23.3
#>  6 subset     10000     2  116.2µs  132.1µs     7228.  273.79KB     40.9
#>  7 bracket     1000    10   77.2µs   85.4µs    11335.   47.52KB     19.9
#>  8 which       1000    10   67.8µs   75.2µs    13073.    7.91KB     23.2
#>  9 subset      1000    10   84.7µs  107.5µs     9281.   59.38KB     18.8
#> 10 bracket    10000    10  130.2µs  169.1µs     5799.   469.4KB     52.2
#> 11 which      10000    10   75.1µs     96µs    10187.   78.23KB     17.4
#> 12 subset     10000    10  222.7µs    253µs     3810.  586.73KB     43.3

Plotting

ggplot2::autoplot() can be used to generate an informative default plot. This plot is colored by gc level (0, 1, or 2) and faceted by parameters (if any). By default it generates a beeswarm plot, however you can also specify other plot types (jitter, ridge, boxplot, violin). See ?autoplot.bench_mark for full details.

ggplot2::autoplot(results)

You can also produce fully custom plots by un-nesting the results and working with the data directly.

library(tidyverse)

results %>%
  unnest(c(time, gc)) %>%
  filter(gc == "none") %>%
  mutate(expression = as.character(expression)) %>%
  ggplot(aes(x = mem_alloc, y = time, color = expression)) +
  geom_point() +
  scale_color_bench_expr(scales::brewer_pal(type = "qual", palette = 3))

system_time()

bench also includes system_time(), a higher precision alternative to system.time().

bench::system_time({ 
  i <- 1
  while(i < 1e7) {
    i <- i + 1
  }
})
#> process    real 
#>   222ms   223ms

bench::system_time(Sys.sleep(.5))
#> process    real 
#>    88µs   502ms

Alternatives

More Repositories

1

devtools

Tools to make an R developer's life easier
R
2,360
star
2

lintr

Static Code Analysis for R
R
1,158
star
3

httr

httr: a friendly http package for R
R
982
star
4

actions

GitHub Actions for the R community
TypeScript
917
star
5

testthat

An R 📦 to make testing 😀
R
867
star
6

usethis

Set up commonly used 📦 components
R
821
star
7

pkgdown

Generate static html documentation for an R package
R
702
star
8

styler

Non-invasive pretty printing of R code
R
692
star
9

pak

A fresh approach to package installation
C
630
star
10

cli

Tools for making beautiful & useful command line interfaces
R
618
star
11

roxygen2

Generate R package documentation from inline R comments
R
579
star
12

rig

The R Installation Manager
Rust
555
star
13

rlang

Low-level API for programming with R
R
484
star
14

progress

Progress bar in your R terminal
R
457
star
15

R6

Encapsulated object-oriented programming for R
R
402
star
16

here

A simpler way to find your files
R
401
star
17

scales

Tools for ggplot2 scales
R
386
star
18

fs

Provide cross platform file operations based on libuv.
C
361
star
19

rex

Friendly regular expressions for R.
R
331
star
20

covr

Test coverage reports for R
R
329
star
21

crayon

🖍️ R package for colored terminal output — now superseded by cli
R
322
star
22

remotes

Install R packages from GitHub, GitLab, Bitbucket, git, svn repositories, URLs
R
321
star
23

memoise

Easy memoisation for R
R
314
star
24

lobstr

Understanding complex R objects with tools similar to str()
R
296
star
25

slider

Sliding Window Functions
R
290
star
26

callr

Call R from R
R
289
star
27

vctrs

Generic programming with typed R vectors
C
280
star
28

waldo

Find differences between R objects
R
275
star
29

zeallot

Variable assignment with zeal! (or multiple, unpacking, and destructuring assignment in R)
R
250
star
30

conflicted

An alternative conflict resolution strategy for R
R
243
star
31

gmailr

Access the Gmail RESTful API from R.
R
230
star
32

processx

Execute and Control Subprocesses from R
R
227
star
33

httr2

Make HTTP requests and process their responses. A modern reimagining of httr.
R
223
star
34

asciicast

Turn R scripts into terminal screencasts
R
223
star
35

xml2

Bindings to libxml2
R
218
star
36

gh

Minimalistic GitHub API client in R
R
217
star
37

cpp11

cpp11 helps you to interact with R objects using C++ code.
C++
192
star
38

keyring

🔐 Access the system credential store from R
R
188
star
39

vdiffr

Visual regression testing and graphical diffing with testthat
C++
181
star
40

svglite

A lightweight svg graphics device for R
C++
178
star
41

pillar

Format columns with colour
R
173
star
42

ragg

Graphic Devices Based on AGG
C++
170
star
43

withr

Methods For Temporarily Modifying Global State
R
169
star
44

hugodown

Make websites with hugo and RMarkdown
R
165
star
45

ymlthis

write YAML for R Markdown, bookdown, blogdown, and more
R
163
star
46

coro

Coroutines for R
R
150
star
47

rprojroot

Finding files in project subdirectories
R
147
star
48

debugme

Easy and efficient debugging for R packages
R
145
star
49

available

Check if a package name is available to use
R
142
star
50

archive

R bindings to libarchive, supporting a large variety of archive formats
C++
141
star
51

ellipsis

Tools for Working with ...
R
139
star
52

later

Schedule an R function or formula to run after a specified period of time.
C++
136
star
53

gert

Simple git client for R
C
136
star
54

itdepends

R
133
star
55

rray

Simple Arrays
R
130
star
56

isoband

isoband: An R package to generate contour lines and polygons.
C++
130
star
57

prettyunits

Pretty, human readable formatting of quantities
JavaScript
128
star
58

fastmap

Fast map implementation for R
C++
127
star
59

desc

Manipulate DESCRIPTION files
R
121
star
60

tidyselect

A backend for functions taking tidyverse selections
R
121
star
61

gargle

Infrastructure for calling Google APIs from R, including auth
R
113
star
62

rcmdcheck

Run R CMD check from R and collect the results
R
109
star
63

evaluate

A version of eval for R that returns more information about what happened
R
108
star
64

prettycode

Syntax highlight R code in the terminal
R
100
star
65

revdepcheck

R package reverse dependency checking
R
99
star
66

sloop

S language OOP ⛵️
R
99
star
67

mockery

A mocking library for R.
R
98
star
68

tree-sitter-r

C
95
star
69

clock

A Date-Time Library for R
R
95
star
70

pkgdepends

R Package Dependency Resolution
R
93
star
71

systemfonts

System Native Font Handling in R
C++
91
star
72

lifecycle

Manage the life cycle of your exported functions and arguments
R
90
star
73

gtable

The layout packages that powers ggplot2
R
85
star
74

askpass

Password Entry for R, Git, and SSH
R
83
star
75

commonmark

High Performance CommonMark and Github Markdown Rendering in R
C
83
star
76

zip

Platform independent zip compression via miniz
C
82
star
77

rappdirs

Find OS-specific directories to store data, caches, and logs. A port of python's AppDirs
R
81
star
78

downlit

Syntax Highlighting and Automatic Linking
R
81
star
79

clisymbols

Unicode symbols for CLI applications, with fallbacks
R
76
star
80

ps

R package to query, list, manipulate system processes
C
72
star
81

sessioninfo

Print Session Information
R
72
star
82

pkgapi

Create a map of functions for an R package - WORK IN PROGRESS!
R
70
star
83

credentials

Tools for Managing SSH and Git Credentials
R
70
star
84

sodium

R bindings to libsodium
R
68
star
85

roxygen2md

Convert elements of roxygen documentation to markdown
R
67
star
86

backports

Reimplementations of Functions Introduced Since R-3.0.0
R
65
star
87

pkgbuild

Find tools needed to build R packages
R
65
star
88

webfakes

Fake web apps for HTTP testing R packages
C
61
star
89

cliapp

Rich Command Line Applications
R
60
star
90

generics

Common generic methods
R
59
star
91

diffviewer

HTML widget to visually compare files
JavaScript
57
star
92

liteq

Serverless R message queue using SQLite
R
56
star
93

pkgload

Simulate installing and loading a package
R
54
star
94

cachem

Key-value caches for R
R
53
star
95

carrier

Create standalone functions for remote execution
R
50
star
96

brio

Basic R Input Output
R
50
star
97

marquee

Markdown Parser and Renderer for R Graphics
C
49
star
98

jose

Javascript Object Signing and Encryption for R
R
48
star
99

urlchecker

Run CRAN URL checks from older versions of R
R
45
star
100

pkgconfig

Private configuration for R packages
R
41
star