Metabench 
A simple framework for compile-time microbenchmarks
Overview
Metabench is a single, self-contained CMake module making it easy to create compile-time microbenchmarks. Compile-time benchmarks measure the performance of compiling a piece of code instead of measuring the performance of running it, as regular benchmarks do. The micro part in microbenchmark means that Metabench can be used to benchmark precise parts of a C++ file, such as the instantiation of a single function. Writing benchmarks of this kind is very useful for C++ programmers writing metaprogramming-heavy libraries, which are known to cause long compilation times. Metabench was designed to be very simple to use, while still allowing fairly complex benchmarks to be written.
Metabench is also a collection of compile-time microbenchmarks written using
the metabench.cmake module. The benchmarks measure the compile-time performance
of various algorithms provided by different metaprogramming libraries. The
benchmarks are updated nightly with the latest version of each library, and
the results are published at http://metaben.ch.
Requirements
Metabench requires CMake 3.1 or higher and Ruby 2.1 or higher. Metabench is known to work with CMake's Unix Makefiles and Ninja generators.
Usage
To use Metabench, make sure you have the dependencies listed above and simply
drop the metabench.cmake file somewhere in your CMake search path for modules.
Then, use include(metabench) to include the module in your CMake file, add
individual datasets to be benchmarked using metabench_add_dataset, and finally
specify which datasets should be put together into a chart via metabench_add_chart.
For example, a minimal CMake file using Metabench would look like:
# Make sure Metabench can be found when writing include(metabench)
list(APPEND CMAKE_MODULE_PATH "path/to/metabench/directory")
# Actually include the module
include(metabench)
# Add new datasets
metabench_add_dataset(dataset1 "path/to/dataset1.cpp.erb" "[1, 5, 10]")
metabench_add_dataset(dataset2 "path/to/dataset2.cpp.erb" "(1...15)")
metabench_add_dataset(dataset3 "path/to/dataset3.cpp.erb" "(1...20).step(5)")
# Add a new chart
metabench_add_chart(chart DATASETS dataset1 dataset2 dataset3)This will create a target named chart, which, when run, will gather benchmark
data from each dataset and output JSON files for easy integration with other
tools. A HTML file is generated for easy visualization of the datasets as a
NVD3 chart. To understand what the path/to/datasetN.cpp.erb files are,
read what follows.
The principle
Benchmarking the compilation time of a single .cpp file is rather useless,
because one could simply run the compiler and time that single execution instead.
What is really useful is to have a means of running variations of the same
.cpp file automatically. For example, we might be interested in benchmarking
the compilation time for creating a std::tuple with many elements in it. To
do so, we could write the following test case:
#include <tuple>
int main() {
auto tuple = std::make_tuple(1, 2, 3, 4, 5);
}We would run the compiler and time the compilation, and then change the test case by augmenting the number of elements in the tuple:
#include <tuple>
int main() {
auto tuple = std::make_tuple(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
}We would measure the compilation time for this file, and repeat the process
until satisfactory data has been gathered. This tedious task of generating
different (but obviously related) .cpp files and running the compiler to
gather timings is what Metabench automates. It does this by taking a .cpp.erb
file written using the ERB template system, and generating a family of .cpp
files from that template. It then compiles these .cpp files and gathers
benchmark data from these compilations.
Concretely, you start by writing a .cpp.erb file (say std_tuple.cpp.erb)
that may contain ERB markup:
#include <tuple>
int main() {
auto tuple = std::make_tuple(<%= (1..n).to_a.join(', ') %>);
}Code contained inside <%= ... %> is just normal Ruby code. When the file
will be rendered, the contents of <%= ... %> will be replaced with the
result of evaluating this Ruby code, which will look like:
#include <tuple>
int main() {
auto tuple = std::make_tuple(1, 2, 3, ..., n);
}The ERB markup language has many other features; we encourage readers to take
a look at the Wikipedia page. What happens is that Metabench will
generate a .cpp file for different values of n, and will gather benchmark
data for each of these values. Now, this isn't the whole story. More often
than not, we're only interested in benchmarking part of a C++ file. Indeed,
if we benchmark the whole file in our example above, we'll end up measuring
the time required to #include the <tuple> header in addition to the time
required for creating the std::tuple. While this might be negligible in our
example, this situation arises in nontrivial examples, and would make the
resulting data nearly worthless. Hence, we have to tell Metabench what part(s)
of the file it should measure. This is done by guarding the relevant part(s)
of the code with a preprocessor #if:
#include <tuple>
int main() {
#if defined(METABENCH)
auto tuple = std::make_tuple(<%= (1..n).to_a.join(', ') %>);
#endif
}What Metabench will actually do is compile the file once with the macro defined (and hence with the content of the block), and once without it. It will then subtract the time for compiling the file without the content of the block to the time for compiling the whole file, which should represent a good approximation of the time for compiling what's inside the block.
On the C++ side of things, the .cpp file will be compiled (to benchmark it)
as if it were located in the directory containing the .cpp.erb file, so that
relative include paths can be used. Furthermore, it will be compiled as if the
.cpp file were part of a CMake executable added in the same directory as the
call to metabench_add_dataset. This way, any variable or property set in CMake
will also apply when benchmarking the file. In other words, Metabench tries to
create the illusion that the code is actually compiled as if it were written
in the .cpp.erb file.
This is it for the basic usage of the module! The example/ directory contains
a fully working example of using Metabench to create benchmarks. For a more
involved example, you can take a look at the benchmark suite in the benchmark/
directory. Note that only the most basic usage of Metabench was covered here.
To know all the features provided by the module, you should read the reference
documentation provided as comments inside the CMake module.
A note on benchmark resolution
Like any measurement tool, Metabench has a limited resolution. For example, when
the code being measured (inside the #ifdef METABENCH/#endif pair) takes only
a few milliseconds to compile, the timings reported by Metabench may be completely
inside the noise. Typically, the resolution of timings taken by Metabench is
similar to that of the time command. A good technique to make sure the results
of a benchmark are not inside the noise is to reduce the relative uncertainty
of the measurement. This can be done by increasing the total compilation time
of the measured block, by repeating the same thing (or a similar one) multiple
times:
#include <tuple>
int main() {
#if defined(METABENCH)
auto tuple1 = std::make_tuple(<%= (1..n).to_a.join(', ') %>);
auto tuple2 = std::make_tuple(<%= (1..n).to_a.join(', ') %>);
auto tuple3 = std::make_tuple(<%= (1..n).to_a.join(', ') %>);
auto tuple4 = std::make_tuple(<%= (1..n).to_a.join(', ') %>);
#endif
}History
Metabench was initially developed inside the Boost.Hana library as a mean to benchmark compile-time algorithms. After seeing that a self-standing framework would be useful to the general C++ community, it was decided to extract it into its own project.
License
Please see LICENSE.md.