• Stars
    star
    766
  • Rank 59,299 (Top 2 %)
  • Language
    C++
  • License
    Boost Software Li...
  • Created about 5 years ago
  • Updated over 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

UT: C++20 μ(micro)/Unit Testing Framework

Version Linux MacOs Windows Coveralls Try it online AUR Badge

"If you liked it then you "should have put a"_test on it", Beyonce rule

UT / μt

| Motivation | Quick Start | Overview | Tutorial | Examples | User Guide | FAQ | Benchmarks |

C++ single header/single module, macro-free μ(micro)/Unit Testing Framework

#include <boost/ut.hpp> // import boost.ut;

constexpr auto sum(auto... values) { return (values + ...); }

int main() {
  using namespace boost::ut;

  "sum"_test = [] {
    expect(sum(0) == 0_i);
    expect(sum(1, 2) == 3_i);
    expect(sum(1, 2) > 0_i and 41_i == sum(40, 2));
  };
}
Running "sum"...
  sum.cpp:11:FAILED [(3 > 0 and 41 == 42)]
FAILED

===============================================================================
tests:   1 | 1 failed
asserts: 3 | 2 passed | 1 failed

https://godbolt.org/z/f4jEcv9vo

Motivation

Testing is a very important part of the Software Development, however, C++ doesn't provide any good testing facilities out of the box, which often leads into a poor testing experience for develops and/or lack of tests/coverage in general.

One should treat testing code as production code!

Additionally, well established testing practises such as Test Driven Development (TDD)/Behaviour Driven Development (BDD) are often not followed due to the same reasons.

The following snippet is a common example of testing with projects in C++.

int main() {
  // should sum numbers
  {
    assert(3 == sum(1, 2));
  }
}

There are quite a few problems with the approach above

  • No names for tests (Hard to follow intentions by further readers)
  • No automatic registration of tests (No way to run specific tests)
  • Hard to debug (Assertions don't provide any information why it failed)
  • Hard to scale (No easy path forward for parameterized tests, multiple suites, parallel execution, etc...)
  • Hard to integrate (No easy way to have a custom output such as XML for CI integration)
  • Easy to make mistakes (With implicit casting, floating point comparison, pointer comparison for strings, etc...)
  • Hard to follow good practises such as TDD/BDD (Lack of support for sections and declarative expressions)
  • ...

UT is trying to address these issues by simplifying testing experience with a few simple steps:

And you good to go!

Okay, great, but why I would use UT over other/similar testing frameworks already available in C++?

Great question! There are a few unique features which makes UT worth trying

  • Firstly, it supports all the basic Unit Testing Framework features (automatic registration of tests, assertions, suites, etc...)
  • It's easy to integrate (it's just one header/module)
  • It's macro free which makes testing experience that much nicer (it uses modern C++ features instead, macros are opt-in rather than being compulsory - Can I still use macros?)
  • It's flexible (all parts of the framework such as: runner, reporter, printer can be customized, basically most other Unit Testing Frameworks can be implemented on top of UT primitives)
  • It has smaller learning curve (just a few simple concepts (expect, test, suite))
  • It leverages C++ features to support more complex testing (parameterized)
  • It's faster to compile and execute than similar frameworks which makes it suitable for bigger projects without additional hassle (Benchmarks)
  • It supports TDD/BDD workflows
  • It supports Gherkin specification
  • It supports Spec
  • ...

Sounds intriguing/interesting? Learn more at

Quick Start

https://bit.ly/ut-quick-start (slides)

Overview

Tutorial

    Step 0: Get it...

Get the latest latest header/module from here!

Include/Import

// #include <boost/ut.hpp> // single header
// import boost.ut;        // single module (C++20)

int main() { }

Compile & Run

$CXX main.cpp && ./a.out
All tests passed (0 assert in 0 test)

[Optional] Install it

cmake -Bbuild -H.
cd build && make         # run tests
cd build && make install # install

[Optional] CMake integration

This project provides a CMake config and target. Just load ut with find_package to import the Boost::ut target. Linking against this target will add the necessary include directory for the single header file. This is demonstrated in the following example.

find_package(ut REQUIRED)
add_library(my_test my_test.cpp)
target_link_libraries(my_test PRIVATE Boost::ut)

[Optional] Conan integration

The boost-ext-ut package is available from Conan Center. Just include it in your project's Conanfile with boost-ext-ut/1.1.9.

    Step 1: Expect it...

Let's write our first assertion, shall we?

int main() {
  boost::ut::expect(true);
}
All tests passed (1 asserts in 0 test)

https://godbolt.org/z/vfx-eB

Okay, let's make it fail now?

int main() {
  boost::ut::expect(1 == 2);
}
main.cpp:4:FAILED [false]
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/7qTePx

Notice that expression 1 == 2 hasn't been printed. Instead we got false?

Let's print it then?

int main() {
  using namespace boost::ut;
  expect(1_i == 2);
}
main.cpp:4:FAILED [1 == 2]
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/7MXVzu

Okay, now we have it! 1 == 2 has been printed as expected. Notice the User Defined Literal (UDL) 1_i was used. _i is a compile-time constant integer value

  • It allows to override comparison operators 👍
  • It disallow comparison of different types 👍

See the User-guide for more details.

Alternatively, a terse notation (no expect required) can be used.

int main() {
  using namespace boost::ut::literals;
  using namespace boost::ut::operators::terse;

  1_i == 2; // terse notation
}
main.cpp:7:FAILED [1 == 2]
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/s77GSm

Other expression syntaxes are also available.

expect(1_i == 2);       // UDL syntax
expect(1 == 2_i);       // UDL syntax
expect(that % 1 == 2);  // Matcher syntax
expect(eq(1, 2));       // eq/neq/gt/ge/lt/le
main.cpp:6:FAILED [1 == 2]
main.cpp:7:FAILED [1 == 2]
main.cpp:8:FAILED [1 == 2]
main.cpp:9:FAILED [1 == 2]
===============================================================================
tests:   0 | 0 failed
asserts: 4 | 0 passed | 4 failed

https://godbolt.org/z/QbgGtc

Okay, but what about the case if my assertion is fatal. Meaning that the program will crash unless the processing will be terminated. Nothing easier, let's just add >> fatal after the expected expression to make it fatal.

expect((1 == 2_i) >> fatal); // fatal assertion
expect(1_i == 2);            // not executed
main.cpp:6:FAILED [1 == 2]
===============================================================================
tests:   1 | 1 failed
asserts: 2 | 0 passed | 2 failed

https://godbolt.org/z/WMe8Y1

But my expression is more complex than just simple comparisons. Not a problem, logic operators are also supported in the expect 👍.

expect(42l == 42_l and 1 == 2_i); // compound expression
main.cpp:5:FAILED [(42 == 42 and 1 == 2)]
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/aEhX4t

Can I add a custom message though? Sure, expect calls are streamable!

int main() {
  expect(42l == 42_l and 1 == 2_i) << "additional info";
}
main.cpp:5:FAILED [(42 == 42 and 1 == 2)] additional info
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

That's nice, can I use custom messages and fatal assertions? Yes, stream the fatal!

expect(1 == 2_i) << "fatal assertion" << fatal;
expect(1_i == 2);
FAILED
in: main.cpp:6 - test condition:  [1 == 2]

 fatal assertion
===============================================================================
tests:   0 | 2 failed
asserts: 0 | 0 passed | 2 failed

I use std::expected, can I stream its error() upon failure? Yes, since std::expected's error() can only be called when there is no value it requires lazy evaluation.

lazy log"_test = [] {
  std::expected<bool, std::string> e = std::unexpected("lazy evaluated");
  expect(e.has_value()) << [&] { return e.error(); } << fatal;
  expect(e.value() == true);
};
Running test "lazy log"... FAILED
in: main.cpp:12 - test condition:  [false]

 lazy evaluated
===============================================================================
tests:   1 | 2 failed
asserts: 0 | 0 passed | 2 failed

> https://godbolt.org/z/v2PDuU

</p>
</details>

<details open><summary>&nbsp;&nbsp;&nbsp;&nbsp;Step 2: Group it...</summary>
<p>

> Assertions are great, but how to combine them into more cohesive units?
> `Test cases` are the way to go! They allow to group expectations for the same functionality into coherent units.

```cpp
"hello world"_test = [] { };

Alternatively test("hello world") = [] {} can be used.

All tests passed (0 asserts in 1 tests)

https://godbolt.org/z/Bh-EmY

Notice 1 tests but 0 asserts.

Let's make our first end-2-end test case, shall we?

int main() {
  "hello world"_test = [] {
    int i = 43;
    expect(42_i == i);
  };
}
Running "hello world"...
  main.cpp:8:FAILED [42 == 43]
FAILED
===============================================================================
tests:   1 | 1 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/Y43mXz

👍 We are done here!

I'd like to nest my tests, though and share setup/tear-down. With lambdas used to represents tests/sections we can easily achieve that. Let's just take a look at the following example.

int main() {
  "[vector]"_test = [] {
    std::vector<int> v(5);

    expect((5_ul == std::size(v)) >> fatal);

    should("resize bigger") = [v] { // or "resize bigger"_test
      mut(v).resize(10);
      expect(10_ul == std::size(v));
    };

    expect((5_ul == std::size(v)) >> fatal);

    should("resize smaller") = [=]() mutable { // or "resize smaller"_test
      v.resize(0);
      expect(0_ul == std::size(v));
    };
  }
}
All tests passed (4 asserts in 1 tests)

https://godbolt.org/z/XWAdYt

Nice! That was easy, but I'm a believer into Behaviour Driven Development (BDD). Is there a support for that? Yes! Same example as above just with the BDD syntax.

int main() {
  "vector"_test = [] {
    given("I have a vector") = [] {
      std::vector<int> v(5);
      expect((5_ul == std::size(v)) >> fatal);

      when("I resize bigger") = [=] {
        mut(v).resize(10);

        then("The size should increase") = [=] {
          expect(10_ul == std::size(v));
        };
      };
    };
  };
}
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/dnvxsE

On top of that, feature/scenario aliases can be leveraged.

int main() {
  feature("vector") = [] {
    scenario("size") = [] {
      given("I have a vector") = [] {
        std::vector<int> v(5);
        expect((5_ul == std::size(v)) >> fatal);

        when("I resize bigger") = [=] {
          mut(v).resize(10);

          then("The size should increase") = [=] {
            expect(10_ul == std::size(v));
          };
        };
      };
    };
  };
}
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/T4cWss

Can I use Gherkin? Yeah, let's rewrite the example using Gherkin specification

int main() {
  bdd::gherkin::steps steps = [](auto& steps) {
    steps.feature("Vector") = [&] {
      steps.scenario("*") = [&] {
        steps.given("I have a vector") = [&] {
          std::vector<int> v(5);
          expect((5_ul == std::size(v)) >> fatal);

          steps.when("I resize bigger") = [&] {
            v.resize(10);
          };

          steps.then("The size should increase") = [&] {
            expect(10_ul == std::size(v));
          };
        };
      };
    };
  };

  "Vector"_test = steps |
    R"(
      Feature: Vector
        Scenario: Resize
          Given I have a vector
           When I resize bigger
           Then The size should increase
    )";
}
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/jb1d8P

Nice, is Spec notation supported as well?

int main() {
  describe("vector") = [] {
    std::vector<int> v(5);
    expect((5_ul == std::size(v)) >> fatal);

    it("should resize bigger") = [v] {
      mut(v).resize(10);
      expect(10_ul == std::size(v));
    };
  };
}
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/6jKKzT

That's great, but how can call the same tests with different arguments/types to be DRY (Don't Repeat Yourself)? Parameterized tests to the rescue!

int main() {
  for (auto i : std::vector{1, 2, 3}) {
    test("parameterized " + std::to_string(i)) = [i] { // 3 tests
      expect(that % i > 0); // 3 asserts
    };
  }
}
All tests passed (3 asserts in 3 tests)

https://godbolt.org/z/Utnd6X

That's it 😮! Alternatively, a convenient test syntax is also provided 👍

int main() {
  "args"_test = [](const auto& arg) {
    expect(arg > 0_i) << "all values greater than 0";
  } | std::vector{1, 2, 3};
}
All tests passed (3 asserts in 3 tests)

https://godbolt.org/z/6FHtpq

Check Examples for further reading.

    Step 3: Scale it...

Okay, but my project is more complex than that. How can I scale? Test suites will make that possible. By using suite in translation units tests defined inside will be automatically registered 👍

suite errors = [] {
  "exception"_test = [] {
    expect(throws([] { throw 0; })) << "throws any exception";
  };

  "failure"_test = [] {
    expect(aborts([] { assert(false); }));
  };
};

int main() { }
All tests passed (2 asserts in 2 tests)

https://godbolt.org/z/_ccGwZ


What's next?

Examples

    Assertions

// operators
expect(0_i == sum());
expect(2_i != sum(1, 2));
expect(sum(1) >= 0_i);
expect(sum(1) <= 1_i);
// message
expect(3_i == sum(1, 2)) << "wrong sum";
// expressions
expect(0_i == sum() and 42_i == sum(40, 2));
expect(0_i == sum() or 1_i == sum()) << "compound";
// matchers
expect(that % 0 == sum());
expect(that % 42 == sum(40, 2) and that % (1 + 2) == sum(1, 2));
expect(that % 1 != 2 or 2_i > 3);
// eq/neq/gt/ge/lt/le
expect(eq(42, sum(40, 2)));
expect(neq(1, 2));
expect(eq(sum(1), 1) and neq(sum(1, 2), 2));
expect(eq(1, 1) and that % 1 == 1 and 1_i == 1);
// floating points
expect(42.1_d == 42.101) << "epsilon=0.1";
expect(42.10_d == 42.101) << "epsilon=0.01";
expect(42.10000001 == 42.1_d) << "epsilon=0.1";
// constant
constexpr auto compile_time_v = 42;
auto run_time_v = 99;
expect(constant<42_i == compile_time_v> and run_time_v == 99_i);
// failure
expect(1_i == 2) << "should fail";
expect(sum() == 1_i or 2_i == sum()) << "sum?";
assertions.cpp:53:FAILED [1 == 2] should fail
assertions.cpp:54:FAILED [(0 == 1 or 2 == 0)] sum?
===============================================================================
tests:   0  | 0 failed
asserts: 20 | 18 passed | 2 failed

https://godbolt.org/z/E1c7G5

    Tests

        Run/Skip/Tag

"run UDL"_test = [] {
  expect(42_i == 42);
};

skip / "don't run UDL"_test = [] {
  expect(42_i == 43) << "should not fire!";
};
All tests passed (1 asserts in 1 tests)
1 tests skipped
test("run function") = [] {
  expect(42_i == 42);
};

skip / test("don't run function") = [] {
  expect(42_i == 43) << "should not fire!";
};
All tests passed (1 asserts in 1 tests)
1 tests skipped
tag("nightly") / tag("slow") /
"performance"_test= [] {
  expect(42_i == 42);
};

tag("slow") /
"run slowly"_test= [] {
  expect(42_i == 43) << "should not fire!";
};
cfg<override> = {.tag = {"nightly"}};
All tests passed (1 asserts in 1 tests)
1 tests skipped

https://godbolt.org/z/X3_kG4

        Sections

"[vector]"_test = [] {
  std::vector<int> v(5);

  expect((5_ul == std::size(v)) >> fatal);

  should("resize bigger") = [=] { // or "resize bigger"_test
    mut(v).resize(10);
    expect(10_ul == std::size(v));
  };

  expect((5_ul == std::size(v)) >> fatal);

  should("resize smaller") = [=]() mutable { // or "resize smaller"_test
    v.resize(0);
    expect(0_ul == std::size(v));
  };
};
All tests passed (4 asserts in 1 tests)

https://godbolt.org/z/cE91bj

        Behavior Driven Development (BDD)

"Scenario"_test = [] {
  given("I have...") = [] {
    when("I run...") = [] {
      then("I expect...") = [] { expect(1_i == 1); };
      then("I expect...") = [] { expect(1 == 1_i); };
    };
  };
};
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/mNBySr

        Gherkin

int main() {
  bdd::gherkin::steps steps = [](auto& steps) {
    steps.feature("*") = [&] {
      steps.scenario("*") = [&] {
        steps.given("I have a number {value}") = [&](int value) {
          auto number = value;
          steps.when("I add {value} to it") = [&](int value) {
            number += value;
          };
          steps.then("I expect number to be {value}") = [&](int value) {
            expect(that % number == value);
          };
        };
      };
    };
  };

  "Gherkin"_test = steps |
    R"(
      Feature: Number
        Scenario: Addition
          Given I have a number 40
           When I add 2 to it
           Then I expect number to be 42
    )";
}
All tests passed (1 asserts in 1 tests)

https://godbolt.org/z/BP3hyt

        Spec

int main() {
  describe("equality") = [] {
    it("should be equal")     = [] { expect(0_i == 0); };
    it("should not be equal") = [] { expect(1_i != 0); };
  };
}
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/BXYJ3a

        Parameterized

for (auto i : std::vector{1, 2, 3}) {
  test("parameterized " + std::to_string(i)) = [i] {
    expect(that % i > 0);
  };
}

"args"_test =
   [](auto arg) {
      expect(arg >= 1_i);
    }
  | std::vector{1, 2, 3};

"types"_test =
    []<class T> {
      expect(std::is_integral_v<T>) << "all types are integrals";
    }
  | std::tuple<bool, int>{};

"args and types"_test =
    []<class TArg>(TArg arg) {
      expect(std::is_integral_v<TArg> >> fatal);
      expect(42_i == arg or "is true"_b == arg);
      expect(type<TArg> == type<int> or type<TArg> == type<bool>);
    }
  | std::tuple{true, 42};
All tests passed (14 asserts in 10 tests)

https://godbolt.org/z/4xGGdo

And whenever I need to know the specific type for which the test failed, I can use reflection::type_name<T>(), like this:

"types with type name"_test =
    []<class T>() {
      expect(std::is_unsigned_v<T>) << reflection::type_name<T>() << "is unsigned";
    }
  | std::tuple<unsigned int, float>{};
Running "types with type name"...PASSED
Running "types with type name"...
  <source>:10:FAILED [false] float is unsigned
FAILED

https://godbolt.org/z/MEnGnbTY4

    Suites

namespace ut = boost::ut;

ut::suite errors = [] {
  using namespace ut;

  "throws"_test = [] {
    expect(throws([] { throw 0; }));
  };

  "doesn't throw"_test = [] {
    expect(nothrow([]{}));
  };
};

int main() { }
All tests passed (2 asserts in 2 tests)

https://godbolt.org/z/CFbTP9

    Misc

        Logging using streams

"logging"_test = [] {
  log << "pre";
  expect(42_i == 43) << "message on failure";
  log << "post";
};
Running "logging"...
pre
  logging.cpp:8:FAILED [42 == 43] message on failure
post
FAILED

===============================================================================

tests:   1 | 1 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/26fPSY

        Logging using formatting

This requires using C++20 with a standard library with std::format support.

"logging"_test = [] {
  log("\npre  {} == {}\n", 42, 43);
  expect(42_i == 43) << "message on failure";
  log("\npost {} == {} -> {}\n", 42, 43, 42 == 43);
};
Running "logging"...
pre  42 == 43
  logging.cpp:8:FAILED [42 == 43] message on failure
post 42 == 43 -> false
FAILED

===============================================================================

tests:   1 | 1 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/26fPSY

        Matchers

"matchers"_test = [] {
  constexpr auto is_between = [](auto lhs, auto rhs) {
    return [=](auto value) {
      return that % value >= lhs and that % value <= rhs;
    };
  };

  expect(is_between(1, 100)(42));
  expect(not is_between(1, 100)(0));
};
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/4qwrCi

        Exceptions/Aborts

"exceptions/aborts"_test = [] {
  expect(throws<std::runtime_error>([] { throw std::runtime_error{""}; }))
    << "throws runtime_error";
  expect(throws([] { throw 0; })) << "throws any exception";
  expect(nothrow([]{})) << "doesn't throw";
  expect(aborts([] { assert(false); }));
};
All tests passed (4 asserts in 1 tests)

https://godbolt.org/z/A2EehK

    Config

        Runner

namespace ut = boost::ut;

namespace cfg {
  class runner {
   public:
    template <class... Ts> auto on(ut::events::test<Ts...> test) { test(); }
    template <class... Ts> auto on(ut::events::skip<Ts...>) {}
    template <class TExpr>
    auto on(ut::events::assertion<TExpr>) -> bool { return true; }
    auto on(ut::events::fatal_assertion) {}
    template <class TMsg> auto on(ut::events::log<TMsg>) {}
  };
} // namespace cfg

template<> auto ut::cfg<ut::override> = cfg::runner{};

https://godbolt.org/z/jdg687

        Reporter

namespace ut = boost::ut;

namespace cfg {
  class reporter {
   public:
    auto on(ut::events::test_begin) -> void {}
    auto on(ut::events::test_run) -> void {}
    auto on(ut::events::test_skip) -> void {}
    auto on(ut::events::test_end) -> void {}
    template <class TMsg> auto on(ut::events::log<TMsg>) -> void {}
    template <class TExpr>
    auto on(ut::events::assertion_pass<TExpr>) -> void {}
    template <class TExpr>
    auto on(ut::events::assertion_fail<TExpr>) -> void {}
    auto on(ut::events::fatal_assertion) -> void {}
    auto on(ut::events::exception) -> void {}
    auto on(ut::events::summary) -> void {}
  };
}  // namespace cfg

template <>
auto ut::cfg<ut::override> = ut::runner<cfg::reporter>{};

https://godbolt.org/z/gsAPKg

        Printer

namespace ut = boost::ut;

namespace cfg {
struct printer : ut::printer {
  template <class T>
  auto& operator<<(T&& t) {
    std::cerr << std::forward<T>(t);
    return *this;
  }
};
}  // namespace cfg

template <>
auto ut::cfg<ut::override> = ut::runner<ut::reporter<cfg::printer>>{};

int main() {
  using namespace ut;
  "printer"_test = [] {};
}

https://godbolt.org/z/XCscF9

User Guide

    API

export module boost.ut; /// __cpp_modules

namespace boost::inline ext::ut::inline v1_1_9 {
  /**
   * Represents test suite object
   */
  struct suite final {
    /**
     * Creates and executes test suite
     * @example suite _ = [] {};
     * @param suite test suite function
     */
    constexpr explicit(false) suite(auto suite);
  };

  /**
   * Creates a test
   * @example "name"_test = [] {};
   * @return test object to be executed
   */
  constexpr auto operator""_test;

  /**
   * Creates a test
   * @example test("name") = [] {};
   * @return test object to be executed
   */
  constexpr auto test = [](const auto name);

  /**
   * Creates a test
   * @example should("name") = [] {};
   * @return test object to be executed
   */
  constexpr auto should = [](const auto name);

  /**
   * Behaviour Driven Development (BDD) helper functions
   * @param name step name
   * @return test object to be executed
   */
  constexpr auto given = [](const auto name);
  constexpr auto when  = [](const auto name);
  constexpr auto then  = [](const auto name);

  /**
   * Evaluates an expression
   * @example expect(42 == 42_i and 1 != 2_i);
   * @param expr expression to be evaluated
   * @param source location https://en.cppreference.com/w/cpp/utility/source_location
   * @return stream
   */
  constexpr OStream& expect(
    Expression expr,
    const std::source_location& location = std::source_location::current()
  );

  struct {
    /**
     * @example (that % 42 == 42);
     * @param expr expression to be evaluated
     */
    [[nodiscard]] constexpr auto operator%(Expression expr) const;
  } that{};

  inline namespace literals {
    /**
     * User defined literals to represent constant values
     * @example 42_i, 0_uc, 1.23_d
     */
    constexpr auto operator""_i;  /// int
    constexpr auto operator""_s;  /// short
    constexpr auto operator""_c;  /// char
    constexpr auto operator""_l;  /// long
    constexpr auto operator""_ll; /// long long
    constexpr auto operator""_u;  /// unsigned
    constexpr auto operator""_uc; /// unsigned char
    constexpr auto operator""_us; /// unsigned short
    constexpr auto operator""_ul; /// unsigned long
    constexpr auto operator""_f;  /// float
    constexpr auto operator""_d;  /// double
    constexpr auto operator""_ld; /// long double

    /**
     * Represents dynamic values
     * @example _i(42), _f(42.)
     */
    constexpr auto _b(bool);
    constexpr auto _c(char);
    constexpr auto _s(short);
    constexpr auto _i(int);
    constexpr auto _l(long);
    constexpr auto _ll(long long);
    constexpr auto _u(unsigned);
    constexpr auto _uc(unsigned char);
    constexpr auto _us(unsigned short);
    constexpr auto _ul(unsigned long);
    constexpr auto _f(float);
    constexpr auto _d(double);
    constexpr auto _ld(long double);

    /**
     * Logical representation of constant boolean (true) value
     * @example "is set"_b     : true
     *          not "is set"_b : false
     */
    constexpr auto operator ""_b;
  } // namespace literals

  inline namespace operators {
    /**
     * Comparison functions to be used in expressions
     * @example eq(42, 42), neq(1, 2)
     */
    constexpr auto eq(Operator lhs, Operator rhs);  /// ==
    constexpr auto neq(Operator lhs, Operator rhs); /// !=
    constexpr auto gt(Operator lhs, Operator rhs);  /// >
    constexpr auto ge(Operator lhs, Operator rhs);  /// >=
    constexpr auto lt(Operator lhs, Operator rhs);  /// <
    constexpr auto le(Operator lhs, Operator rhs);  /// <=

    /**
     * Overloaded comparison operators to be used in expressions
     * @example (42_i != 0)
     */
    constexpr auto operator==;
    constexpr auto operator!=;
    constexpr auto operator>;
    constexpr auto operator>=;
    constexpr auto operator<;
    constexpr auto operator<=;

    /**
     * Overloaded logic operators to be used in expressions
     * @example (42_i != 0 and 1 == 2_i)
     */
    constexpr auto operator and;
    constexpr auto operator or;
    constexpr auto operator not;

    /**
     * Executes parameterized tests
     * @example "parameterized"_test = [](auto arg) {} | std::tuple{1, 2, 3};
     */
    constexpr auto operator|;

    /**
     * Creates tags
     * @example tag("slow") / tag("nightly") / "perf"_test = []{};
     */
    constexpr auto operator/;

    /**
     * Creates a `fatal_assertion` from an expression
     * @example (42_i == 0) >> fatal
     */
    constexpr auto operator>>;
  } // namespace operators

  /**
   * Creates skippable test object
   * @example skip / "don't run"_test = [] { };
   */
  constexpr auto skip = tag("skip");

  struct {
    /**
     * @example log << "message!";
     * @param msg stringable message
     */
    auto& operator<<(Msg msg);
  } log{};

  /**
   * Makes object mutable
   * @example mut(object)
   * @param t object to be mutated
   */
  template<class T> auto mut(const T& t) -> T&;

  /**
   * Default execution flow policy
   */
  class runner {
   public:
    /**
     * @example cfg<override> = {
        .filter  = "test.section.*",
        .colors  = { .none = "" },
        .dry__run = true
       };
     * @param options.filter {default: "*"} runs all tests which names
                                            matches test.section.* filter
     * @param options.colors {default: {
                               .none = "\033[0m",
                               .pass = "\033[32m",
                               .fail  = "\033[31m"
              } if specified then overrides default color values
     * @param options.dry_run {default: false} if true then print test names to be
                                               executed without running them
     */
    auto operator=(options);

    /**
     * @example suite _ = [] {};
     * @param suite() executes suite
     */
    template<class TSuite>
    auto on(ut::events::suite<TSuite>);

    /**
     * @example "name"_test = [] {};
     * @param test.type ["test", "given", "when", "then"]
     * @param test.name "name"
     * @param test.arg parameterized argument
     * @param test() executes test
     */
    template<class... Ts>
    auto on(ut::events::test<Ts...>);

    /**
     * @example skip / "don't run"_test = []{};
     * @param skip.type ["test", "given", "when", "then"]
     * @param skip.name "don't run"
     * @param skip.arg parameterized argument
     */
    template<class... Ts>
    auto on(ut::events::skip<Ts...>);

    /**
     * @example file.cpp:42: expect(42_i == 42);
     * @param assertion.expr 42_i == 42
     * @param assertion.location { "file.cpp", 42 }
     * @return true if expr passes, false otherwise
     */
    template <class TExpr>
    auto on(ut::events::assertion<TExpr>) -> bool;

    /**
     * @example expect((2_i == 1) >> fatal)
     * @note triggered by `fatal`
     *       should std::exit
     */
    auto on(ut::events::fatal_assertion);

    /**
     * @example log << "message"
     * @param log.msg "message"
     */
    template<class TMsg>
    auto on(ut::events::log<TMsg>);

    /**
     * Explicitly runs registered test suites
     * If not called directly test suites are executed with run's destructor
     * @example return run({.report_errors = true})
     * @param run_cfg.report_errors {default: false} if true it prints the summary after runnig
     */
    auto run(run_cfg);

    /**
     * Runs registered test suites if they haven't been explicilty executed already
     */
    ~run();
  };

  /**
   * Default reporter policy
   */
  class reporter {
   public:
    /**
     * @example file.cpp:42: "name"_test = [] {};
     * @param test_begin.type ["test", "given", "when", "then"]
     * @param test_begin.name "name"
     * @param test_begin.location { "file.cpp", 42 }
     */
    auto on(ut::events::test_begin) -> void;

    /**
     * @example "name"_test = [] {};
     * @param test_run.type ["test", "given", "when", "then"]
     * @param test_run.name "name"
     */
    auto on(ut::events::test_run) -> void;

    /**
     * @example "name"_test = [] {};
     * @param test_skip.type ["test", "given", "when", "then"]
     * @param test_skip.name "name"
     */
    auto on(ut::events::test_skip) -> void;

    /**
     * @example "name"_test = [] {};
     * @param test_end.type ["test", "given", "when", "then"]
     * @param test_end.name "name"
     */
    auto on(ut::events::test_end) -> void;

    /**
     * @example log << "message"
     * @param log.msg "message"
     */
    template<class TMsg>
    auto on(ut::events::log<TMsg>) -> void;

    /**
     * @example file.cpp:42: expect(42_i == 42);
     * @param assertion_pass.expr 42_i == 42
     * @param assertion_pass.location { "file.cpp", 42 }
     */
    template <class TExpr>
    auto on(ut::events::assertion_pass<TExpr>) -> void;

    /**
     * @example file.cpp:42: expect(42_i != 42);
     * @param assertion_fail.expr 42_i != 42
     * @param assertion_fail.location { "file.cpp", 42 }
     */
    template <class TExpr>
    auto on(ut::events::assertion_fail<TExpr>) -> void;

    /**
     * @example expect((2_i == 1) >> fatal)
     * @note triggered by `fatal`
     *       should std::exit
     */
    auto on(ut::events::fatal_assertion) -> void;

    /**
     * @example "exception"_test = [] { throw std::runtime_error{""}; };
     */
    auto on(ut::events::exception) -> void;

    /**
     * @note triggered on destruction of runner
     */
    auto on(ut::events::summary) -> void;
  };

  /**
   * Used to override default running policy
   * @example template <> auto cfg<override> = runner<reporter>{};
   */
  struct override {};

  /**
   * Default UT execution policy
   * Can be overwritten with override
   */
  template <class = override> auto cfg = runner<reporter>{};
}

    Configuration

Option Description Example
BOOST_UT_VERSION Current version 1'1'9

FAQ

    How does it work?

suite

/**
 * Reperesents suite object
 * @example suite _ = []{};
 */
struct suite final {
  /**
   * Assigns and executes test suite
   */
  [[nodiscard]] constexpr explicit(false) suite(Suite suite) {
    suite();
  }
};

test

/**
 * Creates named test object
 * @example "hello world"_test
 * @return test object
 */
[[nodiscard]] constexpr Test operator ""_test(const char* name, std::size_t size) {
  return test{{name, size}};
}
/**
 * Represents test object
 */
struct test final {
  std::string_view name{}; /// test case name

  /**
   * Assigns and executes test function
   * @param test function
   */
  constexpr auto operator=(const Test& test) {
    std::cout << "Running... " << name << '\n';
    test();
  }
};

expect

/**
 * Evaluates an expression
 * @example expect(42_i == 42);
 * @param expr expression to be evaluated
 * @param source location https://en.cppreference.com/w/cpp/utility/source_location
 * @return stream
 */
constexpr OStream& expect(
  Expression expr,
  const std::source_location& location = std::source_location::current()
) {
  if (not static_cast<bool>(expr) {
    std::cerr << location.file()
              << ':'
              << location.line()
              << ":FAILED: "
              << expr
              << '\n';
  }

  return std::cerr;
}
/**
 * Creates constant object for which operators can be overloaded
 * @example 42_i
 * @return integral constant object
 */
template <char... Cs>
[[nodiscard]] constexpr Operator operator""_i() -> integral_constant<int, value<Cs...>>;
/**
 * Overloads comparison if at least one of {lhs, rhs} is an Operator
 * @example (42_i == 42)
 * @param lhs Left-hand side operator
 * @param rhs Right-hand side operator
 * @return Comparison object
 */
[[nodiscard]] constexpr auto operator==(Operator lhs, Operator rhs) {
  return eq{lhs, rhs};
}
/**
 * Comparison Operator
 */
template <Operator TLhs, Opeartor TRhs>
struct eq final {
  TLhs lhs{}; // Left-hand side operator
  TRhs rhs{}; // Right-hand side operator

  /**
   * Performs comparison operatation
   * @return true if expression is succesful
   */
  [[nodiscard]] constexpr explicit operator bool() const {
    return lhs == rhs;
  }

  /**
   * Nicely prints the operation
   */
  friend auto operator<<(OStream& os, const eq& op) -> Ostream& {
    return (os << op.lhs << " == " << op.rhs);
  }
};

Sections

/**
 * Convenient aliases for creating test named object
 * @example should("return true") = [] {};
 */
constexpr auto should = [](const auto name) { return test{name}; };

Behaviour Driven Development (BDD)

/**
 * Convenient aliases for creating BDD tests
 * @example feature("Feature") = [] {};
 * @example scenario("Scenario") = [] {};
 * @example given("I have an object") = [] {};
 * @example when("I call it") = [] {};
 * @example then("I should get") = [] {};
 */
constexpr auto feature  = [](const auto name) { return test{name}; };
constexpr auto scenario = [](const auto name) { return test{name}; };
constexpr auto given    = [](const auto name) { return test{name}; };
constexpr auto when     = [](const auto name) { return test{name}; };
constexpr auto then     = [](const auto name) { return test{name}; };

https://godbolt.org/z/6Nk5Mi

Spec

/**
 * Convenient aliases for creating Spec tests
 * @example describe("test") = [] {};
 * @example it("should...") = [] {};
 */
constexpr auto describe = [](const auto name) { return test{name}; };
constexpr auto it       = [](const auto name) { return test{name}; };

Example implementation

Try it online

    Fast compilation times (Benchmarks)?

Implementation

  • Leveraging C++20 features

  • Avoiding unique types for lambda expressions

  template <class Test>
    requires not std::convertible_to<Test, void (*)()>>
  constexpr auto operator=(Test test);

vs

  // Compiles 5x faster because it doesn't introduce a new type for each lambda
  constexpr auto operator=(void (*test)());
  • Type-name erasure (allows types/function memoization)
  eq<integral_constant<42>, int>{ {}, 42 }

vs

  // Can be memoized - faster to compile
  eq<int, int>{42, 42}
  • Limiting preprocessor work

    • Single header/module
    • Minimal number of include files
  • Simplified versions of

    • std::function
    • std::string_view

    C++20 features?

    C++2X integration?

Parameterized tests with Expansion statements (https://wg21.link/P1306r1)

template for (auto arg : std::tuple<int, double>{}) {
  test("types " + std::to_string(arg)) = [arg] {
    expect(type(arg) == type<int> or type(arg) == type<double>);
  };
}
All tests passed (2 asserts in 2 tests)

https://cppx.godbolt.org/z/dMmqmM

    Is standardization an option?

Personally, I believe that C++ standard could benefit from common testing primitives (expect, ""_test) because

  • It lowers the entry-level to the language (no need for third-party libraries)
  • It improves the education aspect (one standard way of doing it)
  • It makes the language more coherent/stable (consistent design with other features, stable API)
  • It makes the testing a first class citizen (shows that the community cares about this aspect of the language)
  • It allows to publish tests for the Standard Library (STL) in the standard way (coherency, easier to extend)
  • It allows to act as additional documentation as a way to verify whether a particular implementation is conforming (quality, self-verification)
  • It helps with establishing standard vocabulary for testing (common across STL and other projects)

    Can I still use macros?

Sure, although please notice that there are negatives of using macros such as

  • Error messages might be not clear and/or point to the wrong line
  • Global scope will be polluted
  • Type safety will be ignored
#define EXPECT(...) ::boost::ut::expect(::boost::ut::that % __VA_ARGS__)
#define SUITE       ::boost::ut::suite _ = []
#define TEST(name)  ::boost::ut::detail::test{"test", name} = [=]() mutable

SUITE {
  TEST("suite") {
    EXPECT(42 == 42);
  };
};

int main() {
  TEST("macro") {
    EXPECT(1 != 2);
  };

  TEST("vector") {
    std::vector<int> v(5);

   EXPECT((5u == std::size(v)) >> fatal) << "fatal";

    TEST("resize bigger") {
      v.resize(10);
      EXPECT(10u == std::size(v));
    };
  };
}
All tests passed (4 asserts in 3 tests)

https://godbolt.org/z/WcEKTr

    What about Mocks/Stubs/Fakes?

Consider using one of the following frameworks

    What about Microbenchmarking?

Example benchmark

Consider using one of the following frameworks

    Related materials/talks?

    How to contribute?

CONTRIBUTING

Benchmarks

Framework Version Standard License Linkage Test configuration
Boost.Test 1.71.0 C++03 Boost 1.0 single header/library static library
GoogleTest 1.10.0 C++11 BSD-3 library static library
Catch 2.10.2 C++11 Boost 1.0 single header CATCH_CONFIG_FAST_COMPILE
Doctest 2.3.5 C++11 MIT single header DOCTEST_CONFIG_SUPER_FAST_ASSERTS
UT 1.1.0 C++20 Boost 1.0 single header/module
Include / 0 tests, 0 asserts, 1 cpp file
Assert / 1 test, 1'000'000 asserts, 1 cpp file
Test / 1'000 tests, 0 asserts, 1 cpp file
Suite / 10'000 tests, 0 asserts, 100 cpp files
Suite+Assert / 10'000 tests, 40'000 asserts, 100 cpp files
Suite+Assert+STL / 10'000 tests, 20'000 asserts, 100 cpp files
Incremental Build - Suite+Assert+STL / 1 cpp file change (1'000 tests, 20'000 asserts, 100 cpp files)
Suite+Assert+STL / 10'000 tests, 20'000 asserts, 100 cpp files
(Headers vs Precompiled headers vs C++20 Modules)

https://github.com/cpp-testing/ut-benchmark


Disclaimer UT is not an official Boost library.