lest – lest errors escape testing

This tiny C++11 test framework is based on ideas and examples by Kevlin Henney [1,2] and on ideas found in the CATCH test framework by Phil Nash [3].

Let writing tests become irresistibly easy and attractive.

Contents

• Example usage
• In a nutshell
• License
• Dependencies
• Installation
• Usage
• Synopsis
• Variants of lest
• Features of lest
• Reported to work with
• Building tests and examples
• Contributions to lest
• Other test frameworks
• Notes and references

Example usage

#include "lest/lest.hpp"

using namespace std;

const lest::test specification[] =
{
    CASE( "Empty string has length zero (succeed)" )
    {
        EXPECT( 0 == string(  ).length() );
        EXPECT( 0 == string("").length() );
    },

    CASE( "Text compares lexically (fail)" )
    {
        EXPECT( string("hello") > string("world") );
    },

    CASE( "Unexpected exception is reported" )
    {
        EXPECT( (throw std::runtime_error("surprise!"), true) );
    },

    CASE( "Unspecified expected exception is captured" )
    {
        EXPECT_THROWS( throw std::runtime_error("surprise!") );
    },

    CASE( "Specified expected exception is captured" )
    {
        EXPECT_THROWS_AS( throw std::bad_alloc(), std::bad_alloc );
    },

    CASE( "Expected exception is reported missing" )
    {
        EXPECT_THROWS( true );
    },

    CASE( "Specific expected exception is reported missing" )
    {
        EXPECT_THROWS_AS( true, std::runtime_error );
    },
};

int main( int argc, char * argv[] )
{
    return lest::run( specification, argc, argv );
}

Note: besides above table approach, lest also supports auto-registration of tests.

Compile and run

prompt>g++ -Wall -Wextra -std=c++11 -I../include -o 05_select.exe 05_select.cpp && 05_select.exe
05_select.cpp:17: failed: Text compares lexically (fail): string("hello") > string("world") for "hello" > "world"
05_select.cpp:22: failed: got unexpected exception with message "surprise!": Unexpected exception is reported: (throw std::runtime_error("surprise!"), true)
05_select.cpp:37: failed: didn't get exception: Expected exception is reported missing: true
05_select.cpp:42: failed: didn't get exception of type std::runtime_error: Specific expected exception is reported missing: true
4 out of 7 selected tests failed.

With Buck:

prompt> buck run example/:05_select
...

In a nutshell

lest is a small C++11 test framework for unit testing, regression testing, Test-driven development (TDD) and Behaviour-driven design (BDD). It replicates innovative ideas in C++ testing from the Catch test framework such as function-level fixtures and expression-decomposing assertion macros in a form that is compact enough to read in five minutes. The lest_cpp03 variant provides similar capabilities to use with C++98/03 compilers.

Features and properties of lest are ease of installation (single header), no boilerplate code, traditional unit test cases and BDD style scenarios, strings as test names, function-level fixtures, expression-decomposing assertion macros, support for floating point comparison, test selection from commandline, test duration timing, test randomisation and sorting, display of passing tests, colourised output (compile-time option), C++11 code and a C++98/03 variant with comparable features (also compilable as C++11).

Features available via other projects are mocking (see integrate Trompeloeil mocking framework) and hamcrest matchers (see variants of lest),

Not provided are things present in other test frameworks, such as suites of tests, value-parameterised tests, type-parameterised tests, test data generators, customisable reporting, easy logging of extra information, breaking into a debugger, concurrent execution of tests, isolated execution of tests, Visual Studio Test Adapter.

License

lest uses the Boost Software License.

Dependencies

lest has no other dependencies than the C++ standard library.

Installation

lest is a single-file header-only library. Put lest.hpp, or a variant of it such as lest_cpp03.hpp directly into the project source tree or somewhere reachable from your project.

Usage

• Making a case – example
• Asserting for success – example
• Asserting for failure – example
• Asserting for exceptions – example
• Asserting for a specific exception – example
• Comparing floating point numbers – example.
• Auto test registration – single-file example
• Auto test registration with multiple source files – example part 1, 2, 3.
• Non-auto test registration – single-file example
• Non-auto test registration with multiple source files – example part 1, 2, 3.
• Using fixtures – example.
• Using lest assertions in a (reusable) user-defined function or lambda – function, templated function, lambda.
• Writing the test main function – single-file auto, non-auto – multi-file auto part 1, 2, 3, non-auto part 1, 2, 3.
• Integrating lest with the Trompeloeil mocking framework – example.
• Reporting a user-defined type – example.
• Running tests

Synopsis

Contents

• Command line
• Test case macro
• Fixture macros
• Assertion macros
• BDD style macros
• Module registration macro
• Feature selection macros
• Standard selection macro
• Namespace
• Tests
• Main
• Main (Trompeloeil)
• Floating point comparison
• Reporting a user-defined type

Command line

Usage: test [options] [test-spec ...]

Options:

• -h, --help, this help message
• -a, --abort, abort at first failure
• -c, --count, count selected tests
• -g, --list-tags, list tags of selected tests
• -l, --list-tests, list selected tests
• -p, --pass, also report passing tests
• -z, --pass-zen, ... without expansion
• -t, --time, list duration of selected tests
• -v, --verbose, also report passing or failing sections
• --order=declared, use source code test order (default)
• --order=lexical, use lexical sort test order
• --order=random, use random test order
• --random-seed=n, use n for random generator seed
• --random-seed=time, use time for random generator seed
• --repeat=n, repeat selected tests n times (-1: indefinite)
• --version, report lest version and compiler used
• --, end options

Test specification:

• "@", "*": all tests, unless excluded
• empty: all tests, unless tagged [hide] or [.optional-name]
• "text": select tests that contain text (case insensitive)
• "!text": omit tests that contain text (case insensitive)

Test descriptions can contain tags such as [option], [hide] and [.integration]. Tests that contain the tag [hide] or a tag that starts with [. in their description are skipped, unless they are specifically selected by specifying "@", "*", or by specifying (part of) the tag.

Test specifications can be combined and are evaluated left-to-right. For example: a !ab abc selects all tests that contain 'a', except those that contain 'ab', but include those that contain 'abc'.

When regular expression selection has been enabled (and works), test specifications can use the regular expression syntax of std::regex_search(). See also lest_FEATURE_REGEX_SEARCH in section Other Macros.

Test case macro

A lest test specification can consist of a) one or more arrays of test cases that use lambdas, or b) auto-registered test cases that use free functions. See also macro lest_FEATURE_AUTO_REGISTER.

CASE( "proposition" ) { code }   (array of cases)
Describe the expected behaviour to test for and specify the actions and expectations. See also section Module registration macro – Single-file code example – Multi-file code example part 1, 2, 3.

CASE_ON( "proposition", ...) { code }   (array of cases, since v1.33)
Describe the expected behaviour to test for and specify the actions and expectations. After the description you can add a lambda capture list to refer to symbols in the enclosing scope. See also section Module registration macro – Single-file code example – Multi-file code example part 1, 2, 3.

lest_CASE( specification, "proposition" ) { code }   (auto-registered cases)
Provide the collection of test cases, describe the expected behaviour to test for and specify the actions and expectations. Consider defining macro CASE(proposition) to hide the collection of test cases and define it in terms of lest_CASE(...) – Single-file code example – Multi-file code example part 1, 2, 3.

Fixture macros

lest provides function-level fixtures. Fixtures are stack-based and their setup and teardown occurs at the block scope of SETUP and (nested) SECTIONs – Code example.

SETUP( "context" ) { code }
Describe and setup the context to use afresh in each enclosed section.

SECTION( "proposition" ) { code }
Describe the expected behaviour to test for using the enclosing context and specify the actions and expectations. The objects in the enclosing setup or section come into existence and go out of scope for each section. A section must be enclosed in setup or in another section.

Assertion macros

lest has expression-decomposing assertion macros. An expression with strings such as hello > world may be reported with code and expansion as hello > world ("hello" > "world"). As a consequence, only a few assertion macro variants are needed – Code example.

EXPECT( expr )
Evaluate the expression and report failure. If an exception is thrown it is caught, reported and counted as a failure.

EXPECT_NOT( expr )
Evaluate the expression, record the logical not of its result and report failure. If an exception is thrown it is caught, reported and counted as a failure. This macro is a workaround to circumvent ! prefixed expressions as these cannot be decomposed.

EXPECT_NO_THROW( expr )
Expect that no exception (of any type) is thrown during evaluation of the expression.

EXPECT_THROWS( expr )
Expect that an exception (of any type) is thrown during evaluation of the expression.

EXPECT_THROWS_AS( expr, exception )
Expect that an exception of the specified type is thrown during evaluation of the expression.

If an assertion fails, the remainder of the test that assertion is part of is skipped.

BDD style macros

lest provides several macros to write Behaviour-Driven Design (BDD) style scenarios – Code example, auto-registration.

lest_SCENARIO( specification, "sketch" ) { code }   (auto-registered cases)

SCENARIO( "sketch" ) { code }   (array of cases)

GIVEN( "context" ) { code }

WHEN( "action" ) { code }

THEN( "result" ) { code }

AND_WHEN( "action" ) { code }

AND_THEN( "result" ) { code }

These macros simply map to macros CASE(), SETUP() and SECTION().

For auto-registered scenarios, consider defining macro SCENARIO(proposition) to hide the collection of scenarios and define it in terms of lest_SCENARIO(...).

Module registration macro

When using arrays of test cases written across multiple files, you can use macro MODULE() to add a module's test cases to the overall specification – Code example part 1, 2, 3.

MODULE( overall-specification, module-specification )
Register this module's test specification with the overall specification.

Note that with lest using auto test case registration there's no need for macro MODULE(), see the auto-registration example part 1, 2, 3. The same holds for lest_cpp03, see cpp03 example part 1, 2, 3.

Feature selection macros

-Dlest_NO_SHORT_MACRO_NAMES
-Dlest_NO_SHORT_ASSERTION_NAMES (deprecated)
All public API macros of lest exist as lest_MACRO and shorthand MACRO variant. Define this macro to omit the shorthand macros.

-Dlest_FEATURE_AUTO_REGISTER=0
Define this to 1 to enable auto registration of test cases. Default is 0.

See also section Test case macro.

-Dlest_FEATURE_COLOURISE=0
Define this to 1 to emphasise success and failure with colour. Default is 0.

Note: ANSI colour codes are used. On Windows versions that lack support for this you can use the ANSICON terminal. Executables can be obtained here.

-Dlest_FEATURE_LITERAL_SUFFIX=0
Define this to 1 to append u, l, a combination of these, or f to numeric literals. Default is 0.

-Dlest_FEATURE_REGEX_SEARCH=0
Define this to 1 to enable regular expressions to select tests. Default is 0.

Note: You have to make sure the compiler's library has a working std::regex_search(); not all do currently. GCC 4.8.1's regex search function doesn't work yet. Visual C++ probably has a working regex search function since VC9, Visual Studio 2008 (tested VC10, Visual Studio 2010).

-Dlest_FEATURE_TIME_PRECISION=0
Define this to set the precision of the duration in ms reported with option --time. Default is 0.

-Dlest_FEATURE_WSTRING=1
Define this to 0 to remove references to std::wstring. Default is 1.

-Dlest_FEATURE_RTTI (undefined)
lest tries to determine if RTTI is available itself. If that doesn't work out, define this to 1 or 0 to include or remove uses of RTTI (currently a single occurrence of typeid used for reporting a type name). Default is undefined.

Standard selection macro

-Dlest_CPLUSPLUS=199711L
Define this macro to override the auto-detection of the supported C++ standard, or if your compiler does not set the __cplusplus macro correctly.

Namespace

namespace lest { }
Types and functions are located in namespace lest.

Tests

struct env { };

struct test
{
 std::string name;
 std::function<void( env & )> behaviour;
};

You'll need type env and variable lest_env when you have a test case that calls a user-defined function or lambda that uses lest assertions like EXPECT() – Call reusable function, Call reusable templated function, and Call reusable lambda.

Main

A typical main() function for lest may look as follows:

#include "lest/lest.hpp"

const lest::test specification[] = { CASE("..."){} };

int main( int argc, char *argv[] )
{
    if ( int failures = lest::run( specification, argc, argv ) )
        return failures;
    
    return std::cout << "All tests passed\n", EXIT_SUCCESS;
}

Compile and run:

prompt>g++ -std=c++11 -o main.exe -I../include main.cpp && main.exe
All tests passed

Or, if feedback on success is moved to the command line:

#include "lest/lest.hpp"

const lest::test specification[] = { CASE("..."){} };

int main( int argc, char *argv[] )
{
    return lest::run( specification, argc, argv );
}

Compile and run with feedback on success:

prompt>g++ -std=c++11 -o main.exe -I../include main.cpp && main.exe && echo All tests passed
All tests passed

You can use the following variants of lest's run() function in main.

inline
int run( std::vector<test> specification, std::vector<std::string> arguments, std::ostream & os = std::cout );

inline
int run( std::vector<test> specification, int argc, char * argv[], std::ostream & os = std::cout );

template<std::size_t N>
int run( test const (& specification )[N], std::ostream & os = std::cout );

template<std::size_t N>
int run( test const (& specification )[N], std::vector<std::string> arguments, std::ostream & os = std::cout );

template<std::size_t N>
int run( test const (& specification )[N], int argc, char * argv[], std::ostream & os = std::cout );

• specification - collection of tests
• arguments - options and arguments to select and omit tests
• argc, arcv - options and arguments to select and omit tests
• os - stream to report to
• returns number of failing tests

Main (Trompeloeil)

You can integrate the Trompeloeil mocking framework with lest by providing a reporter for Trompeloeil – Code example.

#include "lest/lest.hpp"
#include "trompeloeil.hpp"

int main( int argc, char * argv[] )
{
    std::ostream & stream = std::cout;
    
    trompeloeil::set_reporter(
        [&stream]( trompeloeil::severity severity, const char * file, unsigned long line, std::string const & msg )
    {
        if ( severity == trompeloeil::severity::fatal )
        {
            throw lest::message{"", lest::location{ line ? file : "[file/line unavailable]", int(line) }, "", msg };
        }
        else
        {   
            stream << lest::location{ line ? file : "[file/line unavailable]", int(line) } << ": " << msg;
        }
    });

    return lest::run( specification, argc, argv, stream );
}

Floating point comparison

lest provides class approx to compare floating point values – Code example.

class approx { };

Use approx as follows:

EXPECT( 1.23 == approx( 1.23 ) );
EXPECT( 1.23 != approx( 1.24 ) );

EXPECT( 1.23 != approx( 1.231 ) );
EXPECT( 1.23 == approx( 1.231 ).epsilon( 0.1 ) );

approx custom = approx::custom().epsilon( 0.1 );

EXPECT( approx( 1.231 ) != 1.23 );
EXPECT( custom( 1.231 ) == 1.23 );

Class approx also provides less-than or equal and greater-than or equal operators.

Reporting a user-defined type

lest allows you to report a user-defined type via operator<<() – Code example.

To report a type not yet supported by lest, define a streaming function for it:

namespace ns {
 struct user-defined-type { ... };
 std::ostream & operator<< ( std::ostream & os, user-defined-type const & type )
 {
  using lest::to_string;
  return os << ... ;
 }
}

In it, stream the constituent parts of the type via lest's to_string() conversion functions.

Variants of lest

Various variants of lest are kept here. The simple ones, such as lest_basic and lest_decompose provide an easy read into the techniques used and remain the tiny test frameworks that are a good fit to include with small projects.

You are encouraged to take it from here and change and expand it as you see fit and publish your variant. If you do, I'd much appreciate to hear from you!

• lest.hpp - lest's latest development, this project.
• lest_basic.hpp - lest at its very basic, this project.
• lest_decompose.hpp - lest with expression decomposition, this project.
• lest_cpp03.hpp - lest with expression decomposition for C++03, this project.
• hamlest - matchers for lest.
• lest with groups - Pavel Medvedev

Features of lest

Reported to work with

The table below mentions the lowest version of a compiler lest is reported to work with.

Note: I've made a few concessions to enable compilation of lest.hpp with Visual C++:

• Prevent error C2797: replace braced member initialisation with C++98 style initialisation.
• Prevent error C2144: use enum{ value } instead of static constexpr bool in struct is_container (for VC only).

Building tests and examples

Tests and examples can be build with Buck, via Makefiles or by using CMake.

To build the tests and examples as described below you need:

• Buck or CMake version 3.5 or later to be installed and in your PATH.
• A suitable compiler.

The following steps assume that the lest source code has been cloned into a directory named lest.

Buck

lest> buck run test:test_lest_basic
lest> buck run test:test_lest_cpp03
lest> buck run test:test_lest_decompose
lest> buck run test:test_lest

CMake

1. Create a directory for the build outputs for a particular architecture.
   Here we use lest/build.

    lest> mkdir build && cd build
   

2. Configure CMake to use the compiler of your choice (run cmake --help for a list) and build the tests for lest and the examples.

    lest/build> cmake -G "Unix Makefiles" [see 3. below] ..
   

3. Optional. You can control above configuration through the following options:

   • -DLEST_BUILD_TEST=ON: build the tests for lest, default on
   • -DLEST_BUILD_EXAMPLE=ON: build the examples, default on
   • -DLEST_BUILD_CONTRIB=OFF: build the contrib folder, default off
   • -DLEST_EXTRA_WARNINGS=OFF: build with extra warnings and warnings as errors, default off

4. Build the test suite.

    lest/build> cmake --build .
   

5. Run the test suite.

    lest/build> ctest -V
   

All tests should pass, indicating your platform is supported and you are ready to use lest. Note that quite some examples fail. They do so to demonstrate the usage of things.

Contributions to lest

Folder contrib contains extensions to lest. These extensions are not part of lest itself because for example they use non-standard behaviour, they are considered to be for a too-specific use case, or they are considered not yet ripe for inclusion in lest and we first like to gain more experience with them.

Other test frameworks

This comparison of Catch, doctest and lest in table form may help you to discover similarities and differences of these frameworks.

• Catch - A modern, C++-native, header-only, framework for unit-tests, TDD and BDD.
• doctest - The lightest feature-rich C++ single-header testing framework for unit tests and TDD. Catch/lest-inspired tests that accompany code.
• stf - Simple TDD Framework. Catch/lest-inspired with an emphasis on floating point precision testing.
• cute - C++ unit test executor.
• bandit - Human friendly unit testing for C++11.
• igloo - BDD style unit testing for C++.
• mettle - A C++14 unit test framework.
• CppUTest - Unit testing and mocking framework for C/C++.
• UnitTest++ - A lightweight unit testing framework for C++.
• xUnit++ - A unit testing platform for C++ inspired by xUnit.net.
• ZenUnit - Single-header C++ unit testing framework for value- and type-parameterized tests.
• Search for C++98 test frameworks on GitHub
• Search for C++11 test frameworks on GitHub
• Search for C++14 test frameworks on GitHub
• List of C++ unit testing frameworks at Wikipedia.

Notes and references

[1] Kevlin Henney on Rethinking Unit Testing in C++ (Video).

[2] Martin Moene. Elefant C++11 test setup on the ACCU mailing list accu-general (requires login). It mentions the C++11 test appoach Andrzej Krzemieński uses for Optional. A library for optional (nullable) objects for C++11.

[3] Phil Nash. CATCH, an automated test framework for C, C++ and Objective-C.

[4] A more technically informed name: lest - lambda engaged small tester.