Ember Exam

Ember Exam is an addon to allow you more control over how you run your tests when used in conjunction with Ember QUnit or Ember Mocha. It provides the ability to randomize, split, parallelize, and load-balance your test suite by adding a more robust CLI command.

It started as a way to help reduce flaky tests and encourage healthy test driven development. It's like Head & Shoulders for your tests!

The documentation website contains examples and API information.

Table of Contents

• Installation
• How To Use
  • Version < 3.0.0
  • Randomization
    • Randomization Iterator
  • Splitting
    • Split Test Parallelization
  • Test Load Balancing
    • Test Failure Reproduction
  • Preserve Test Name
• Advanced Configuration
  • Ember Try & CI Integration
  • Test Suite Segmentation

Installation

Installation is as easy as running:

$ ember install ember-exam

How To Use

Using Ember Exam is fairly straightforward as it extends directly from the default Ember-CLI test command. So, by default, it will work exactly the same as ember test.

$ ember exam
$ ember exam --filter='acceptance'
$ ember exam --server
$ ember exam --load-balance --parallel=1

For more information and examples, please visit the documentation website.

# A value of filter is acceptance
$ ember exam --filter 'acceptance'

# A value of parallel is 2
$ ember exam --load-balance --parallel=2 --server

# If a `=` is not used to pass a value to an option that requires a value, it will take anything passed after a space as it's value
# In this instance, the value of parallel is --server
$ ember exam --load-balance --parallel --server

The idea is that you can replace ember test with ember exam and never look back.

To get the unique features of Ember Exam (described in-depth below), you will need to replace the use of start() from Ember-Qunit or Ember-Mocha in test-helper.js with start() from ember-exam:

// test-helper.js
import start from 'ember-exam/test-support/start';

// Options passed to `start` will be passed-through to ember-qunit or ember-mocha
start();

Version < 3.0.0

Prior to 2.1.0, Ember Exam must be loaded by importing addon-test-support/load.js and calling loadEmberExam:

// test-helper.js
import loadEmberExam from 'ember-exam/test-support/load';

loadEmberExam();

Randomization

$ ember exam --random[=<seed>]

The random option allows you to randomize the order in which your tests run. You can optionally specify a "seed" value from which to randomize your tests in order to reproduce results. The seed can be any string value. Regardless of whether you specify a seed or not, Ember Exam will log the seed value used for the randomization at the beginning of the test run:

$ ember exam --random
$ Randomizing tests with seed: liv5d1ixkco6qlatl6o7mbo6r

$ ember exam --random=this_is1337
$ Randomizing tests with seed: this_is1337

If you use random without specifying a seed, it must be the last argument you pass. Otherwise, Ember Exam will attempt to interpret any following arguments as the seed value. In other words:

# don't do this
ember exam --random --split=2
Randomizing tests with seed: --split=2 # this is not what we wanted

# do this instead
ember exam --split=2 --random
Randomizing tests with seed: hwr74nkk55vzpvi

Note: You must be using QUnit version 1.23.0 or greater for this feature to work properly. This feature is not currently supported by Mocha.

Randomization Iterator

Randomization can be helpful for identifying non-atomic or order-dependent tests. To that end, Ember Exam provides an iterator to make it easy to test lots of variations in your test suite order quickly.

$ ember exam:iterate <num>

This command will build your application once, and then run the test suite with the random option for the specified number of iterations. You can optionally skip the build by using a previous build via the path option:

$ ember exam:iterate <num> --path <build-path>

Finally, you can pass additional options through to the exam command used to run the tests via the options flag:

$ ember exam:iterate <num> --options <options>

The options should be a string matching what you would use via the CLI.

Note: This feature is not currently supported by Mocha.

Generating Module Metadata File For Test Execution

$ ember exam --write-module-metadata-file
$ ember exam --wmmf

The --write-module-metadata-file, wmmf as an alias, allows you to generate a module metadata file after a test run. The file provides metadata about the test modules executed.

It creates a json file, module-metadata-<timestamp>.json, which contains an array of elements representing metadata of modules executed by sorted by ascending order:

[
  {
    "moduleName": "Module-name",
    "total": "Total number of tests in the module",
    "passed": "A number of passed tests in the module",
    "failed": "A number of failed tests in the module",
    "skipped": "A number of skipped tests in the module",
    "duration": "ms in Total duration to execute the module",
    "failedTests": "A list of failed tests"
  }
]

and it looks something like below:

[
  {
    "moduleName": "Slowest-module",
    "total": 12,
    "passed": 9,
    "failed": 1,
    "skipped": 2,
    "duration": 153,
    "failedTests": ["failed-test-1"]
  },
  {
    "moduleName": "Fastest-module",
    "total": 2,
    "passed": 1,
    "failed": 0,
    "skipped": 0,
    "duration": 123,
    "failedTests": []
  }
]

Note: This feature is not currently supported by Mocha.

Splitting

$ ember exam --split=<num>

The split option allows you to specify the number of partitions greater than one to spread your tests across. Ember Exam will then proceed to run the first batch of tests.

$ ember exam --split=<num> --partition=<num>

The partition option allows you to specify which test group to run after using the split option. It is one-indexed, so if you specify a split of 3, the last group you could run is 3 as well. You can also run multiple partitions, e.g.:

$ ember exam --split=4 --partition=1 --partition=2

_Note: Ember Exam splits tests by modifying the Ember-QUnit/Ember-Mocha's TestLoader to bucket each test file into a partition, where each partition has an even number of test files. This makes it possible to have unbalanced partitions. To run your tests with balanced partitions, consider using --load-balance. For more info, see Test Load Balancing.

Split Test Parallelization

$ ember exam --split=<num> --parallel

The parallel option allows you to run your split tests across multiple test pages in parallel in Testem. It will use a separate browser instance for each group of tests. So, if you specify a split of 3, then 3 browser instances will be spawned with the output looking something like:

ok 1 PhantomJS 1.9 - Exam Partition 1 - some test
ok 2 PhantomJS 1.9 - Exam Partition 3 - some other other test
ok 3 PhantomJS 1.9 - Exam Partition 2 - some other test

You can also combine the parallel option with the partition option to split tests, and then recombine partitions into parallel runs. This would, for example, allow you to run tests in multiple CI containers and have each CI container parallelize its list of tests.

For example, if you wanted to run your tests across two containers, but have one of them run twice as many tests as the other, and run them in parallel, you could do this:

# container 1
ember exam --split=3 --partition=1,2 --parallel

# container 2
ember exam --split=3 --partition=3 --parallel

Note 1: Ember Exam will respect the parallel setting of your Testem config file while running tests in parallel. The default value for parallel in Testem is 1, which means you'll need a non-default value to actually see parallel behavior.

Note 2: Ember Exam sets process.env.EMBER_EXAM_SPLIT_COUNT for convenience. You can use this in your Testem file.

Note 3: You must be using Testem version 1.5.0 or greater for this feature to work properly.

Filtering

Ember Exam provides options to filter test suites by two types - module path and test file path.

$ ember exam --module-path=<module-path>

The module-path option allows you to filter module paths by a given value. Module paths are mapped by test files and they are generated during ember build. After the build, tests.js file is created and it resides under /assets. The file is combined of all tests in an application and it has a form of define("<module-path>", others...

The value for module-path can have either string or regular expression, for instance:

# When module path value is string. This will run all modules which match with the passed value
$ ember exam --module-path='dummy/tests/helpers/module-for-acceptance'

# When module path value is regex. This will run all modules which have `dummy` in it
$ ember exam --module-path='!/dummy/'

The file-path option is to filter tests by test file path. The test file path is a location of the test file in a file system. You can specify file-path to a location of specific test file path or you can use wildcards in paths to target multiple test files.

# This will run tests that are defined in `/my-application/tests/unit/my-test.js`
$ ember exam --file-path='/my-application/tests/unit/my-test.js'

# This will run all test files that are under `/my-application/tests/unit/`
$ ember exam --file-path='/my-application/tests/unit/*.js'

Test Load Balancing

$ ember exam --parallel=<num> --load-balance

The load-balance option allows you to load balance test files against multiple browsers. It will order the test files by test types, e.g. acceptance | integration | unit | eslint, and load balance the ordered test files between the browsers dynamically rather than statically. Note: parallel must be used along with load-balance to specify a number of browser(s)

The load-balance option was added to version 1.1 to address execution performance when running against a large test suite.

Web browsers and the testem server communicate via promise in order to send and receive test file. The promise timeout value is set to 15 seconds, and is configurable by adding asyncTimeout=[timeout] as a querystring param in the test URL or adding to the test_page option in the testem config. For example, if you specify load-balance and parallel equals 3, then three browser instances will be created and the output will look something like:

# ember exam --parallel=3 --load-balance
ok 1 Chrome 66.0 - Browser Id 1 - some test
ok 2 Chrome 66.0 - Browser Id 2 - some another test
ok 3 Chrome 66.0 - Browser Id 3 - some the other test

You can also specify the split and partition options with load-balance to load a portion of test modules on multiple CI containers.

$ ember exam --split=<num> --partition=<num> --parallel=<num> --load-balance

This command will split test files and load-balance tests from the specified partition across the browsers. For example ember exam --split=2 --partition=1 --parallel=3 --load-balance, the complete list of test files are split into two halves. With the first half of the list load balanced against three browsers. The output will look something like below:

# ember exam --split=2 --partition=1 --parallel=3 --load-balance
ok 1 Chrome 66.0 - Exam Partition 1 - browser Id 1 - some test
ok 2 Chrome 66.0 - Exam Partition 1 - browser Id 2 - another test
ok 3 Chrome 66.0 - Exam Partition 1 - browser Id 3 - some the other test

Important information on Load Balancing

1. The --load-balance option is currently only supported in CI mode and for that reason no-launch cannot be used with load-balance.
2. You must be using ember-cli version 3.2.0 or greater for load balancing and test failure reproduction features to work properly.
3. You must be using ember-qunit version 4.1.1 or greater for this feature to work properly.
4. You must be using qunit version 2.13.0 or greater for this feature to work properly.
5. This feature is not currently supported by Mocha.

Test Failure Reproduction

Due to the dynamic nature of the load-balance option, test file execution order can vary between runs. In order to reproduce a past test execution, the execution must be recorded via passing --write-execution-file or --wef, which allows generating a JSON file that enables rerunning the past test execution. The option is only allowed when load-balance is passed.

# The command will load in test balanced mode with <num> of browser(s). After the test suite execution, it will generate a test-execution json file.
$ ember exam --parallel=<num> --load-balance --wef
$ ember exam --parallel=<num> --load-balance --write-execution-file

The file is stored in the root directory and the naming structure is test-execution-<timestamp>.json. To replay the test execution for particular browser(s), do the following:

# The command will read a test execution file specified for `replay-execution` and execute a browser Id(s) from `replay-browser`
$ ember exam --replay-execution=[string] --replay-browser=[num]

replay-execution allows you to specify a path to the json file to run execution against and replay-browser is to specify browser ID(s) to execute.

# The command will read test-execution-000000.json and load the list of modules mapped to browserId 1
$ ember exam --replay-execution=test-execution-000000.json --replay-browser=1

The above command will read test-execution-000000.json and load the list of modules which is mapped by browser ID #1.

replay-browser can be an array of browser IDs. For instance --replay-browser=1,2 will start two browsers and execute a list of modules which were previously run by browsers #1 and #2.

# The command will read test-execution-000000.json and load the list of module mapped to browserId 1 and 2
$ ember exam --replay-execution=test-execution-000000.json --replay-browser=1,2

When replay-browser value is not specified it will execute browserId(s) read from failedBrowser in the test execution file.

# The command will read test-execution-000000.json and load the list of modules mapped to browserIds from failedBrowser in the json file.
$ ember exam --replay-execution=test-execution-000000.json

When replay-browser value is not specified and there is no value for failedBrowser in the json file it will rerun all list of modules.

# The command will read test-execution-000000.json and load the list of module mapped to all browserIds when failedBrowser is none in the json file
$ ember exam --replay-execution=test-execution-000000.json

Important information on --replay-execution and --replay-browser

1. You must be using ember-cli version 3.2.0 or greater for load-balnce and test failure reproduction features to work properly.
2. You must be using ember-qunit version 4.1.1 or greater for this feature to work properly.
3. You must be using qunit version 2.8.0 or greater for this feature to work properly.
4. This feature is not currently supported by Mocha.

Preserve Test Name

When using --split and/or --load-balance the output will look something like:

# ember exam --split=2 --partition=1 --parallel=3 --load-balance
ok 1 Chrome 66.0 - Exam Partition 1 - browser Id 1 - some test
ok 2 Chrome 66.0 - Exam Partition 1 - browser Id 2 - another test
ok 3 Chrome 66.0 - Exam Partition 1 - browser Id 3 - some the other test

However, if you change the amount of parallelization, or randomize across partitions, the output will change for the same test, which may be an issue if you are tracking test insights over time.

# ember exam --split=2 --partition=1 --parallel=2 --load-balance
ok 1 Chrome 66.0 - Exam Partition 1 - browser Id 2 - some test
ok 2 Chrome 66.0 - Exam Partition 1 - browser Id 1 - another test
ok 3 Chrome 66.0 - Exam Partition 1 - browser Id 2 - some the other test

You can add --preserve-test-name to remove the dynamic segments of the output (partition and browser) to ensure the output test names are always the same.

# ember exam --split=2 --partition=1 --parallel=3 --load-balance --preserve-test-name
ok 1 Chrome 66.0 - some test
ok 2 Chrome 66.0 - another test
ok 3 Chrome 66.0 - some the other test

Advanced Configuration

Ember Exam does its best to allow you to run your test suite in a way that is effective for your individual needs. To that end, there are lots of advanced ways to configure your setup by integrating with other aspects of the Ember testing environment. The following sections will cover a few of the more common scenarios.

Ember Try & CI Integration

Integrating ember-exam with ember-try is remarkably easy. Define a command in your ember-try.js config that leverages the exam command:

// config/ember-try.js
module.exports = {
  command: 'ember exam --split 3 --parallel',
  // ...
};

Using environmental variables gives you flexibility in how you run your tests. For instance, you could distribute your tests across processes instead of parallelizing them by specifying a PARTITION variable in your process environment and then consuming it like so:

module.exports = {
  command: 'ember exam --split 20 --partition ' + process.env.PARTITION,
  // ...
};

If you are working with Travis CI then you can also easily set up seeded-random runs based on PR numbers. Similar to the following:

const command = [ 'ember', 'exam', '--random' ];
const pr = process.env.TRAVIS_PULL_REQUEST;

if (pr) {
  command.push(pr);
}

module.exports = {
  command: command.join(' '),
  // ...
};

You can refer to Travis' default environment variables to see what else you could possibly leverage for your test setup.

Test Suite Segmentation

Some test suites like to segment which tests run based on various facets such as type of test, feature being tested, and so on. This can be accomplished by leveraging Testem's ability to have multiple test pages:

{
  "test_page": [
    "tests/index.html?filter=acceptance",
    "tests/index.html?filter=!acceptance"
  ]
}

You can use this feature in conjunction with Ember Exam's features, which will allow you to segment your test suite but still gain benefits from randomization and splitting.