Kobold

Visual regression testing framework, comparing screenshots from multiple builds.

Table of Contents

• Installation
• Getting Started
• Command-Line
• Examples
• API-Documentation
• Tests
• Project Naming
• Third-party libraries
• License

##Installation

Install this module with the following command:

npm install kobold

Install this module globally with the following command:

npm install -g kobold

This will make sure that you don't have to enter the whole path when running Kobold.

Add the module to your package.json dependencies:

npm install --save kobold

Add the module to your package.json dev-dependencies:

npm install --save-dev kobold

##Getting Started

Kobold can be executed by running kobold script in the bin directory of the package. When you installed the module globally, then you can us only kobold to run it.

Only one parameter is required to run the tests:

kobold test/ui/regression

With this parameter, Kobold starts running tests on the test/ui/regression directory. It analyzes what screens were already approved, what screens are available in the current build, and what screens it has to compare. Then, it creates the test-code and executes the image-comparison, creating the test results in the highlight directory.

Instead of implicitly using the last argument for the test-path, you can also be more specific and supply the --test-path parameter:

kobold --test-path test/ui/regression

By default, Kobold will look for four directories:

• approved - All approved screens should be available in this folder. These are the "golden standard" images that will be used to compare them to build screens.
• build - Screens that were just taken are saved in this folder. This is most likely done by some Selenium tests, taking screenshots and saving them in this directory. These images will be compared to the images in the approved folder, creating comparison result images in highlight.
• highlight - Image comparison results of the latest test-run are saved in this folder. These images will highlight the differences between the approved-image and the build-image.
• config - Custom configuration files (.js or .json) for image specific comparison options. Sometimes, it is helpful to tune the comparison of a specific image. These options overwrite globally set comparison options. They are also optional. See the Blink-Diff Project WebSite for more information on these options.

The folder names for approved, build, highlight, and config can be changed by supplying the parameters --approved-folder, --build-folder, --highlight-folder, and --config-folder respectively:

kobold --approved-folder "golden" --highlight-folder "differences" /test/ui/regression

This test-run will look for golden and differences for the approved and highlight folder respectively instead of using the default folder names. In this specific case, build was not changed.

Sometimes, you want the tests to fail when previously approved screens are missing from the build directory. This can be the case when for example the tests that created these screenshots didn't fully run. Use the --fail-orphans flag to make these tests fail:

kobold --fail-orphans test/ui/regression

The --fail-additions flag will fail tests for screens that are new, screens that were never approved before.

kobold --fail-orphans --fail-additions test/ui/regression

Kobold also supports all Mocha parameters which can be interspersed with the Kobold parameters:

kobold --slow 3000 --fail-orphans --fail-additions --reporter dot --test-path test/ui/regression

##Command-Line Since Kobold is built on top of Mocha, it supports all of its parameters in addition to the following:

• approved-folder - Name of the approved folder (default: 'approved')
• build-folder - Name of the build folder (default: 'build')
• highlight-folder - Name of the highlight folder (default: 'highlight')
• config-folder - Name of the config folder (default: 'config')
• fail-orphans - Flag that determines that tests should fail when screens that were previously approved, being in the approve folder, are not found in the build folder.
• fail-additions - Flag that determines that tests should fail when unapproved screens are found in the build folder, meaning screens that are missing from the approve folder.
• test-path - Path to directory with all test related folders as mentioned above (required). Uses implicitly the last parameter element when none is given.
• config - Path to a config file (.js or .json) that is used to overwrite comparison options. (default: none) See the Blink-Diff Project WebSite for more information on these options.

##Configuration The global configuration file that can be selected by supplying --config to the command-line describes default-values for the comparison.

Here is an example of the file (config.js - can also be a JSON file):

module.exports = {
	"delta": 35 // Changing the distance of pixel-comparison
};

This file then can be called as follows:

kobold --config config.js test/ui/regression

Configuration files for specific screens on the other hand describe only the comparison options for that screen. Here is an example for a screen that is named "YDN_Missing"; the path is config/YDN_Missing.json (could also be a .js):

{
	"outputBackgroundOpacity": 0.3,
	"outputBackgroundGreen": 100
}

This example will make the background for the comparison image look greener, and lightens-up the whole comparison by reducing the opacity of the background-mask.

##Examples Kobold creates a regular test report as if the tests were manually written, and it also creates the following images, comparing the approved (left) with the build screenshot (right), pointing out the differences (middle).

• Changed color
• Missing items
• Messed-up sorting
• Image swap
• Text formatting

These examples can be found in the examples folder. The above results were produced by running the following command in the project root with a globally installed Kobold:

kobold test/ui/regression

##API-Documentation

Generate the API-documentation with following command:

npm run docs

This will generate the source-code documentation in the docs folder of the module root.

##Tests

Run the tests with the following command:

npm run test

The code-coverage will be written to the coverage folder in the module root.

##Project Name A Kobold is a gnome that can make itself invisible and is often the source of nuisances - so, pretty much like visual regressions.

##Third-party libraries

The following third-party libraries are used by this module:

###Dependencies

• blink-diff: https://github.com/yahoo/blink-diff
• preceptor-core: https://github.com/yahoo/preceptor-core
• kobold-core: https://github.com/yahoo/kobold-core
• log4js: https://github.com/nomiddlename/log4js-node
• promise: https://github.com/then/promise
• mocha: https://github.com/visionmedia/mocha

###Dev-Dependencies

• chai: http://chaijs.com
• istanbul: https://github.com/gotwarlost/istanbul
• yuidocjs: https://github.com/yui/yuidoc

##License

The MIT License

Copyright 2014 Yahoo Inc.