Scooby is an optimized regression and fidelity testing framework. It can be used to test for regressions (both visual and textual), and provides all tools and integrations necessary to streamline the experience (a CLI, a review UI, web service, and more).

For a quick introduction and demo, see this video:

Example use-cases:

• Prevent visual regressions in your website UI
  • After a PR is opened, you can use Scooby to verify that the changes didn't cause any unwanted regression in your website UI, flagging them for review and approval if necessary.
• Test for regressions in generated configuration files
  • You might use Scooby to do Snapshot testing of your UI structure (JSON) or other configuration files. With Scooby, you can avoid committing the snapshots into your repository, and instead generate and compare them on the fly inside a CI pipeline.
• Support migrations by comparing the output of two systems
  • If you are migrating a system, you might want to verify that the output of the new system is equal to the legacy one. Scooby can be used to compare and visualize the differences between the outputs, flagging them up for review and approval if necessary
• Test for Storybook components regressions (soon)
  • Verify that a code change doesn't cause any visual regression in your Storybook components

Scooby was built with these goals in mind:

• Make it easy to add new regression and fidelity tests, both visual and textual
• Support multiple test formats out of the box (such as HTML, PNG, code, etc) to minimize the amount of boilerplate glue code.
• Being cost-efficient
• Be highly extensible to support new kinds of tests and reports
• Integrate well with our GitHub-based workflows

Scooby has been extensively battle-tested at Anima, where it's powering most of our production pipelines.

Overview

Scooby is composed of three main components:

• A CLI used to run reports, either locally or inside a CI pipeline
• A Web UI used to view reports, and leave reviews (approvals, rejections, etc)
• (Optional) An API service to send reviews, and other actions. Scooby can also work without this component, but approvals won't work in the Web UI without it.

While Scooby has been built to be storage-agnostic, as of now, only AWS S3 buckets are supported as destination for reports and artifacts. If you are interested in other storage options, please open an issue!

The typical flow of a Scooby-powered report looks as follows:

1. The Scooby CLI is invoked to run a given report, pointing it to the relevant test dataset
2. After all report tasks have been run, the Scooby CLI is called to update the GitHub Status
3. Reports can be then viewed on the Scooby Web UI, and reviewers can approve/reject them

The following diagram illustrates an high-level overview of the system:

Getting started

This section covers the basics to use Scooby in your projects. We'll start from the initial repository setup, gradually covering all the steps to get a simple working pipeline.

Note for Scooby developers: this section does not cover how to contribute to the Scooby project. If your goal is to extend or modify Scooby itself, please head to the Developer guide section below.

First-time repository setup

If this is the first time Scooby is used inside your project, you'll need to set up a couple of things beforehand.

Environment variables

Scooby requires several environment variables to work. These variables provide the information and access tokens necessary to generate reports, upload artifacts, update GitHub statuses and more.

These are the environment variables you need to specify in the project CI:

If you are using CircleCI in your project, you'll need to specify them inside the Project Settings > Environment Variables section.

After the necessary environment variables have been set, you are ready to generate a Scooby report.

Generating your first Scooby report on CircleCI

In this section, we'll discuss the process to generate a visual regression report. In particular, the goal is to have an automated check that fails whenever some HTML pages change visually.

While this example focuses on visual regression reports, other report types can be generated following a similar process. Please refer to the Reference section below for more information.

Assuming that the test HTML files are located in the path/to/tests folder, you can generate a Scooby regression report by running this command:

npx @animaapp/scooby-cli regression --name "my-html-regression" --tests path/to/tests --file-type=html

This command tells Scooby to generate a regression report named my-html-regression based on all HTML files located in the path/to/tests folder. Scooby will automatically take care of comparing these HTML files with the known reference (usually the one on main branch).

Now that we generated the report, we are ready to trigger the GitHub status update, which will show the status for each report in our PRs and commits:

To update the GitHub status, you'll need to run:

npx @animaapp/scooby-cli update-status

For efficiency reasons, it's better to run the update-status step only after all Scooby report tasks have been completed. If you are running multiple reports in parallel on CircleCI, a common pattern is to use job dependencies:

- run-scooby-fidelity-tests
- run-scooby-regression-tests
- scooby-update-status:
    requires:
      - run-scooby-regression-tests
      - run-scooby-fidelity-tests

In this example, the scooby-update-status job is only run after the two other report tasks have been completed, thus invoking update-status only once.

FAQ

This section discusses some of the commonly asked questions related to Scooby or testing in general.

What's the difference between Fidelity and Regression tests?

While Fidelity and Regression tests share some common aspects, they are designed to answer two very different questions:

• Regression tests compare two different versions of the same dataset. For example, a regression test might verify that the output of a code generator inside a feature branch is not different from a known-good state on the main branch.
• Fidelity tests compare the same version of two different datasets. For example, a fidelity test might compare how visually similar a given web page is to its Figma design.

From these definitions, we can draw several conclusions:

• Regression tests usually produce a binary good/bad result for each test entry, while fidelity tests produce a continuous similarity ratio.
• Regression tests aim at having zero differences between the two datasets, as we are testing the same system at two different points in time.
• Fidelity tests could have differences and still be successful, as having a perfect match between datasets produced by two different systems might not be achieavable.

Reference

This section acts as a reference for the different tests that can be performed with Scooby, along with other useful commands.

Visual Regression tests

Visual regression tests can be used to verify if changes introduced in a commit (PR) are causing visual regressions.

In a nutshell, Scooby compares the dataset generated in a given commit (or PR) with the dataset of the base commit (main branch). This allow us to determine whether the changes in a given PR are changing the behavior of our production systems (the ones on main) in a visual way.

graph LR;
  prod(Production - main branch)
  pr(PR commit - feature branch)
  pr-->Scooby
  prod-->Scooby
  Scooby-->Regression[Regression Report]

Basic usage

npx @animaapp/scooby-cli regression --name "regression-name" --tests /path/to/tests --file-type=html

Visual Fidelity tests

npx @animaapp/scooby-cli fidelity --name "fidelity-name" --expected /path/to/expected/tests --actual /path/to/actual/tests --file-type=png

Developer guide

Running the CLI in Dev Mode

When working on Scooby, you might need to invoke the development version of the CLI with your changes. From the repository root, you can run the following command:

yarn cli:run:dev --help

This command will build the CLI from source and invoke it with the --help flag. It is equivalent to running npx @animaapp/scooby-cli --help.

By changing the argument, you can then test the various CLI commands, for example:

yarn cli:run:dev fidelity --name "fidelity-name" --expected /path/to/expected/tests --actual /path/to/actual/tests --file-type=png

Troubleshooting

Error: Could not find expected browser (chrome) locally. Run npm install to download the correct Chromium revision (1045629).

If you are on macOS you might experience the following error when running Scooby:

Error: Could not find expected browser (chrome) locally. Run `npm install` to download the correct Chromium revision (1045629).

To solve it, you can force a chromium installation with:

node node_modules/puppeteer/install.js