Discover avmnu-sng/rspec-tracer Open Source project

RSpec Tracer is a specs dependency analyzer, flaky tests detector, tests accelerator, and coverage reporter tool. It maintains a list of files for each test, enabling itself to skip tests in the subsequent runs if none of the dependent files are changed.

It uses Ruby's built-in coverage library to keep track of the coverage for each test. For each test executed, the coverage diff provides the desired file list. RSpec Tracer takes care of reporting the correct code coverage when skipping tests by using the cached reports. Also, note that it will never skip:

Flaky examples
Failed examples
Pending examples

Knowing the examples and files dependency gives us a better insight into the codebase, and we have a clear idea of what to test for when making any changes. With this data, we can also analyze the coupling between different components and much more.

RSpec Tracer requires Ruby 2.5+ and rspec-core >= 3.6.0. To use with Rails 5+, make sure to use rspec-rails >= 4.0.0. If you are using SimpleCov, it is recommended to use simplecov >= 0.17.0. To use RSpec Tracer cache on CI, you need to have an S3 bucket and AWS CLI installed.

You should take some time and go through the document describing the intention and implementation details of managing dependency, managing flaky tests, skipping tests, and caching on CI.

Getting Started
- Working with JRuby
- Working with Parallel Tests
Configuring CI
Advanced Configuration
- Available Settings
Filters
Duplicate Examples
Demo

Getting Started

Add this line to your Gemfile and bundle install:

gem 'rspec-tracer', '~> 1.0', group: :test, require: false

And, add the followings to your .gitignore:

rspec_tracer.lock
rspec_tracer_cache/
rspec_tracer_coverage/
rspec_tracer_report/

Load and launch RSpec Tracer at the very top of spec_helper.rb (or rails_helper.rb, test/test_helper.rb). Note that RSpecTracer.start must be issued before loading any of the application code.
```
# Load RSpec Tracer
require 'rspec_tracer'
RSpecTracer.start
```
If you are using SimpleCov, load RSpec Tracer right after the SimpleCov load and launch:
```
require 'simplecov'
SimpleCov.start

# Load RSpec Tracer
require 'rspec_tracer'
RSpecTracer.start
```
If you use RSpec Tracer with SimpleCov, then SimpleCov would not report branch coverage results even when enabled.
After running your tests, open rspec_tracer_report/index.html in the browser of your choice.

Working with JRuby

It is recommend to use JRuby 9.2.10.0+. Also, configure it with JRUBY_OPTS="--debug -X+O" or have the .jrubyrc file:

debug.fullTrace=true
objectspace.enabled=true

Working with Parallel Tests

The Rspec tracer, by default, supports working with parallel_tests gem. It maintains a lock file rspec_tracer.lock to identify the last running process. Usually, you are not required to do anything special unless you interrupt the execution in between and the process did not complete correctly. In such a case, you must delete the lock file before the next run.

rm -f rspec_tracer.lock && bundle exec parallel_rspec

Configuring CI

To enable RSpec Tracer to share cache between different builds on CI, update the Rakefile in your project to have the following:

spec = Gem::Specification.find_by_name('rspec-tracer')

load "#{spec.gem_dir}/lib/rspec_tracer/remote_cache/Rakefile"

Before running tests, download the remote cache using the following rake task:

bundle exec rake rspec_tracer:remote_cache:download

After running tests, upload the local cache using the following rake task:

bundle exec rake rspec_tracer:remote_cache:upload

You must set the following environment variables:

GIT_DEFAULT_BRANCH is the default branch name for the repo, e.g., main or master.
GIT_BRANCH is the git branch name you are running the CI build on.
TEST_SUITES is the total number of different test suites you are running.
```
export TEST_SUITES=8
```
TEST_SUITE_ID is the CI build ID. If you have a large set of tests to run, it is recommended to run them in separate groups. This way, RSpec Tracer is not overwhelmed with loading massive cached data in the memory. Also, it generates and uses cache for specific test suites and not merge them.
```
TEST_SUITE_ID=1 bundle exec rspec spec/models
TEST_SUITE_ID=2 bundle exec rspec spec/helpers
```

Advanced Configuration

Configuration settings must be defined in .rspec-tracer file:

RSpecTracer.configure do
  config_option 'foo'
end

You can additionally define a global config file ~/.rspec-tracer to share the common settings across projects.

Available Settings

root dir to set the project root. The default value is the current working directory.
project_name name to set the project name. The default value is the last part of the project root directory.
cache_dir dir to set the cache directory. The default value is ./rspec_tracer_cache. You can also set the RSPEC_TRACER_CACHE_DIR environment variable.
coverage_dir dir to set the coverage reports directory. The default is the ./rspec_tracer_coverage. You can also set the RSPEC_TRACER_COVERAGE_DIR environment variable.
report_dir dir to set the HTML reports directory. The default is the ./rspec_tracer_report. You can also set the RSPEC_TRACER_REPORT_DIR environment variable.
reports_s3_path uri to set the AWS S3 URI for all the reports from the current run. You can also set the RSPEC_TRACER_REPORTS_S3_PATH environment variable.
use_local_aws bool_flag to use the awslocal AWS CLI with LocalStack. You can also set the RSPEC_TRACER_USE_LOCAL_AWS environment variable.
upload_non_ci_reports bool_flag to upload execution reports in a non-CI environment. You can also set the RSPEC_TRACER_UPLOAD_NON_CI_REPORTS environment variable.
run_all_examples bool_flag to always run all the examples irrespective of cache. You can also set the RSPEC_TRACER_RUN_ALL_EXAMPLES environment variable.
fail_on_duplicates bool_flag to fail with a non-zero exit code in case of duplicate examples. The default value is true. You can also set the RSPEC_TRACER_FAIL_ON_DUPLICATES environment variable.
lock_file file to set the lock file when executing with parallel_tests. The default value is ./rspec_tracer.lock. You can also set the RSPEC_TRACER_LOCK_FILE environment variable.
log_level level to set the log level. The default value is info. The possible values are off, debug, info, warn, and error. You can also set the RSPEC_TRACER_LOG_LEVEL environment variable.
add_filter filter to apply filters on the source files to exclude them from the dependent files list.
filters.clear to remove the configured dependent files filters so far.
add_coverage_filter filter to apply filters on the source files to exclude them from the coverage report.
coverage_filters.clear to remove the configured coverage files filters so far.
coverage_track_files glob to include files in the given glob pattern in the coverage report if these files are not already present.

RSpecTracer.configure do
  # Configure project root
  root '/tmp/my_project'

  # Clear existing filters
  filters.clear
  # Add dependent files filter
  add_filter %r{^/tasks/}

  # Clear existing coverage filters
  coverage_filters.clear
  # Add coverage files filter
  add_coverage_filter %w[/features/ /spec/ /tests/]

  # Define glob to track files in the coverage report
  coverage_track_files '{app,lib}/**/*.rb'
end

Filters

By default, RSpec Tracer ignores all the files outside of the project root directory - otherwise you would end up with the source files in the gems you are using in the project. It also applies the following filters:

RSpecTracer.configure do
  log_level 'warn'

  add_filter '/vendor/bundle/'

  add_coverage_filter %w[
    /autotest/
    /features/
    /spec/
    /test/
    /vendor/bundle/
  ].freeze
end

Defining Custom Filteres

You can currently define a filter using either a String or Regexp (that will then be Regexp-matched against each source file's name relative to the project root), a block or by passing in your own Filter class.

String Filter: The string filter matches files that have the given string in their name. For example, the following string filter will remove all files that have "/helpers/" in their name.
```
RSpecTracer.configure do
  add_filter '/helpers/'
end
```
Regex Filter: The regex filter removes all files that have a successful name match against the given regex expression. This simple regex filter will remove all files that start with %r{^/helper/} in their name:
```
RSpecTracer.configure do
  add_filter %r{^/helpers/}
end
```
Block Filter: Block filters receive a Hash object and expect your block to return either true (if the file is to be removed from the result) or false (if the result should be kept). In the below example, the filter will remove all files that match "/helpers/" in their path.
```
RSpecTracer.configure do
  add_filter do |source_file|
    source_file[:file_path].include?('/helpers/')
  end
end
```
You can also use source_file[:name] to define the return value of the block filter for the given source file.
Array Filter: You can pass in an array containing any of the other filter types:
```
RSpecTracer.configure do
  add_filter ['/helpers/', %r{^/utils/}]
end
```

Duplicate Examples

To uniquely identify the examples is one of the requirements for the correctness of the RSpec Tracer. Sometimes, it would not be possible to do so depending upon how we have written the specs. The following attributes determine the uniqueness:

The example group
The example full description
The spec file location, i.e., file name and line number
All the shared examples and contexts

Consider the following Calculator module:

module Calculator
  module_function

  def add(a, b) a + b; end
  def sub(a, b) a - b; end
  def mul(a, b) a * b; end
end

And the corresponding spec file spec/calculator_spec.rb:

RSpec.describe Calculator do
  describe '#add' do
    [
      [1, 2, 3],
      [0, 0, 0],
      [5, 32, 37],
      [-1, -8, -9],
      [10, -10, 0]
    ].each { |a, b, r| it { expect(described_class.add(a, b)).to eq(r) } }
  end

  describe '#sub' do
    [
      [1, 2, -1],
      [10, 0, 10],
      [37, 5, 32],
      [-1, -8, 7],
      [10, 10, 0]
    ].each do |a, b, r|
      it 'performs subtraction' do
        expect(described_class.sub(a, b)).to eq(r)
      end
    end
  end

  describe '#mul' do
    [
      [1, 2, -2],
      [10, 0, 0],
      [5, 7, 35],
      [-1, -8, 8],
      [10, 10, 100]
    ].each do |a, b, r|
      it "multiplies #{a} and #{b} to #{r}" do
        expect(described_class.mul(a, b)).to eq(r)
      end
    end
  end
end

Running the spec with bundle exec rspec spec/calculator_spec.rb generates the following output:

Calculator
  #mul
    multiplies 5 and 7 to 35
    multiplies 10 and 10 to 100
    multiplies 10 and 0 to 0
    multiplies 1 and 2 to -2 (FAILED - 1)
    multiplies -1 and -8 to 8
  #add
    example at ./spec/calculator_spec.rb:13
    example at ./spec/calculator_spec.rb:13
    example at ./spec/calculator_spec.rb:13
    example at ./spec/calculator_spec.rb:13
    example at ./spec/calculator_spec.rb:13
  #sub
    performs subtraction
    performs subtraction
    performs subtraction
    performs subtraction
    performs subtraction

In this scenario, RSpec Tracer cannot determine the Calculator#add and Calculator#sub group examples.

================================================================================
   IMPORTANT NOTICE -- RSPEC TRACER COULD NOT IDENTIFY SOME EXAMPLES UNIQUELY
================================================================================
RSpec tracer could not uniquely identify the following 10 examples:
  - Example ID: eabd51a899db4f64d5839afe96004f03 (5 examples)
      * Calculator#add (spec/calculator_spec.rb:13)
      * Calculator#add (spec/calculator_spec.rb:13)
      * Calculator#add (spec/calculator_spec.rb:13)
      * Calculator#add (spec/calculator_spec.rb:13)
      * Calculator#add (spec/calculator_spec.rb:13)
  - Example ID: 72171b502c5a42b9aa133f165cf09ec2 (5 examples)
      * Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
      * Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
      * Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
      * Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
      * Calculator#sub performs subtraction (spec/calculator_spec.rb:24)