• Stars
    star
    3,257
  • Rank 13,789 (Top 0.3 %)
  • Language
    Ruby
  • Created over 15 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Ruby: 2 CPUs = 2x Testing Speed for RSpec, Test::Unit and Cucumber

parallel_tests

Gem Version Build status

Speedup Minitest + RSpec + Turnip + Cucumber + Spinach by running parallel on multiple CPU cores.
ParallelTests splits tests into balanced groups (by number of lines or runtime) and runs each group in a process with its own database.

Setup for Rails

RailsCasts episode #413 Fast Tests

Install

Gemfile:

gem 'parallel_tests', group: [:development, :test]

Add to config/database.yml

ParallelTests uses 1 database per test-process.

Process number123
ENV['TEST_ENV_NUMBER']'''2''3'
test:
  database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>

Create additional database(s)

rake parallel:create

(Multi-DB) Create individual database

rake parallel:create:<database>
rake parallel:create:secondary

Copy development schema (repeat after migrations)

rake parallel:prepare

Run migrations in additional database(s) (repeat after migrations)

rake parallel:migrate

(Multi-DB) Run migrations in individual database

rake parallel:migrate:<database>

Setup environment from scratch (create db and loads schema, useful for CI)

rake parallel:setup

Drop all test databases

rake parallel:drop

(Multi-DB) Drop individual test database

rake parallel:drop:<database>

Run!

rake parallel:test          # Minitest
rake parallel:spec          # RSpec
rake parallel:features      # Cucumber
rake parallel:features-spinach       # Spinach

rake "parallel:test[1]" --> force 1 CPU --> 86 seconds
rake parallel:test    --> got 2 CPUs? --> 47 seconds
rake parallel:test    --> got 4 CPUs? --> 26 seconds
...

Test by pattern with Regex (e.g. use one integration server per subfolder / see if you broke any 'user'-related tests)

rake "parallel:test[^test/unit]" # every test file in test/unit folder
rake "parallel:test[user]"  # run users_controller + user_helper + user tests
rake "parallel:test['user|product']"  # run user and product related tests
rake "parallel:spec['spec\/(?!features)']" # run RSpec tests except the tests in spec/features

Example output

2 processes for 210 specs, ~ 105 specs per process
... test output ...

843 examples, 0 failures, 1 pending

Took 29.925333 seconds

Run an arbitrary task in parallel

RAILS_ENV=test parallel_test -e "rake my:custom:task"
# or
rake "parallel:rake[my:custom:task]"
# limited parallelism
rake "parallel:rake[my:custom:task,2]"

Running things once

require "parallel_tests"

# preparation:
# affected by race-condition: first process may boot slower than the second
# either sleep a bit or use a lock for example File.lock
ParallelTests.first_process? ? do_something : sleep(1)

# cleanup:
# last_process? does NOT mean last finished process, just last started
ParallelTests.last_process? ? do_something : sleep(1)

at_exit do
  if ParallelTests.first_process?
    ParallelTests.wait_for_other_processes_to_finish
    undo_something
  end
end

Even test group runtimes

Test groups will often run for different times, making the full test run as slow as the slowest group.

Step 1: Use these loggers (see below) to record test runtime Step 2: Your next run will use the recorded test runtimes (use --runtime-log <file> if you picked a location different from below)

RSpec

Rspec: Add to your .rspec_parallel (or .rspec) :

--format progress
--format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log

To use a custom logfile location (default: tmp/parallel_runtime_rspec.log), use the CLI: parallel_test spec -t rspec --runtime-log my.log

Minitest

Add to your test_helper.rb:

require 'parallel_tests/test/runtime_logger' if ENV['RECORD_RUNTIME']

results will be logged to tmp/parallel_runtime_test.log when RECORD_RUNTIME is set, so it is not always required or overwritten.

TODO: add instructions for other frameworks

Loggers

RSpec: SummaryLogger

Log the test output without the different processes overwriting each other.

Add the following to your .rspec_parallel (or .rspec) :

--format progress
--format ParallelTests::RSpec::SummaryLogger --out tmp/spec_summary.log

RSpec: FailuresLogger

Produce pasteable command-line snippets for each failed example. For example:

rspec /path/to/my_spec.rb:123 # should do something

Add to .rspec_parallel or use as CLI flag:

--format progress
--format ParallelTests::RSpec::FailuresLogger --out tmp/failing_specs.log

(Not needed to retry failures, for that pass --only-failures to rspec)

RSpec: VerboseLogger

Prints a single line for starting and finishing each example, to see what is currently running in each process.

# PID, parallel process number, spec status, example description
[14403] [2] [STARTED] Foo foo
[14402] [1] [STARTED] Bar bar
[14402] [1] [PASSED] Bar bar

Add to .rspec_parallel or use as CLI flag:

  --format ParallelTests::RSpec::VerboseLogger

Cucumber: FailuresLogger

Log failed cucumber scenarios to the specified file. The filename can be passed to cucumber, prefixed with '@' to rerun failures.

Usage:

cucumber --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log

Or add the formatter to the parallel: profile of your cucumber.yml:

parallel: --format progress --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log

Note if your cucumber.yml default profile uses <%= std_opts %> you may need to insert this as follows parallel: <%= std_opts %> --format progress...

To rerun failures:

cucumber @tmp/cucumber_failures.log

Setup for non-rails

gem install parallel_tests
# go to your project dir
parallel_test
parallel_rspec
parallel_cucumber
parallel_spinach
  • use ENV['TEST_ENV_NUMBER'] inside your tests to select separate db/memcache/etc. (docker compose: expose it)

  • Only run a subset of files / folders:

    parallel_test test/bar test/baz/foo_text.rb

  • Pass test-options and files via --:

    parallel_rspec -- -t acceptance -f progress -- spec/foo_spec.rb spec/acceptance

  • Pass in test options, by using the -o flag (wrap everything in quotes):

    parallel_cucumber -n 2 -o '-p foo_profile --tags @only_this_tag or @only_that_tag --format summary'

Options are:

-n [PROCESSES]                   How many processes to use, default: available CPUs
-p, --pattern [PATTERN]          run tests matching this regex pattern
    --exclude-pattern [PATTERN]  exclude tests matching this regex pattern
    --group-by [TYPE]            group tests by:
                                 found - order of finding files
                                 steps - number of cucumber/spinach steps
                                 scenarios - individual cucumber scenarios
                                 filesize - by size of the file
                                 runtime - info from runtime log
                                 default - runtime when runtime log is filled otherwise filesize
-m, --multiply-processes [FLOAT] use given number as a multiplier of processes to run
-s, --single [PATTERN]           Run all matching files in the same process
-i, --isolate                    Do not run any other tests in the group used by --single(-s)
    --isolate-n [PROCESSES]      Use 'isolate'  singles with number of processes, default: 1
    --highest-exit-status        Exit with the highest exit status provided by test run(s)
    --failure-exit-code [INT]    Specify the exit code to use when tests fail
    --specify-groups [SPECS]     Use 'specify-groups' if you want to specify multiple specs running in multiple
                                 processes in a specific formation. Commas indicate specs in the same process,
                                 pipes indicate specs in a new process. Cannot use with --single, --isolate, or
                                 --isolate-n.  Ex.
                                 $ parallel_tests -n 3 . --specify-groups '1_spec.rb,2_spec.rb|3_spec.rb'
                                   Process 1 will contain 1_spec.rb and 2_spec.rb
                                   Process 2 will contain 3_spec.rb
                                   Process 3 will contain all other specs
    --only-group INT[,INT]       Only run the given group numbers.
                                 Changes `--group-by` default to 'filesize'.
-e, --exec [COMMAND]             execute this code parallel and with ENV['TEST_ENV_NUMBER']
-o, --test-options '[OPTIONS]'   execute test commands with those options
-t, --type [TYPE]                test(default) / rspec / cucumber / spinach
    --suffix [PATTERN]           override built in test file pattern (should match suffix):
                                 '_spec.rb$' - matches rspec files
                                 '_(test|spec).rb$' - matches test or spec files
    --serialize-stdout           Serialize stdout output, nothing will be written until everything is done
    --prefix-output-with-test-env-number
                                 Prefixes test env number to the output when not using --serialize-stdout
    --combine-stderr             Combine stderr into stdout, useful in conjunction with --serialize-stdout
    --non-parallel               execute same commands but do not in parallel, needs --exec
    --no-symlinks                Do not traverse symbolic links to find test files
    --ignore-tags [PATTERN]      When counting steps ignore scenarios with tags that match this pattern
    --nice                       execute test commands with low priority.
    --runtime-log [PATH]         Location of previously recorded test runtimes
    --allowed-missing [INT]      Allowed percentage of missing runtimes (default = 50)
    --allow-duplicates           When detecting files to run, allow duplicates
    --unknown-runtime [FLOAT]    Use given number as unknown runtime (otherwise use average time)
    --first-is-1                 Use "1" as TEST_ENV_NUMBER to not reuse the default test environment
    --fail-fast                  Stop all groups when one group fails (best used with --test-options '--fail-fast' if supported
    --verbose                    Print debug output
    --verbose-command            Combines options --verbose-process-command and --verbose-rerun-command
    --verbose-process-command    Print the command that will be executed by each process before it begins
    --verbose-rerun-command      After a process fails, print the command executed by that process
    --quiet                      Print only tests output
-v, --version                    Show Version
-h, --help                       Show this.

You can run any kind of code in parallel with -e / --exec

parallel_test -n 5 -e 'ruby -e "puts %[hello from process #{ENV[:TEST_ENV_NUMBER.to_s].inspect}]"'
hello from process "2"
hello from process ""
hello from process "3"
hello from process "5"
hello from process "4"
1 Process2 Processes4 Processes
RSpec spec-suite18s14s10s
Rails-ActionPack88s53s44s

TIPS

RSpec

Cucumber

  • Add a parallel: foo profile to your config/cucumber.yml and it will be used to run parallel tests
  • ReportBuilder can help with combining parallel test results
    • Supports Cucumber 2.0+ and is actively maintained
    • Combines many JSON files into a single file
    • Builds a HTML report from JSON with support for debug msgs & embedded Base64 images.

General

  • [ZSH] use quotes to use rake arguments rake "parallel:prepare[3]"
  • [Memcached] use different namespaces
    e.g. config.cache_store = ..., namespace: "test_#{ENV['TEST_ENV_NUMBER']}"
  • Debug errors that only happen with multiple files using --verbose and cleanser
  • export PARALLEL_TEST_PROCESSORS=13 to override default processor count
  • Shell alias: alias prspec='parallel_rspec -m 2 --'
  • [Spring] Add the spring-commands-parallel-tests gem to your Gemfile to get parallel_tests working with Spring.
  • --first-is-1 will make the first environment be 1, so you can test while running your full suite.
    export PARALLEL_TEST_FIRST_IS_1=true will provide the same result
  • email_spec and/or action_mailer_cache_delivery
  • zeus-parallel_tests
  • Distributed Parallel Tests on CI systems) learn how parallel_tests can run on distributed servers such as Travis and GitLab-CI. Also shows you how to use parallel_tests without adding TEST_ENV_NUMBER-backends
  • Capybara setup
  • Sphinx setup
  • Capistrano setup let your tests run on a big box instead of your laptop

Contribute your own gotchas to the Wiki or even better open a PR :)

Authors

inspired by pivotal labs

Michael Grosser
[email protected]
License: MIT

More Repositories

1

parallel

Ruby: parallel processing made simple and fast
Ruby
4,052
star
2

pru

Pipeable Ruby - forget about grep / sed / awk / wc ... use pure, readable Ruby!
Ruby
579
star
3

smusher

Ruby/CLI: Automatic lossless reduction of all your images
Ruby
555
star
4

maxitest

Minitest + all the features you always wanted.
Ruby
441
star
5

fast_gettext

Ruby GetText, but 12x faster + 530x less garbage + simple + clean namespace + threadsafe + extendable + multiple backends
Ruby
390
star
6

wwtd

WWTD: Travis simulator - faster + no more waiting for build emails
Ruby
366
star
7

rspec-instafail

Show failing specs instantly
Ruby
272
star
8

gettext_i18n_rails

Rails: FastGettext, I18n integration -- simple, threadsafe and fast!
Ruby
257
star
9

test_after_commit

Make after_commit callbacks fire in tests for Rails 3+ with transactional_fixtures = true.
Ruby
240
star
10

rpx_now

Ruby: RPXNow.com user login/creation and view helpers Facebook, Twitter, Google, MSN, OpenID, MySpace, Yahoo -- All in One
Ruby
230
star
11

single_cov

Actionable code coverage.
Ruby
226
star
12

bitfields

n Booleans = 1 Integer, saves columns and migrations.
Ruby
221
star
13

i18n_data

Ruby: country/language names and 2-letter-code pairs, in 85 languages, for country/language i18n
Ruby
186
star
14

vendorer

Vendorer keeps your dependencies documented, cached and up to date
Ruby
186
star
15

ar_after_transaction

Execute irreversible actions only when transactions are not rolled back
Ruby
155
star
16

url_store

Data securely stored in urls.
Ruby
146
star
17

kennel

Datadog monitors/dashboards/slos as code, avoid chaotic management via UI
Ruby
129
star
18

ruco

Desktop-style, Intuitive, Commandline Editor in Ruby. "Better than nano, simpler than vim."
Ruby
125
star
19

programming_pearls

eBook: Programming Pearls Rewritten in Ruby
Ruby
108
star
20

easy_esi

Rails: Cached pages with updated partials
Ruby
106
star
21

reduce

Ruby/CLI: minify javascript + stylesheets, lossless image optimization
JavaScript
90
star
22

git-autobisect

Find the first broken commit without having to learn git bisect
Ruby
84
star
23

parallel_split_test

Split a big test file into multiple chunks and run them in parallel
Ruby
81
star
24

tic_tac_toe

Play Tic-Tac-Toe in Ruby using Curses(full-screen-commandline app)
Ruby
71
star
25

sort_alphabetical

Ruby: sort UTF8 Strings alphabetical via Enumerable extension
Ruby
69
star
26

soft_deletion

Explicit soft deletion for ActiveRecord via deleted_at and default scope
Ruby
68
star
27

youtube_search

Search youtube via this simple ruby api
Ruby
67
star
28

simple_auto_complete

Rails: Simple, customizable, unobstrusive - Autocomplete
JavaScript
64
star
29

dispel

Ruby: Remove evil curses
Ruby
63
star
30

preoomkiller

Softly kills your process with SIGTERM before it runs out of memory.
Ruby
63
star
31

single_test

Rake tasks to invoke single tests/specs with rakish syntax
Ruby
60
star
32

dotfiles

Clean and powerful dotfiles -- bash / git / ruby / irb / nano / ruco
Shell
57
star
33

bundler-organization_audit

Automatic Gemfile security audit for all your organizaition/user repos
Ruby
52
star
34

fallback

Fallback when original attribute is not present or somethings not right.
Ruby
47
star
35

record_activities

Rails: Record user activities without controller helpers, build on top of userstamps plugin
Ruby
47
star
36

ie_iframe_cookies

Rails: Normal cookies inside IFrames for IE via P3P headers
Ruby
45
star
37

cachy

Ruby: Caching library to simplify and organize caching
Ruby
44
star
38

zombie_passenger_killer

Guaranteed zombie passengers death.
Ruby
44
star
39

tracked_plugins

script/plugin now keeps track of installation, can list urls/revisions/install-dates/plugin-locally-hacked? and update.
Ruby
43
star
40

stub_server

Boot up a real server to serve testing replies
Ruby
43
star
41

rails2_asset_pipeline

Familiar asset handling for those stuck on Rails 2
Ruby
41
star
42

request_recorder

Record your rack/rails requests and store them for future inspection
Ruby
40
star
43

ar_merge

Merge 2 ActiveRecords, preserving attributes, associations and counters
Ruby
40
star
44

gem-dependent

How many gems depend on your gem ?
Ruby
39
star
45

travis_dedup

Stop all builds on the same PR when a new job starts
Ruby
38
star
46

safe_regexp

Ruby Regex Timeout / Backtracking Bomb Safety
Ruby
30
star
47

translation_db_engine

Rails/AR: engine to manage translations inside a database
Ruby
30
star
48

rubinjam

Covert ruby gem to universal cross-platform binary
Ruby
30
star
49

gettext_i18n_rails_example

Rails example application using FastGettext + gettext_i18n_rails + gettext_test_log
Ruby
29
star
50

url_to_media_tag

Convert an Youtube/Vimeo/Image... Url to image or video embed.
Ruby
27
star
51

forking_test_runner

Run every test in a fork to avoid pollution and get clean output per test
Ruby
27
star
52

restful_catch_all_route

One rule for complete restful routing, no helpers, no worries.
Ruby
26
star
53

scopify

Add named scopes and chainable scopes to any Object / Model.
Ruby
24
star
54

key_value

Abuse Sql database as Key-Value Store
Ruby
22
star
55

acts_as_feed

Rails/AR: Transform a Model into a Feed Representation (Feed Reader)
Ruby
22
star
56

sinatra-magick

Sinatra app to manipulate images given by url via mini_magick and image_magick. completly evented
Ruby
22
star
57

concern

Ruby: Seperation of concerns without meta-madness and namespace pollution.
Ruby
22
star
58

has_a_location

AR: Easy location (lat/long) handling + in_radius + find on a given map section
Ruby
22
star
59

virtual_asset_path

Improve Rails Asset Caching with MD5 and virtual folders
Ruby
21
star
60

s3_slider

jQuery: slideshow displaying images + description, ~1kb for js+css, simple and elegant
HTML
21
star
61

git-whence

Find the merge and pull request a commit came from + fuzzy search for cherry-picks
Ruby
21
star
62

readable_random

Ruby: Readable random strings for coupons or tokens
Ruby
20
star
63

testrbl

Run ruby Minitest/Test::Unit/Spec/Shoulda tests by line-number / files / folder
Ruby
20
star
64

s3_meta_sync

Efficiently sync folders with s3 using a metadata file with md5 sums.
Ruby
20
star
65

countries_and_languages

Rails: Countries and languages in I18n.locale for select_tag or output in 85 languages
Ruby
19
star
66

autoscaling

Amazon AWS/EC2 autoscaling All in one
Shell
19
star
67

helpful_fields

Simple & Helpful Field Helpers for Rails e.g. check_box_with_label or prefilled fields from params
Ruby
18
star
68

go-testcov

`go test` that fails on uncovered lines and shows them
Go
18
star
69

cleanser

Find polluting test by bisecting your tests.
Ruby
17
star
70

repo_dependency_graph

Graph the dependencies of your repositories
Ruby
16
star
71

get_pomo

Ruby/Gettext: A .mo and .po file parser/generator
Ruby
16
star
72

honeypot

Rails: Simple honeypots
Ruby
16
star
73

random_records

Rails/AR: Fast random records for ActiveRecord
Ruby
16
star
74

rpx_now_example

Example Rails app using RPXNow plugin
Ruby
16
star
75

logrecycler

Re-process logs from applications you cannot modify to convert them to json and add prometheus/stats metrics
Go
15
star
76

after_commit_exception_notification

Rails: Get notified when an after_commit block blows up
Ruby
15
star
77

textpow

Read TextMate syntax files and parse text with them
Ruby
14
star
78

codeclimate_batch

Report a batch of codeclimate results by merging and from multiple servers
Ruby
13
star
79

dockerb

Dockerfile.erb - use ruby in your dynamic Dockerfile
Ruby
12
star
80

cia

Central Internal Auditing: Audit model events like update/create/delete + attribute changes + grouped them by transaction, in normalized table layout for easy query access.
Ruby
12
star
81

cc-amend

Unify reports from all your tests runs and send them as one.
Ruby
12
star
82

gem_of_thrones

Everybody wants to be king, but only one can rule (synchronized via a distributed cache)
Ruby
12
star
83

github-grep

Grep through github search results
Ruby
11
star
84

kube-leader

Simple Kubernetes Leader Election via ConfigMap as ENTRYPOINT
Go
10
star
85

travis_cron

Run travis as cron (also supports travis PRO)
Ruby
10
star
86

air_man

Email notifications for high-frequency Airbrake errors
Ruby
10
star
87

i18n-backend-http

Rails I18n Backend for Http APIs with etag-aware distributed background polling and lru-memory+[memcache] caching.
Ruby
9
star
88

gem_on_demand

Run your own gem server that fetches from github, uses tags as version and builds gems on demand
Ruby
9
star
89

matching_bundle

Find a matching bundler version for a Gemfile and use it
Ruby
9
star
90

active_record-comments

Add comments to ActiveRecord queries to see where they came from or what user caused them
Ruby
9
star
91

translated_attributes

AR/Rails translatable attributes through virtual fields
Ruby
9
star
92

cmd2json

Covert command output and exit status to json to pipe them atomically into logs
Ruby
8
star
93

autolang

Automatic translation to a new language for Gettext/JSON using Google translate
Ruby
8
star
94

db_graph

Easy graphs from AR date fields
Ruby
8
star
95

organization_license_audit

Audit all licenses used by your github organization/user
Ruby
8
star
96

ruby-cli-daemon

Make all gem executables execute instantly
Ruby
8
star
97

ar_multi_threaded_transactional_tests

Execute multithreaded code while still using transactional fixtures by synchronizing db access to a single connection
Ruby
8
star
98

rhr

Ruby Hypertext Refinement -- the ease of PHP with the elegance of Ruby
Ruby
7
star
99

git-graph

Date porn from your git history
Ruby
7
star
100

unicorn_wrangler

Unicorn: out of band GC / restart on max memory bloat / restart after X requests
Ruby
7
star