skaes/railsbench

Railsbench is a small collection of ruby scripts which make measuring
raw performance of rails apps a snap. All tests are run from the
command prompt, making performance regression testing easy.

In addition, a patch for the ruby garbage collector is provided, which
can be used to reduce the amount of time spent doing garbage
collection, trading memory for speed, as usual (see file GCPATCH for
details). Applying the patch will enhance performance data obtained
from the various scripts (and some won't run at all without the
patch).

This software was written and conceived by Stefan Kaes. The author can
be reached via email: <[email protected]>. Please send comments, bug
reports, patches or feature requests to this address.


FILES

  railsbench.rb
                - defines classes RailsBenchmark and RailsBenchmarkWithActiveRecordStore
                  switches:
                  -gcXXX        : perform gc after XXX requests
                  -log[=level]  : turn on rails logging (at given level)
                  -nocache      : turn off rails caching
                  -path         : print $: after loading rails and exit

  config/benchmarks.yml
                - specification of urls to benchmark
                  install into $RAILS_ROOT/config using `railsbench install'
                  (use railsbench install)

  config/benchmarks.rb
                - defines constant RAILSBENCH which is used by script perf_bench
                  modify this to add custom argument processing and to put data
                  into the session
                  install to $RAILS_ROOT/config using `railsbench install'

  perf_bench n options
                - main ruby script to run a given benchmark
                  switches (in addition to railsbench switches):
                  -mix      : alternates urls in given benchmark

  run_urls n options
                - run a given benchmark (without benchmarking)
                  useful for checking response codes, like so:

                      run_urls 1 -bm=all | grep Status:

                  switches as for perf_bench plus
                  -warmup   : run all urls once before measuring
                  -svlPV    : run test using SVL Ruby Performance Validator
                  -svlMV    : run test using SVL Ruby Memory Validator

  perf_run n [ option-string [ config-name ] ]
                - run a given benchmark, store performance data in a file
                  in directory $RAILS_PERF_DATA and print results

  perf_diff n common-options option-string1 option-string2 [ config-name1 [ config-name2 ] ]
                - run a given benchmark with two different sets of arguments
                  store data into directory $RAILS_PERF_DATA and print
                  comparison data

  perf_loop n options
                - used by perf_run and perf_diff
                  calls perf_bench $RAILS_PERF_RUNS times

  perf_times file
                - analyse and print performance data

  perf_comp [-narrow] [-skip_urls] file1 file2
                - compare two performance data sets and print results
                  -narrow    => produce narrow output
                  -skip_urls => don't print url map (use with -narrow)

  perf_html [-nocss] [-gc] [file]
                - format output taken from perf_comp and produce a HTML table
                  for inclusion in HTML pages. Reads from standard input.
                  options:
                    -noccs   suppress CSS output
                    -notable suppress table output
                    -gc      include gc statistics

  perf_run_gc n [ option-string [ config-name ] ]
                - run a given benchmark, store performance data in a file
                  in directory $RAILS_PERF_DATA and print results
                - requires Ruby GC patch (or Ruby Enterprise Edition)

  perf_times_gc file
                - analyse and print garbage collection statistics, which you
                  can produce by running perf_run_gc

  perf_diff_gc n common-options option-string1 option-string2 [ config-name1 [ config-name2 ] ]
                - run a given benchmark with two different sets of arguments
                  store GC data into directory $RAILS_PERF_DATA and print
                  GC stats comparison using perf_comp_gc
                - requires Ruby GC patch

  perf_comp_gc file1 file2
                - compare two GC performance data sets and print results

  perf_prof n [ option-string [ config-name ] ]
                - run a given benchmark using ruby-prof for profiling,
                  store profile data in a HTML file in directory $RAILS_PERF_DATA
                  file name is computed from date and benchmark name as described above
                  but has a .html extension instead of .txt
                - a number of options to steer ruby-prof are available:
                    -ruby_prof=number[/number]
                      sets threshold and min_percent for ruby-prof (defaults to 0.1/1)
                    -profile_type=stack|grind|flat|graph|multi
                      selects the profile format (defaults to stack)
                    -measure_mode=process_time|wall_time|cpu_time|allocations|memory
                      selects what to measure (default to wall_time)

  perf_plot [ options ] file1 file2 ...
                - plot performance data from raw performance files using gruff or gnuplot
                  see source for options

  perf_plot_gc [ options ] file1 file2 ...
                - plot data points from GC performance data stored in raw GC log files using gruff or gnuplot
                  this basically shows object type distribution across garbage collections
                  see source for options

  perf_table [ options ] file1 file2 ...
                - produces a tabular overview of perf data from given raw data files
                  see source for options

  analyze_heap_dump file
                - produces a html representation of a ruby heap dump.
                  useful for finding memory leaks.

ENVIRONMENT

  RAILS_ROOT
                - can be set to point to your rails app. if not set, railsbench can only
                  be called from the top level directory of your rails app

  RAILS_PERF_DATA
                - performance data sets will be stored into this directory
                  if not set, $HOME will be used

  RAILS_PERF_RUNS
                - the number of times perf_loop will run perf_bench on a single invocation
                  if not set, 3 runs will be performed

  RAILS_BENCHMARK_FILE
                - perf_bench sends its output to this file


INVOCATION

  The gem version installs a driver script called 'railsbench' into
  Ruby's bin directory. Individual commands can be called by prefixing
  with railsbench. If you tire of typing railsbench all the time, you
  can either define an alias (alias rb="railsbench"), or you can
  include railsbench's script directory into your seach path. In this
  case you need to run 'sudo railsbench postinstall' to make the
  scripts executable.


USAGE

  The two main scripts are named perf_run and perf_diff.

      perf_run 100

  runs the list of urls named "default" specified in benchmkarks.yml
  (see below), requesting each url $RAILS_PERF_RUNS * 100 times.

     perf_run 100 "-bm=list -aws"

  runs the benchmark named 'list' and passes the expanded second
  argument to the rails app. By processing arguments inside your
  environment.rb file, you can set performance affecting options. For
  example, you could load action web service only if -aws is passed and
  measure the performance effect of omitting it.

  Benchmark data is stored in directory $RAILS_PERF_DATA, which should
  be set in your environment. If not set, $HOME is used. By default,
  data is stored in file $RAILS_PERF_DATA/perf_run.$BENCHMARK.txt, where
  BENCHMARK will be set according to the -bm option.

     perf_run 100 "-bm=index -mail" mail

  will store benchmark data in file

     $RAILS_PERF_DATA/<current-date>.index.mail.txt.

  You can get nicely formatted output of benchmark data by running

    perf_times file

  Script perf_run will automatically print the obtained data after
  finishing the run using perf_times.

  Script perf_diff runs a list of urls with two different option lists. So

    perf_diff 100 "-bm=blogs -mysql_session" "-mail=0" "-mail=1" cf1 cf2

  would run benchmark 'blogs' twice, first as

    perf_run 100 "-bm=blogs -mysql_session -mail=0" cf1

  and then

    perf_run 100 "-bm=blogs -mysql_session -mail=1" cf2

  printing a comparison of the bechmark data obtained after finishing
  the second run. cf1 and cf2 can be omitted, in which case data is
  stored in $RAILS_PERF_DATA/perf_run1.$BENCHMARK.txt and
  $RAILS_PERF_DATA/perf_run2.$BENCHMARK.txt.

  Script perf_bench can also be invoked manually to run a given
  benchmark like so:

    perf_bench 100 -bm=blogs -mysql_session -mail=1 >/dev/null

  Performance data is sent to $RAILS_BENCHMARK_FILE, HTML output ends up
  on stdout. If RAILS_BENCHMARK_FILE is not set, performance data is
  sent to stderr.

  Scripts perf_run_gc and perf_times_gc can be used to analyse GC performance:

    perf_run_gc 100 "-bm=all -lib=stable11" gc.log
    perf_times_gc gc.log

  It will produce output looking like this:

  GC data file: d:/perfdata/xp/perf_run.uncached.gc.txt

  collections            :        40
  garbage total          :   8188429
  gc time total (sec)    :         4.03
  garbage per request    :      2047.11
  requests per collection:       100.00

                       mean  stddev%          min          max
  gc time(ms):       101.38     10.0        93.00       125.00
  heap slots :    400000.00      0.0    400000.00    400000.00
  live       :    192914.31      0.2    191787.00    193197.00
  freed      :    207085.69      0.2    206803.00    208213.00
  freelist   :         0.00      0.0         0.00         0.00

  Note that these numbers, especially requests per collection, are
  only an approximation. This is due to the fact that perf_run_gc will
  add one final garbage collection call at the end of the run. Of
  course, higher number of iterations will produce more accurate
  data. Also, if the benchmark lists several uris, garbage per request
  will not give you meaningful information.


CONFIGURATION

  Benchmarks are configured through file benchmarks.yml. Example:

    default:
      index, query, alpha

    index:
      uri: /test/index
      new_session: true

    query:
        uri: /test/list
        method: post
        post_data: search_string=tomatoes

    alpha:
        uri: /test/alphabetic
        query_string: page=7&letter=A

  defines 4 benchmarks:

    "index"    will run (/test/index) using method GET
    "query"    will run (/test/list) using method POST
    "alpha"    will run (/test/alphabetic) using method GET
    "default"  will run benchmarks "index", "query" and "alpha"

  uri: is mandatory, query_params: and new_session: are optional.

  Instead of

      uri: /test/alphabetic
      query_string: page=7

  one could have written

      uri: /test/alphabetic?page=7&letter=A

  A single test session is created before running the benchmarks and
  stored in the sesion container of your choice. The corresponding
  _session_id value is sent with each request. If you specifiy
  new_session: true, railsbench will not send the session_id value, so
  Rails will create a new session per request for the given benchmark.

  Session data can either be set on the benchmarker instance, or
  specified in the benchmark config file like so:

    list_user_5:
      uri: /test/list
      method: post
      post_data: search_string=tomatoes
      session_data:
        user_id: 5

  benchmarks.yml is loaded using ERB. This makes it possible to avoid
  using primary keys in the config file:

    list_user_stefan:
      uri: /test/list
      method: post
      post_data: search_string=tomatoes
      session_data:
        user_id: <%= User.find_by_login('stefan').id %>

  An inital benchmark configuration file can be generated using command
  generate_benchmarks.

     generate_benchmarks -exclude_controllers=application,admin \
        -exclude_actions=edit,delete,destroy

  will generate a benchmark configuration file containing definitions
  for the following benchmarks:

    a) an entry for each named route
    b) an entry for each route generated by unnamed route definitions
    c) an entry for each controller, combining all actions (named xyz_controller)
    d) an entry combining all controllers (all_controllers), combining all
       benchmarks generated by step c.

  After generating the benchmark configuration file you will need to
  edit the benchmarks and change the placeholders in urls to real
  values; i.e., change something like /blog/edit/:id to /blog/edit/235.

  generate_benchmarks can be used on an existing configuration file. It
  will keep exisiting entries, which will be moved to the end of the
  config file. Thus it can be used to quickly update the config file
  after you've added/deleted or renamed controllers and/or actions.