ruby_memcheck

This gem provides a sane way to use Valgrind's memcheck on your native extension gem.

Table of contents

1. What is this gem?
   1. Who should use this gem?
   2. How does it work?
   3. Limitations
2. Installation
3. Setup
4. Configuration
5. Suppression files
6. License

What is this gem?

Valgrind's memcheck is a great tool to find and debug memory issues (e.g. memory leak, use-after-free, etc.). However, it doesn't work well on Ruby because Ruby does not free all of the memory it allocates during shutdown. This results in Valgrind reporting thousands (or more) false positives, making it very difficult for Valgrind to actually be useful. This gem solves the problem by using heuristics to filter out false positives.

Who should use this gem?

Only gems with native extensions can use this gem. If your gem is written in plain Ruby, this gem is not useful for you.

How does it work?

This gem runs Valgrind with the --xml option to generate an XML of all the errors. It will then parse the XML and use various heuristics based on the type of the error and the stack trace to filter out errors that are false positives.

For more details, read this blog post.

Limitations

Because of the aggressive heuristics used to filter out false positives, there are various limitations of what this gem can detect.

1. This gem is only expected to work on Linux.

2. This gem runs your gem's test suite to find errors and memory leaks. It will only be able to report errors on code paths that are covered by your tests. So make sure your test suite has good coverage!

3. It will not find memory leaks in Ruby. It filters out everything in Ruby.

4. It will not find memory leaks of allocations that occurred in Ruby (even if the memory leak is caused by your native extension).

   An example of this is if a string is allocated in Ruby, passed into your native extension, you change the pointer of the string without freeing the contents, so the contents of the string becomes leaked.

5. To filter out false positives, it will only find definite leaks (i.e. memory regions with no pointers to it). It will not find possible leaks (i.e. memory regions with pointers to it).

6. It will not find leaks that occur in the Init function of your native extension.

7. It will not find uses of undefined values (e.g. conditional jumps depending on undefined values). This is just a technical limitation that has not been solved yet (contributions welcome!).

Installation

gem install ruby_memcheck

Setup

Note If you encounter errors from Valgrind that looks like this:

### unhandled dwarf2 abbrev form code 0x25

Then you need a newer version of Valgrind (>= 3.20.0) with DWARF5 support. The current versions of Valgrind in Ubuntu Packages is not new enough.

You can install Valgrind from source using the following commands:

sudo apt-get install -y libc6-dbg
wget https://sourceware.org/pub/valgrind/valgrind-3.20.0.tar.bz2
tar xvf valgrind-3.20.0.tar.bz2
cd valgrind-3.20.0
./configure
make
sudo make install

The easiest way to use this gem is to use it on your test suite (minitest or RSpec) using rake.

0. Install Valgrind.

1. In your Rakefile, require this gem.

   require "ruby_memcheck"

   • For RSpec: If you're using RSpec, also add the following require.

     require "ruby_memcheck/rspec/rake_task"

2. Configure the gem by calling RubyMemcheck.config. You must pass it your binary name. This is the same value you passed into create_makefile in your extconf.rb file. Make sure this value is correct or it will filter out almost everything as a false-positive!

   RubyMemcheck.config(binary_name: "your_binary_name")

3. Setup the test task for your test framework.

   • minitest

     Locate your test task(s) in your Rakefile. You can identify it with a call to Rake::TestTask.new.

     Create a namespace under the test task and create a RubyMemcheck::TestTask with the same configuration.

     For example, if your Rakefile looked like this before:

     Rake::TestTask.new(test: :compile) do |t|
       t.libs << "test"
       t.test_files = FileList["test/unit/**/*_test.rb"]
     end

     You can change it to look like this:

     test_config = lambda do |t|
       t.libs << "test"
       t.test_files = FileList["test/**/*_test.rb"]
     end
     Rake::TestTask.new(test: :compile, &test_config)
     namespace :test do
       RubyMemcheck::TestTask.new(valgrind: :compile, &test_config)
     end

   • RSpec

     Locate your rake task(s) in your Rakefile. You can identify it with a call to RSpec::Core::RakeTask.new.

     Create a namespace under the test task and create a RubyMemcheck::RSpec::RakeTask with the same configuration.

     For example, if your Rakefile looked like this before:

     RSpec::Core::RakeTask.new(spec: :compile)

     You can change it to look like this:

     RSpec::Core::RakeTask.new(spec: :compile)
     namespace :spec do
       RubyMemcheck::RSpec::RakeTask.new(valgrind: :compile)
     end

4. You're ready to run your test suite with Valgrind using rake test:valgrind or rake spec:valgrind! Note that this will take a while to run because Valgrind will make Ruby significantly slower.

5. (Optional) If you find false positives in the output, you can create Valgrind suppression files. See the Suppression files section for more details.

Configuration

When you run RubyMemcheck.config, you are creating a default RubyMemcheck::Configuration. By default, the Rake tasks for minitest and RSpec will use this configuration. You can also manually pass in a Configuration object as the first argument to the constructor of RubyMemcheck::TestTask or RubyMemcheck::RSpec::RakeTask to use a different Configuration object rather than the default one.

RubyMemcheck::Configuration accepts a variety of keyword arguments. Here are all the arguments:

• binary_name: Required. The binary name of your native extension gem. This is the same value you passed into create_makefile in your extconf.rb file.
• ruby: Optional. The command to run to invoke Ruby. Defaults to the Ruby that is currently being used.
• valgrind: Optional. The command to run to invoke Valgrind. Defaults to the string "valgrind".
• valgrind_options: Optional. Array of options to pass into Valgrind. This is only present as an escape hatch, so avoid using it. This may be deprecated or removed in future versions.
• valgrind_suppressions_dir: Optional. The string path of the directory that stores suppression files for Valgrind. See the Suppression files section for more details. Defaults to suppressions.
• valgrind_generate_suppressions: Optional. Whether suppressions should also be outputted along with the errors. the Suppression files section for more details. Defaults to false.
• skipped_ruby_functions: Optional. Ruby functions that are ignored because they are considered a call back into Ruby. This is only present as an escape hatch, so avoid using it. If you find another Ruby function that is a false positive because it calls back into Ruby, please send a patch into this repo. Otherwise, use a Valgrind suppression file.
• valgrind_xml_dir: Optional. The directory to store temporary XML files for Valgrind. It defaults to a temporary directory. This is present for development debugging, so you shouldn't have to use it.
• output_io: Optional. The IO object to output Valgrind errors to. Defaults to standard error.
• filter_all_errors: Optional. Whether to filter all kinds of Valgrind errors (not just memory leaks). This feature should only be used if you're encountering a large number of illegal memory accesses coming from Ruby. If you need to use this feature, you may have found a bug inside of Ruby. Consider reporting it to the Ruby bug tracker. Defaults to false.

Suppression files

If you find false positives in the output, you can create suppression files in a suppressions directory in the root directory of your gem. In this directory, you can create Valgrind suppression files.

The most basic suppression file is ruby.supp. If you want some suppressions for only specific versions of Ruby, you can add the Ruby version to the filename. For example, ruby-3.supp will suppress for any Rubies with a major version of 3 (e.g. 3.0.0, 3.1.1, etc.), while suppression file ruby-3.1.supp will only be used for Ruby with a major and minor version of 3.1 (e.g. 3.1.0, 3.1.1, etc.).

Success stories

Let's celebrate wins from this gem! If this gem was useful for you, please share your story below too!

• liquid-c:
  • Found 2 memory leaks: #157, #161
  • Running on CI: #162
• nokogiri:
  • Found 5 memory leaks: 4 in #2345, #2347
  • Running on CI: #2344
• rotoscope:
  • Found a memory leak in Ruby TracePoint
  • Running on CI: #89
• protobuf:
  • Found 1 memory leak: #9150
• gRPC:
  • Found 1 memory leak: #27900
• wasmtime-rb:
  • Found 1 memory leak: #26
• yarp:
  • Found 6 memory leaks and 1 memory error: #292, #292
  • Running on CI: #293
• libxml2:
  • Found 1 memory leak: memory leak from `xmlSchemaValidateStream` in v2.11.x (#530)
  • Running in Nokogiri's CI pipeline: #2868

License

The gem is available as open source under the terms of the MIT License.