rd The Record & Debug Tool

The Record & Debug Tool (rd) is a Rust language port of the rr-debugger/rr debugger.

With rd you can record Linux program executions. Subsequently you can replay these executions back exactly and debug them in the gdb front-end. If you know how to use rr then you already know how to use rd.

Why is it a good idea to port rr to Rust? See below.

Current Status : The port is substantially complete and is currently of an alpha level quality. You should be able to use rd for the tasks you would ordinarily use rr for. The rr project keeps accumulating features and fixes and many of them have not found their way into rd yet. However, the expectation is that rd should be reasonably robust, complete and usable now. Please report any issues!

Credits

The rd project would not have been possible without the work done in the rr-debugger/rr project. Many human-years of development effort have gone into making rr the truly amazing piece of software that it is.

The rd project is grateful to all the contributors of the rr-debugger/rr project.

For more details see CREDITS.md

Background

rd works on the same principles as rr. Please see rr-debugger/rr where you will find further information. More specifically, an excellent technical overview of rr can be found at arXiv:1705.05937.

Contributions

Contributions to the Record and Debug Tool (rd) are welcome and encouraged!

By contributing to rd you agree to license your contributions under the MIT License without any additional terms or conditions.

Please see LICENSE

Requirements

As rd is a port of rr, core requirements are essentially the same:

• Intel processors with Nehalem (2010) or later microarchitecture
• Linux >= 3.11
• sysctl kernel.perf_event_paranoid should be <= 1
• Linux should have access to CPU performance counters:
  • If you are running on bare-metal then you should not have to worry about this
  • If you wish to run rd in KVM/VMWare Workstation/VMWare Fusion etc. consult this rr resource

However, because rd is not as complete and mature as rr there are some additional restrictions:

• AMD Architectures are currently not supported on rd. Please use rr if you need to use AMD.
• rr has experimental Aarch64 support. This support has not yet been ported to rd.
• Only 64-bit Linux distributions are currently supported (Hopefully this should not be a big issue because both 32-bit and 64-bit programs are supported on 64-bit distributions)
  • 32-bit Linux distributions are currently not supported

rd has unfortunately not received testing in a variety of Linux distributions apart from Ubuntu 20.04. In general if rr works on your current setup, rd should work there too -- except for the additional restrictions outlined above. Please report any problems!

Building

rd requires a nightly version of the rust x86_64-unknown-linux-gnu toolchain to compile.

$ git clone [email protected]:sidkshatriya/rd.git
$ cd rd
$ cargo build --release 

Alternatively, build in debug mode . Things will run much more slowly as the code will be compiled with lower compiler optimizations, extra debug-mode assertions etc.

# Defaults to debug mode by default
$ cargo build 

In general, use release mode as the debug mode can be much slower. Run rd in debug mode if you run into issues or are working on developing rd.

The program has been tested to compile and run properly on a 64-bit Ubuntu 20.04 installation at the moment only.

Please file a ticket if rd does not work properly for your specific Linux distribution. In general, if rr compiles and runs properly in your Linux distro, rd should do the same.

Before trying to install or run rd make sure:

$ sudo apt install cmake make capnproto libcapnp-dev gdb g++-multilib libclang-11-dev

Running rd via cargo

Invoking cargo run without any command line parameters will give you help.

$ cd rd
# This command will provide general help on rd
# To run debug mode simply omit `--release`
$ cargo run --release

To get help on specific rd sub-command e.g. record

$ cargo run --release -- record --help

Here is a simple way to record and replay (the replay is non-interactive) ls -l.

# Note that we add another '--' in case we are passing any command line params to rd 
$ cargo run --release -- record ls -- -l
$ cargo run --release -- replay -a

Installing rd

It can get pretty tiresome to keep running rd via cargo. A simple script install.sh has been provided to install the rd binary and related support files to your directory of choice.

$ cd rd
# builds release mode and installs to ~/myrd
$ PREFIX=~/myrd ./install.sh

This builds rd in release mode and then installs rd at $HOME/myrd. Files will be stored at ~/myrd/bin, ~/myrd/lib and ~/myrd/share. The install script is extremely simple and can be easily understood. The rd executable will be in $HOME/myrd/bin. You may also want to add ~/myrd/bin to your PATH at this point.

Assuming that ~/myrd/binis in your PATH it is very easy to invoke rd now.

# Records ls -l invocation
$ rd record ls -- -l

# Non-interatively replays the ls -l recording
$ rd replay -a

# Interatively replays the ls -l recording
$ rd replay

Please note that the install.sh script is rudimentary. As always, check (and modify as necessary) the unix permissions/ownership of the files and directories created, especially, if you install to a location like ~/.local /usr/local etc. In general it is recommended to install rd to a separate directory in $HOME e.g. ~/myrd, ~/rd etc. as that interferes the least with your existing system.

Installing a debug build

If you want to build and install a debug build (warning: a debug build is going to be much slower than the release build), then:

$ cd rd
# builds debug mode and installs to ~/myrd
$ PREFIX=~/myrd DEBUG=1 ./install.sh

What works

The port is substantially complete and is ready for end-user usage and developer contributions. The project contains 55,000+ lines of ported over Rust code.

The following work:

• rd rerun
• rd replay
  • Both interactive replay (which uses the gdb front-end) and non-interative replay (-a flag) are supported
• rd record
• rd buildid
• rd dump
• rd traceinfo

A 64-bit build of rd supports the record/replay of both 32 & 64-bit Linux programs. However, building and running rd to record/replay 32-bit programs in a 32-bit Linux distribution is currently not supported.

Tips and Suggestions

Recording & Debugging program runs

To record a program

$ rd record <program to be recorded>

If you want to pass arguments to the program:

$ rd record <program to be recorded> -- <arguments>

Example:

$ rd record ls -- -l

To replay and debug the last recorded program in the gdb front-end

$ rd replay

Logging

The various logging levels are debug, info, warn, info and fatal. To log at warn by default and debug for all messages from the auto_remote_syscalls rust module (as an example) do:

$ RD_LOG=all:warn,auto_remote_syscalls:debug rd /* rd params here */

Why implement in Rust?

Here are some (necessarily subjective) reasons why it might be a good idea to have a Rust port of rr.

Reduce complexity, increase reliability

rr is written in C/C++. C/C++ is a complex beast. As the Linux userspace/kernel interface gets more and more complex and gathers even more quirks, the rr codebase gets more and more complex too. With Rust the hope is that we have a clean and modern language that allows us to manage the complexity of record/replay. Of course, we still need to deal with the inherent complexity of record/replay but Rust helps with writing robust and reliable code. Once the code is written it is also easier to do refactorings with more confidence in Rust.

Reduce barriers to understanding and contribution

Once you understand the core principles, Rust can be easier than C/C++ to grok. Hopefully this will allow more people to inspect, improve and offer fixes to the rd codebase.

Provide an alternative, compatible implementation

Just like there can be multiple compilers for the same language, it might be a good idea to have multiple compatible implementations of rr. This might help with debugging weird bugs and optimize various implementations around different parameters.

Be a playground for experimental features

This is something for the future. The hope is rd can become a playground for experimentation and implement some innovative features in record/replay. Also rd has access to the awesome Rust cargo ecosystem which means that functionality already implemented elsewhere can be added to it much more easily.