• Stars
    star
    182
  • Rank 211,158 (Top 5 %)
  • Language
  • License
    GNU General Publi...
  • Created almost 10 years ago
  • Updated almost 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

SystemTap Lightweight Tools

SystemTap Lightweight Tools (lwtools)

A miscellaneous collection of in-development and unsupported SystemTap tools and one-liners, which use a limited set of SystemTap capabilities (lightweight). These provide advanced observability for the analysis of system and cloud performance.

An aim of this collection is to explore what SystemTap can do using basic features and without the need for kernel debuginfo, which can be an onerous requirement. For a similar set of ftrace- and perf_events-based tools, see the perf tools collection, which is more developed. An advantage of these SystemTap tools is that they can perform custom in-kernel aggregations (eg, latency distributions) with low overhead.

Some expert assembly will likely be required with these SystemTap tools. Also see warnings.

Warnings

SystemTap is in-development kernel-level software, so there is the possibility of kernel panics or freezes. I've developed these for use in a lab environment, where panics can be tolerated.

Also, tools that use the inherently unstable dynamic tracing interface may either not work at all (print errors), or work and show bogus data, or work and show plausible but still incorrect data. The same is true for any dynamic tracing-based scripts (including my DTrace ones).

Note that the panics may be due to SystemTap or other kernel origins. Use crash dump analysis to determine the real cause, and please report bugs.

Prerequisites

SystemTap, for the stap command, and an environment that is fault tolerant (see Warnings). It is strongly recommended that you try the latest version of SystemTap, which probably means compiling from source. See git://sourceware.org/git/systemtap.git, and the build readme.

Where possible, these tools avoid the requirement of having kernel debuginfo.

Contents

There are also man pages under man/man8.

Screenshots

Histogram of syscall writes by requested size (one-liner):

# stap -e 'global a; probe nd_syscall.write { a <<< int_arg(3); } probe end { print(@hist_log(a)); }'
^Cvalue |-------------------------------------------------- count
    0 |                                                    0
    1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                35
    2 |@                                                   1
    4 |@                                                   1
    8 |                                                    0
   16 |@@@@@@@@@@@@@@@@@@@@@@                             22
   32 |@@@@@@@@@@@@@@@                                    15
   64 |@@@@@@@@@@@@@@@@@                                  17
  128 |@@                                                  2
  256 |@@                                                  2
  512 |                                                    0
 1024 |                                                    0

Tracing exec() syscalls to show new process execution:

# ./execsnoop-nd.stp
TIME                        UID   PPID    PID           COMM ARGS
Sat Jan 31 20:39:37 2015      0    878   5744           sshd /usr/sbin/sshd -R
Sat Jan 31 20:39:38 2015      0   5744   5746           bash -bash
Sat Jan 31 20:39:38 2015      0   5747   5748             id id -un
Sat Jan 31 20:39:38 2015      0   5749   5750       hostname /bin/hostname
Sat Jan 31 20:39:38 2015      0   5751   5752            tty tty -s
Sat Jan 31 20:39:38 2015      0   5751   5753           tput tput colors
Sat Jan 31 20:39:38 2015      0   5754   5755      dircolors dircolors --sh /etc/DIR_COLORS.256color
Sat Jan 31 20:39:38 2015      0   5746   5756           grep grep -qi ^COLOR.*none /etc/DIR_COLORS.256color
Sat Jan 31 20:39:40 2015      0   5746   5757             ls ls --color=auto
Sat Jan 31 20:39:40 2015      0   5746   5758           date date

Tracing open() syscalls with process and pathnames:

# ./opensnoop-nd.stp
   UID    PID             COMM   FD PATH
     0  13858           iostat    3 /etc/ld.so.cache
     0  13858           iostat    3 /lib64/libc.so.6
     0  13858           iostat    3 /usr/lib/locale/locale-archive
     0  13858           iostat    3 /sys/devices/system/cpu
     0  13858           iostat    3 /proc/diskstats
     0  13858           iostat    3 /etc/localtime
     0  13858           iostat    3 /proc/uptime
     0  13858           iostat    3 /proc/stat
     0  13858           iostat    3 /proc/diskstats
     0  13858           iostat    4 /etc/sysconfig/sysstat.ioconf
     0  13858           iostat    3 /proc/uptime
     0  13858           iostat    3 /proc/stat
     0  13858           iostat    3 /proc/diskstats
[...]

Summarizing read() and write() syscall latency for processes named "node":

# ./rwtime-nd.stp node
Tracing read/write syscalls for processes named "node"... Hit Ctrl-C to end.
^C
syscall read latency (ns):
value |-------------------------------------------------- count
  512 |                                                     0
 1024 |                                                     0
 2048 |                                                     2
 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@     278
 8192 |                                                     1
16384 |                                                     5
32768 |                                                     0
65536 |                                                     0

syscall write latency (ns):
 value |-------------------------------------------------- count
  4096 |                                                     0
  8192 |                                                     0
 16384 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@      135
 32768 |@@                                                   8
 65536 |                                                     0
131072 |                                                     0

Internals

These tools are designed to be short, simple, and documented, and use as few SystemTap capabilities as possible. They also avoid using kernel debuginfo, so that these scripts can be used in environments that lack it (eg, cloud instances that are created and destroyed quickly, where including debuginfo can be onerous). This can complicate or obfuscate some scripts.

Each tool has an examples file under examples, a man page under man/man8, a symlink in bin, and a link in this README.

See Also

For SystemTap tools that use its full capabilities, see the SystemTap examples page, particularly the "best examples" section, as well as the examples in the SystemTap source under share/doc/systemtap/examples. Most of these are likely to require kernel debuginfo.

More Repositories

1

FlameGraph

Stack trace visualizer
Perl
16,893
star
2

perf-tools

Performance analysis tools based on Linux perf_events (aka perf) and ftrace
Shell
9,533
star
3

bpf-perf-tools-book

Official repository for the BPF Performance Tools book
Python
1,621
star
4

HeatMap

Heat map generation tools
Perl
305
star
5

pmc-cloud-tools

PMC (Performance Monitoring Counter) tools for the cloud
Shell
233
star
6

Chaosreader

An any-snarf program that processes application protocols (HTTP/FTP/...) from tcpdump or snoop files and stores session and file data
216
star
7

dtrace-cloud-tools

Some DTrace tools written for the SmartOS/SmartDataCenter cloud (illumos-based)
D
202
star
8

wss

Working Set Size tools
C
200
star
9

bpf-perf-workshop

C
179
star
10

msr-cloud-tools

MSR Cloud Tools
Shell
173
star
11

DTrace-book-scripts

Scripts from "DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD", by Brendan Gregg and Jim Mauro, Prentice Hall, 2011.
167
star
12

BPF-tools

Performance Tools using Linux eBPF
C
118
star
13

perf-labs

Performance analysis labs
C
89
star
14

DTrace-tools

DTrace tools for FreeBSD
DTrace
69
star
15

PerfModels

Performance Scalability Models
R
67
star
16

Misc

Misc
Shell
65
star
17

Dump2PNG

Visualize file data as a PNG
C
40
star
18

GuessingGame

Guessing game written in many programming languages
Batchfile
33
star
19

proc-profiler

Linux /proc/PID/stack profiler
Perl
32
star
20

bpf-typewriter

BPF noisy typewriter (bpftrace)
23
star
21

p1bench

Perturbation Benchmark
C
20
star
22

DTraceToolkit

A collection of useful, tested and documented DTrace scripts
16
star
23

FlameScope

coming soon
Perl
13
star
24

skid-testing

Processor PMC sample skid testing
C
7
star
25

wrmsrbench

WRMSR micro benchmark
C
7
star
26

Test

testing github
JavaScript
3
star
27

brendangregg.github.io

HTML
3
star