• Stars
    star
    9,533
  • Rank 3,732 (Top 0.08 %)
  • Language
    Shell
  • License
    GNU General Publi...
  • Created over 10 years ago
  • Updated 12 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Performance analysis tools based on Linux perf_events (aka perf) and ftrace

perf-tools

A miscellaneous collection of in-development and unsupported performance analysis tools for Linux ftrace and perf_events (aka the "perf" command). Both ftrace and perf are core Linux tracing tools, included in the kernel source. Your system probably has ftrace already, and perf is often just a package add (see Prerequisites).

These tools are designed to be easy to install (fewest dependencies), provide advanced performance observability, and be simple to use: do one thing and do it well. This collection was created by Brendan Gregg (author of the DTraceToolkit).

Many of these tools employ workarounds so that functionality is possible on existing Linux kernels. Because of this, many tools have caveats (see man pages), and their implementation should be considered a placeholder until future kernel features, or new tracing subsystems, are added.

These are intended for Linux 3.2 and newer kernels. For Linux 2.6.x, see Warnings.

Presentation

These tools were introduced in the USENIX LISA 2014 presentation: Linux Performance Analysis: New Tools and Old Secrets

Contents

Using ftrace:

Using perf_events:

Using eBPF:

  • As a preview of things to come, see the bcc tracing Tools section. These use bcc, a front end for using eBPF. bcc+eBPF will allow some of these tools to be rewritten and improved, and additional tools to be created.

Screenshots

Showing new processes and arguments:

# ./execsnoop 
Tracing exec()s. Ctrl-C to end.
   PID   PPID ARGS
 22898  22004 man ls
 22905  22898 preconv -e UTF-8
 22908  22898 pager -s
 22907  22898 nroff -mandoc -rLL=164n -rLT=164n -Tutf8
 22906  22898 tbl
 22911  22910 locale charmap
 22912  22907 groff -mtty-char -Tutf8 -mandoc -rLL=164n -rLT=164n
 22913  22912 troff -mtty-char -mandoc -rLL=164n -rLT=164n -Tutf8
 22914  22912 grotty

Measuring block device I/O latency from queue insert to completion:

# ./iolatency -Q
Tracing block I/O. Output every 1 seconds. Ctrl-C to end.

  >=(ms) .. <(ms)   : I/O      |Distribution                          |
       0 -> 1       : 1913     |######################################|
       1 -> 2       : 438      |#########                             |
       2 -> 4       : 100      |##                                    |
       4 -> 8       : 145      |###                                   |
       8 -> 16      : 43       |#                                     |
      16 -> 32      : 43       |#                                     |
      32 -> 64      : 1        |#                                     |

[...]

Tracing the block:block_rq_insert tracepoint, with kernel stack traces, and only for reads:

# ./tpoint -s block:block_rq_insert 'rwbs ~ "*R*"'
   cksum-11908 [000] d... 7269839.919098: block_rq_insert: 202,1 R 0 () 736560 + 136 [cksum]
   cksum-11908 [000] d... 7269839.919107: 
 => __elv_add_request
 => blk_flush_plug_list
 => blk_finish_plug
 => __do_page_cache_readahead
 => ondemand_readahead
 => page_cache_async_readahead
 => generic_file_read_iter
 => new_sync_read
 => vfs_read
 => SyS_read
 => system_call_fastpath

[...]

Count kernel function calls beginning with "bio_", summarize every second:

# ./funccount -i 1 'bio_*'
Tracing "bio_*"... Ctrl-C to end.

FUNC                              COUNT
bio_attempt_back_merge               26
bio_get_nr_vecs                     361
bio_alloc                           536
bio_alloc_bioset                    536
bio_endio                           536
bio_free                            536
bio_fs_destructor                   536
bio_init                            536
bio_integrity_enabled               536
bio_put                             729
bio_add_page                       1004

[...]

There are many more examples in the examples directory. Also see the man pages.

Prerequisites

The intent is as few as possible. Eg, a Linux 3.2 server without debuginfo. See the tool man page for specifics.

ftrace

FTRACE configured in the kernel. You may already have this configured and available in your kernel version, as FTRACE was first added in 2.6.27. This requires CONFIG_FTRACE and other FTRACE options depending on the tool. Some tools (eg, funccount) require CONFIG_FUNCTION_PROFILER.

perf_events

Requires the "perf" command to be installed. This is in the linux-tools-common package. After installing that, perf may tell you to install an additional linux-tools package (linux-tools-kernel_version). perf can also be built under tools/perf in the kernel source. See perf_events Prerequisites for more details about getting perf_events to work fully.

debugfs

Requires a kernel with CONFIG_DEBUG_FS option enabled. As with FTRACE, this may already be enabled (debugfs was added in 2.6.10-rc3). The debugfs also needs to be mounted:

# mount -t debugfs none /sys/kernel/debug

awk

Many of there scripts use awk, and will try to use either mawk or gawk depending on the desired behavior: mawk for buffered output (because of its speed), and gawk for synchronous output (as fflush() works, allowing more efficient grouping of writes).

Install

These are just scripts. Either grab everything:

git clone --depth 1 https://github.com/brendangregg/perf-tools

Or use the raw links on github to download individual scripts. Eg:

wget https://raw.githubusercontent.com/brendangregg/perf-tools/master/iosnoop

This preserves tabs (which copy-n-paste can mess up).

Warnings

Ftrace was first added to Linux 2.6.27, and perf_events to Linux 2.6.31. These early versions had kernel bugs, and lockups and panics have been reported on 2.6.32 series kernels. This includes CentOS 6.x. If you must analyze older kernels, these tools may only be useful in a fault-tolerant environment, such as a lab with simulated issues. These tools have been primarily developed on Linux 3.2 and later kernels.

Depending on the tool, there may also be overhead incurred. See the next section.

Internals and Overhead

perf_events is evolving. This collection began development circa Linux 3.16, with Linux 3.2 servers as the main target, at a time when perf_events lacks certain programmatic capabilities (eg, custom in-kernel aggregations). It's possible these will be added in a forthcoming kernel release. Until then, many of these tools employ workarounds, tricks, and hacks in order to work. Some of these tools pass event data to user space for post-processing, which costs much higher overhead than in-kernel aggregations. The overhead of each tool is described in its man page.

WARNING: In extreme cases, your target application may run 5x slower when using these tools. Depending on the tool and kernel version, there may also be the risk of kernel panics. Read the program header for warnings, and test before use.

If the overhead is a problem, these tools can be improved. If a tool doesn't already, it could be rewritten in C to use perf_events_open() and mmap() for the trace buffer. It could also implement frequency counts in C, and operate on mmap() directly, rather than using awk/Perl/Python. Additional improvements are possible for ftrace-based tools, such as use of snapshots and per-instance buffers.

Some of these tools are intended as short-term workarounds until more kernel capabilities exist, at which point they can be substantially rewritten. Older versions of these tools will be kept in this repository, for older kernel versions.

As my main target is a fleet of Linux 3.2 servers that do not have debuginfo, these tools try not to require it. At times, this makes the tool more brittle than it needs to be, as I'm employing workarounds (that may be kernel version and platform specific) instead of using debuginfo information (which can be generic). See the man page for detailed prerequisites for each tool.

I've tried to use perf_events ("perf") where possible, since that interface has been developed for multi-user use. For various reasons I've often needed to use ftrace instead. ftrace is surprisingly powerful (thanks Steven Rostedt!), and not all of its features are exposed via perf, or in common usage. This tool collection is in some ways a demonstration of hidden Linux features using ftrace.

Since things are changing, it's very possible you may find some tools don't work on your Linux kernel version. Some expertise and assembly will be required to fix them.

Links

A case study and summary:

Related articles:

More Repositories

1

FlameGraph

Stack trace visualizer
Perl
16,893
star
2

bpf-perf-tools-book

Official repository for the BPF Performance Tools book
Python
1,621
star
3

HeatMap

Heat map generation tools
Perl
305
star
4

pmc-cloud-tools

PMC (Performance Monitoring Counter) tools for the cloud
Shell
233
star
5

Chaosreader

An any-snarf program that processes application protocols (HTTP/FTP/...) from tcpdump or snoop files and stores session and file data
216
star
6

dtrace-cloud-tools

Some DTrace tools written for the SmartOS/SmartDataCenter cloud (illumos-based)
D
202
star
7

wss

Working Set Size tools
C
200
star
8

systemtap-lwtools

SystemTap Lightweight Tools
182
star
9

bpf-perf-workshop

C
179
star
10

msr-cloud-tools

MSR Cloud Tools
Shell
173
star
11

DTrace-book-scripts

Scripts from "DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD", by Brendan Gregg and Jim Mauro, Prentice Hall, 2011.
167
star
12

BPF-tools

Performance Tools using Linux eBPF
C
118
star
13

perf-labs

Performance analysis labs
C
89
star
14

DTrace-tools

DTrace tools for FreeBSD
DTrace
69
star
15

PerfModels

Performance Scalability Models
R
67
star
16

Misc

Misc
Shell
65
star
17

bcc

BPF Compiler Collection
C
61
star
18

Dump2PNG

Visualize file data as a PNG
C
40
star
19

GuessingGame

Guessing game written in many programming languages
Batchfile
33
star
20

proc-profiler

Linux /proc/PID/stack profiler
Perl
32
star
21

bpf-typewriter

BPF noisy typewriter (bpftrace)
23
star
22

p1bench

Perturbation Benchmark
C
20
star
23

DTraceToolkit

A collection of useful, tested and documented DTrace scripts
16
star
24

FlameScope

coming soon
Perl
13
star
25

skid-testing

Processor PMC sample skid testing
C
7
star
26

wrmsrbench

WRMSR micro benchmark
C
7
star
27

hexyl

A command-line hex viewer
Rust
4
star
28

Test

testing github
JavaScript
3
star
29

brendangregg.github.io

HTML
3
star