Statemap
This repository contains the software for rendering statemaps, a software visualization in which time is on the X axis and timelines for discrete entities are stacked on the Y axis, with different states for the discrete entities rendered in different colors.
Generating a statemap consists of two steps: instrumentation and rendering. The result is a SVG that can be visualized with a SVG viewer (e.g., a web browser), allowing interaction.
Installation
To compile the command to render a statemap from instrumentation data:
cargo build --release
Note that statemap requires Rust.
Instrumentation
Statemaps themselves are methodology- and OS-agnostic, but instrumentation
is usually more system-specific.
The contrib
directory contains instrumentation for specific methodologies
and systems that will generate data that can be used
as input to the statemap
command:
Name | Method | OS | Statemap description |
---|---|---|---|
cpu-statemap.d | DTrace | SmartOS | CPU activity by CPU |
cpu-statemap-tagged.d | DTrace | SmartOS | CPU activity by CPU, tagged by origin of activity |
io-statemap.d | DTrace | SmartOS | SCSI devices in terms of number of outstanding I/O operations |
lx-cmd-statemap.d | DTrace | SmartOS | Processes and threads of a specified command in an LX zone |
lx-statemap.d | DTrace | SmartOS | Threads in a specified process in an LX zone |
postgres-statemap.d | DTrace | SmartOS | PostgreSQL processes |
postgres-zfs-statemap.d | DTrace | SmartOS | PostgreSQL processes, with ZFS-specific states |
spa-sync-statemap.d | DTrace | SmartOS | ZFS SPA sync thread state |
vdev-statemap.d | DTrace | SmartOS | I/O activity by ZFS vdev |
Data format
To generate data for statemap generation, instrumentation should create a file that consists of a stream of concatenated JSON. The expectation is that one JSON payload will consist of metadata, with many JSON payloads containing data, but the metadata may be split across multiple JSON payloads. (No field can appear more than once, however.)
Metadata
The following metadata fields are required:
-
start
: A two-element array of integers consisting of the start time of the data in seconds (the 0th element) and nanoseconds within the second (the 1st element). The start time should be expressed in UTC. -
states
: An object in which each member is the name of a valid entity state. Each member object can contain the following :-
value
: The value by which this state will be referred to in the data stream. -
color
: The color that should be used to render the state. If the color is not specified, a color will be selected at random.
For example, here is a valid
states
object:"states": { "on-cpu": {"value": 0, "color": "#DAF7A6" }, "off-cpu-waiting": {"value": 1, "color": "#f9f9f9" }, "off-cpu-futex": {"value": 2, "color": "#f0f0f0" }, "off-cpu-io": {"value": 3, "color": "#FFC300" }, "off-cpu-blocked": {"value": 4, "color": "#C70039" }, "off-cpu-dead": {"value": 5, "color": "#581845" } }
-
In addition, the metadata can contain the following optional fields are optional:
-
title
: The title of the statemap, such that it can meaningfully be in the clause "statemap oftitle
activity." -
host
: The host on which the data was gathered.
Data
The data for a statemap is provided following the metadata as concatenated JSON (that is, each JSON payload is a datum). Each datum is a JSON object that must contain the following members:
-
entity
: The name of the entity. -
time
: The time of the datum, expressed as a nanosecond offset from thestart
member present in the metadata. -
state
: The value of the state that begins at the time of the datum.
Each datum may also contain an additional member:
tag
: The tag for the state. See State tagging, below.
State tagging
It is often helpful to examine additional dimensionality within a particular
state or states. For example, in understanding CPU activity, it may be
helpful to understand not just that a CPU was in a state in which it was
executing a user thread, but the nature of the thread itself: the thread
identifier, process identifier, process name, and so on. To facilitate this,
statemaps support state tagging whereby an immutable tag is associated with a
particular transition to a particular state. There can be an arbitrary
number of such tags, but the expectation is that there are many more state
transitions than there are tags. Tags are indicated by the tag
member of
the state datum payload. Elsewhere in the stream of data (though not
necessarily before the tag is used), the tag should be defined with
a tag-defining JSON payload that contains the following two members:
-
tag
: A string that is the tag that is being defined. -
state
: The state that corresponds to this tag. Eachstate
/tag
tuple must have its own tag definition.
Beyond these two members, the tag definition can have any number of scalar
members. Tags are immutable; if a tag is redefined, the last tag definition
will apply to all uses of that tag. The tag should not contain member
definitions that would cause it to be ambiguous with respect to data (namely,
entity
and time
members).
As an example, here is a tag definition for a state that is associated with interrupt activity that indicates the source device:
{ "state": 6, "tag": "ffffd0c4f8f52000", "driver": "mpt_sas", "instance": 1 }
And here is an example of a tagged state datum:
{ "time": "1579579142", "entity": "55", "state": 6, "tag": "ffffd0c4f8f52000" }
This would indicate that at time 1579579142, entity 55 went into state 6 --
and the tag for this state (in this case, the interrupting device) was
instance 1 of the mpt_sas
driver.
Rendering
To render a statemap, run the statemap
command, providing an instrumentation
data file. The resulting statemap will be written as a SVG on standard
output:
statemap my-instrumentation-output.out > statemap.svg
Statemaps are interactive; the resulting SVG will contain controls that enable it to be zoomed, panned, states selected, etc. (See Interaction, below.)
By default, statemaps consist of all states for the entire time duration
represented in the input data. Because there can be many, many states
represented in the input, states will (by default) be coalesced when
the time spent in a state is deemed a sufficiently small fraction of
the overall time. For a coalesced state, the statemap will track the
overall fraction of states present (and will use a color that represents
a proportional blend of those states' colors). When a statemap contains
coalesced states, some information will be lost (namely, the exact
time delineations of state transitions within the coalesced state).
Coalesced states can be eliminated in one of two ways: either the
state coalescence target can be increased via the -c
option, or the
statemap can be regenerated to cover a smaller range of time with some
combination of the -b
option (to denote a beginning time) and the -d
option (to denote a duration). The number of coalesced states can be
determined by looking at the metadata placed at the end of of the output
SVG.
Options
The statemap
command has the following options:
-
-b
(--begin
): Takes a time offset at which the statemap should begin. The time offset may be expressed in floating point with an optional suffix (e.g.,-b 12.719s
). -
-c
(--coalesce
): Specifies the coalescing factor. Higher numbers will result in less coalescence. -
-d
(--duration
): Takes a duration time for the statemap. The time may be expressed in floating point with an optional suffix (e.g.,-d 491.2ms
). -
-h
(--state-height
): The height (in pixels) of each state in the statemap. -
-i
(--ignore-tags
): Ignore tags in the input, acting as if each state is untagged. (This will result in shorter run-time and a smaller resulting SVG.) -
-s
(--sortby
): The state by which to sort (default is to sort by entity). -
-S
(--stacksortby
): The state by which to sort statemaps, when multiple like statemaps are stacked (default is for the statemaps to be in the order specified).
Interaction
A statemap has icons for zooming and panning. As the statemap is zoomed, the time labels on top of the X axis will be updatd to reflect the current duration.
Clicking on a statemap will highlight both the time at the point of the click as well as the state. Zooming when a time is selected will center the zoomed statemap at the specified time. To clear the time, click on the time label above the statemap; to select another time, simply click on the statemap.
Shift-clicking (or Option-/Alt-clicking) on a statemap when a time is highlighted will highlight a dotted line at the time selected, as well as an indication of the time between the initial time highlighted and the time selected. This allows for the time delta between two events to be easily ascertained.
Stacked statemaps
To render a single SVG that contains multiple statemaps, multiple data files can be provided:
statemap data-1.out data-2.out > statemap.svg
The resulting statemaps will be stacked in the order of the data files as
provided on the command line, with the first data file dictating the time
bounds of the resulting stack. The statemaps can be similar statemaps from
dissimilar entities (e.g., different machines), or they can be dissimilar
statemaps (e.g., different statemaps), or any mix of these. Legends for
similar statemaps will be shared. The -S
option can control the sorting
of stacked like statemaps.
When statemaps are stacked, the coalescing factor applies to each statemap rather than to the entire stack of statemaps. When stacking many statemaps, low coalescing factors will be needed to prevent the resulting SVG from becoming excessively large.