• Stars
    star
    219
  • Rank 181,133 (Top 4 %)
  • Language
    Rust
  • License
    Mozilla Public Li...
  • Created almost 7 years ago
  • Updated over 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Software for rendering statemaps

Statemap

This repository contains the software for rendering statemaps, a software visualization in which time is on the X axis and timelines for discrete entities are stacked on the Y axis, with different states for the discrete entities rendered in different colors.

Generating a statemap consists of two steps: instrumentation and rendering. The result is a SVG that can be visualized with a SVG viewer (e.g., a web browser), allowing interaction.

Installation

To compile the command to render a statemap from instrumentation data:

cargo build --release

Note that statemap requires Rust.

Instrumentation

Statemaps themselves are methodology- and OS-agnostic, but instrumentation is usually more system-specific. The contrib directory contains instrumentation for specific methodologies and systems that will generate data that can be used as input to the statemap command:

Name Method OS Statemap description
cpu-statemap.d DTrace SmartOS CPU activity by CPU
cpu-statemap-tagged.d DTrace SmartOS CPU activity by CPU, tagged by origin of activity
io-statemap.d DTrace SmartOS SCSI devices in terms of number of outstanding I/O operations
lx-cmd-statemap.d DTrace SmartOS Processes and threads of a specified command in an LX zone
lx-statemap.d DTrace SmartOS Threads in a specified process in an LX zone
postgres-statemap.d DTrace SmartOS PostgreSQL processes
postgres-zfs-statemap.d DTrace SmartOS PostgreSQL processes, with ZFS-specific states
spa-sync-statemap.d DTrace SmartOS ZFS SPA sync thread state
vdev-statemap.d DTrace SmartOS I/O activity by ZFS vdev

Data format

To generate data for statemap generation, instrumentation should create a file that consists of a stream of concatenated JSON. The expectation is that one JSON payload will consist of metadata, with many JSON payloads containing data, but the metadata may be split across multiple JSON payloads. (No field can appear more than once, however.)

Metadata

The following metadata fields are required:

  • start: A two-element array of integers consisting of the start time of the data in seconds (the 0th element) and nanoseconds within the second (the 1st element). The start time should be expressed in UTC.

  • states: An object in which each member is the name of a valid entity state. Each member object can contain the following :

    • value: The value by which this state will be referred to in the data stream.

    • color: The color that should be used to render the state. If the color is not specified, a color will be selected at random.

    For example, here is a valid states object:

      "states": {
              "on-cpu": {"value": 0, "color": "#DAF7A6" },
              "off-cpu-waiting": {"value": 1, "color": "#f9f9f9" },
              "off-cpu-futex": {"value": 2, "color": "#f0f0f0" },
              "off-cpu-io": {"value": 3, "color": "#FFC300" },
              "off-cpu-blocked": {"value": 4, "color": "#C70039" },
              "off-cpu-dead": {"value": 5, "color": "#581845" }
      }
    

In addition, the metadata can contain the following optional fields are optional:

  • title: The title of the statemap, such that it can meaningfully be in the clause "statemap of title activity."

  • host: The host on which the data was gathered.

Data

The data for a statemap is provided following the metadata as concatenated JSON (that is, each JSON payload is a datum). Each datum is a JSON object that must contain the following members:

  • entity: The name of the entity.

  • time: The time of the datum, expressed as a nanosecond offset from the start member present in the metadata.

  • state: The value of the state that begins at the time of the datum.

Each datum may also contain an additional member:

  • tag: The tag for the state. See State tagging, below.

State tagging

It is often helpful to examine additional dimensionality within a particular state or states. For example, in understanding CPU activity, it may be helpful to understand not just that a CPU was in a state in which it was executing a user thread, but the nature of the thread itself: the thread identifier, process identifier, process name, and so on. To facilitate this, statemaps support state tagging whereby an immutable tag is associated with a particular transition to a particular state. There can be an arbitrary number of such tags, but the expectation is that there are many more state transitions than there are tags. Tags are indicated by the tag member of the state datum payload. Elsewhere in the stream of data (though not necessarily before the tag is used), the tag should be defined with a tag-defining JSON payload that contains the following two members:

  • tag: A string that is the tag that is being defined.

  • state: The state that corresponds to this tag. Each state/tag tuple must have its own tag definition.

Beyond these two members, the tag definition can have any number of scalar members. Tags are immutable; if a tag is redefined, the last tag definition will apply to all uses of that tag. The tag should not contain member definitions that would cause it to be ambiguous with respect to data (namely, entity and time members).

As an example, here is a tag definition for a state that is associated with interrupt activity that indicates the source device:

{ "state": 6, "tag": "ffffd0c4f8f52000", "driver": "mpt_sas", "instance": 1 }

And here is an example of a tagged state datum:

{ "time": "1579579142", "entity": "55", "state": 6, "tag": "ffffd0c4f8f52000" }

This would indicate that at time 1579579142, entity 55 went into state 6 -- and the tag for this state (in this case, the interrupting device) was instance 1 of the mpt_sas driver.

Rendering

To render a statemap, run the statemap command, providing an instrumentation data file. The resulting statemap will be written as a SVG on standard output:

statemap my-instrumentation-output.out > statemap.svg

Statemaps are interactive; the resulting SVG will contain controls that enable it to be zoomed, panned, states selected, etc. (See Interaction, below.)

By default, statemaps consist of all states for the entire time duration represented in the input data. Because there can be many, many states represented in the input, states will (by default) be coalesced when the time spent in a state is deemed a sufficiently small fraction of the overall time. For a coalesced state, the statemap will track the overall fraction of states present (and will use a color that represents a proportional blend of those states' colors). When a statemap contains coalesced states, some information will be lost (namely, the exact time delineations of state transitions within the coalesced state). Coalesced states can be eliminated in one of two ways: either the state coalescence target can be increased via the -c option, or the statemap can be regenerated to cover a smaller range of time with some combination of the -b option (to denote a beginning time) and the -d option (to denote a duration). The number of coalesced states can be determined by looking at the metadata placed at the end of of the output SVG.

Options

The statemap command has the following options:

  • -b (--begin): Takes a time offset at which the statemap should begin. The time offset may be expressed in floating point with an optional suffix (e.g., -b 12.719s).

  • -c (--coalesce): Specifies the coalescing factor. Higher numbers will result in less coalescence.

  • -d (--duration): Takes a duration time for the statemap. The time may be expressed in floating point with an optional suffix (e.g., -d 491.2ms).

  • -h (--state-height): The height (in pixels) of each state in the statemap.

  • -i (--ignore-tags): Ignore tags in the input, acting as if each state is untagged. (This will result in shorter run-time and a smaller resulting SVG.)

  • -s (--sortby): The state by which to sort (default is to sort by entity).

  • -S (--stacksortby): The state by which to sort statemaps, when multiple like statemaps are stacked (default is for the statemaps to be in the order specified).

Interaction

A statemap has icons for zooming and panning. As the statemap is zoomed, the time labels on top of the X axis will be updatd to reflect the current duration.

Clicking on a statemap will highlight both the time at the point of the click as well as the state. Zooming when a time is selected will center the zoomed statemap at the specified time. To clear the time, click on the time label above the statemap; to select another time, simply click on the statemap.

Shift-clicking (or Option-/Alt-clicking) on a statemap when a time is highlighted will highlight a dotted line at the time selected, as well as an indication of the time between the initial time highlighted and the time selected. This allows for the time delta between two events to be easily ascertained.

Stacked statemaps

To render a single SVG that contains multiple statemaps, multiple data files can be provided:

statemap data-1.out data-2.out > statemap.svg

The resulting statemaps will be stacked in the order of the data files as provided on the command line, with the first data file dictating the time bounds of the resulting stack. The statemaps can be similar statemaps from dissimilar entities (e.g., different machines), or they can be dissimilar statemaps (e.g., different statemaps), or any mix of these. Legends for similar statemaps will be shared. The -S option can control the sorting of stacked like statemaps.

When statemaps are stacked, the coalescing factor applies to each statemap rather than to the entire stack of statemaps. When stacking many statemaps, low coalescing factors will be needed to prevent the resulting SVG from becoming excessively large.

More Repositories

1

libuv

Go to
C
3,271
star
2

smartos-live

For more information, please see http://smartos.org/ For any questions that aren't answered there, please join the SmartOS discussion list: http://smartos.org/smartos-mailing-list/
C
1,437
star
3

triton

Joyent Triton DataCenter: a cloud management platform with first class support for containers.
Shell
1,202
star
4

node-verror

Rich JavaScript errors
JavaScript
1,125
star
5

containerpilot

A service for autodiscovery and configuration of applications running in containers
Go
1,104
star
6

manta

Manta is a scalable HTTP-based object store
Makefile
565
star
7

node-workflow

Task orchestration, creation and running using NodeJS
JavaScript
445
star
8

node-http-signature

Reference implementation of Joyent's HTTP Signature Scheme
JavaScript
392
star
9

node-stackvis

Stacktrace visualization tools
JavaScript
340
star
10

node-vasync

utilities for observable asynchronous control flow
JavaScript
315
star
11

v8plus

Node.js native add-ons in C
C++
265
star
12

rfd

Requests for Discussion
Roff
251
star
13

mdb_v8

postmortem debugging for Node.js and other V8-based programs
C
235
star
14

manatee

Automated fault monitoring and leader-election system for strongly-consistent, highly-available writes to PostgreSQL (Joyent SDC, Manta).
JavaScript
228
star
15

restdown

Pretty REST API docs authored in Markdown
Python
203
star
16

sdc-docker

Docker Engine for Triton
JavaScript
182
star
17

triton-kubernetes

Kubernetes on Triton
Go
174
star
18

node-sshpk

Parse, convert, fingerprint and use SSH keys in pure node.js
JavaScript
159
star
19

nodejs-advisory-board

Meeting Minutes and Working Group Discussions
158
star
20

nhttpsnoop

Trace Node.js HTTP server activity
Shell
138
star
21

pgsqlstat

report top-level postgres stats
Shell
129
star
22

node-panic

Postmortem debugging facility for Node.js
JavaScript
120
star
23

node-assert-plus

Extra assertions on top of node's assert module
JavaScript
119
star
24

illumos-kvm

KVM driver for illumos
C
117
star
25

node-snmpjs

SNMP toolkit for Node.js
JavaScript
111
star
26

node-ctype

Read and write binary structures with node
JavaScript
89
star
27

node-manta

Node.js SDK for Manta
JavaScript
75
star
28

node-bunyan-syslog

Syslog Stream for node-bunyan
JavaScript
68
star
29

illumos-kvm-cmd

qemu-kvm for illumos-kvm
C
65
star
30

node-watershed

Simple WebSockets Client/Server (RFC6455)
Makefile
65
star
31

node-smartdc

Client SDK and CLI for the Joyent SmartDataCenter API
JavaScript
63
star
32

mi-centos-7

Shell
63
star
33

node-asn1

Contains parsers and serializers for ASN.1 (currently BER only)
AGS Script
61
star
34

smartos_cookbooks

Chef Cookbooks for managing the SmartOS Global Zone
JavaScript
58
star
35

moray

Moray, the highly-available key/value store (Joyent Triton, Manta)
JavaScript
58
star
36

node-vstream

instrumented streams
JavaScript
56
star
37

node-triton

Triton client tool and node.js library
JavaScript
55
star
38

node-docker-registry-client

node.js client for the docker registry
JavaScript
55
star
39

kang

Introspection for distributed systems
JavaScript
49
star
40

smfgen

Generate SMF manifests from a JSON description
JavaScript
49
star
41

jsstyle

cstyle-based JavaScript style checker
Perl
49
star
42

node-debug-school

nodeschool curriculum for debugging Node.js
JavaScript
49
star
43

node-getopt

POSIX-style getopt() for Node.js
JavaScript
47
star
44

dtruss-osx

Shell
43
star
45

node-ip6addr

IPv6/IPv4 address parsing and manipulation for node.js
JavaScript
43
star
46

pg_prefaulter

Faults pages into PostgreSQL shared_buffers or filesystem caches in advance of WAL apply
Go
43
star
47

node-camp

Asynchronous IO ...camp
JavaScript
43
star
48

manatee-state-machine

design ideas for manatee
JavaScript
42
star
49

node-docker-file-parser

Parses a dockerfile contents string and returns the array of docker commands
JavaScript
42
star
50

smartos-vmtools

Shell
40
star
51

illumos-extra

Extra non-ON software required for Illumos
C
39
star
52

sdc-nfs

user-level NFS server written in node.js
JavaScript
35
star
53

node-extsprintf

Extended POSIX-style sprintf
JavaScript
34
star
54

node-kstat

A node.js addon for reading illumos kstats
Perl
32
star
55

node-jsprim

utilities for primitive JavaScript types
JavaScript
32
star
56

knife-joyent

Opscode Chef knife plug-in for Joyent CloudAPI
Ruby
32
star
57

eng

Joyent Engineering Guide
JavaScript
31
star
58

pkgsrc-joyent

Various pkgsrc packages used by Joyent, not committed upstream yet
Makefile
31
star
59

smartos-overlay

Overlay directory specific to open-source SmartOS
30
star
60

node-fast

streaming JSON RPC over TCP
JavaScript
29
star
61

convertvm

convert OVF vm packages to smartos compatible images
JavaScript
29
star
62

minecrab

Minecraft on Joyent's Cloud & Manta on Demand
Shell
28
star
63

cloud-perf-labs

Student labs for Cloud Performance training
C
28
star
64

node-consulite

Tiny consul Node.js module for client discovery
JavaScript
28
star
65

node-piloted

Service discovery in node using ContainerPilot
JavaScript
27
star
66

node-in-the-industry

This is the script that used to generate fresh "node in the industry" content. It is no longer being maintained. See: https://github.com/nodejs/nodejs.org.
HTML
27
star
67

mi-freebsd-10

Custom FreeBSD 10 ISO builder
Shell
26
star
68

javascriptlint

JavaScript Lint
C
25
star
69

binder

Triton/Manta DNS server over Apache Zookeeper
JavaScript
25
star
70

node-tracing

User definable tracing API
JavaScript
25
star
71

python-manta

Python SDK for Manta (community maintained)
Python
24
star
72

manufacturing

Manufacturing specifications
Python
24
star
73

pglockanalyze

analyze postgres locking behavior
Makefile
23
star
74

sdcboot

SDC FDUM environment
C
23
star
75

pkgsrc-wip

Conversion of the pkgsrc-wip CVS project
Makefile
23
star
76

conch-api

Datacenter build and management service
Perl
22
star
77

node-tab

Unix-style tables for command-line utilities
Makefile
22
star
78

triton-go

Go SDK for Joyent Triton (Compute) and Triton Object Storage (Manta)
Go
21
star
79

node-spawn-async

spawn child processes asynchronously
JavaScript
19
star
80

smartmachine_cookbooks

Chef Cookbooks for managing SmartOS SmartMachines
19
star
81

syslinux

replica of syslinux repo from git://git.kernel.org/pub/scm/boot/syslinux/syslinux.git
C
19
star
82

manta-nfs

NFSv3 Manta Storage Server Gateway
JavaScript
19
star
83

daggr

filter and aggregate numeric data in plaintext or json form
JavaScript
18
star
84

mod_usdt

DTrace provider for Apache
D
18
star
85

freebsd-vpc

Control plane for `projects/VPC` branch of `joyent/freebsd`
Go
18
star
86

mibe

Machine Image Build Environment
PHP
17
star
87

node-zfs

Node.js library to interface with ZFS utilities
JavaScript
17
star
88

ruby-manta

Ruby interface for Joyent's Manta service
Ruby
17
star
89

pgstatsmon

Node.js service for shoveling Postgres stats into Prometheus
JavaScript
17
star
90

tsg-infrastructure

Shell
17
star
91

java-manta

Java Manta Client SDK
Java
16
star
92

manta-thoth

Thoth is a Manta-based system for core and crash dump management
JavaScript
16
star
93

node-nfs

Node.js SDK for writing Portmap/Mount/NFS (v3) servers
JavaScript
16
star
94

triton-terraform

16
star
95

java-http-signature

Library for performing RSA signed HTTP requests in Java
Java
16
star
96

summit-workshop

Node.js Summit - Day Zero Workshop
JavaScript
16
star
97

sdc-adminui

Operator portal for SmartDataCenter
JavaScript
15
star
98

sdc-headnode

Responsible for building and setting up the Triton (formerly SmartDataCenter) headnode.
JavaScript
15
star
99

openbsd-kvm-image-builder

Scripts to create a custom OpenBSD install ISO and a KVM image for use in SmartOS and Triton.
Shell
15
star
100

ipxe

C
14
star