• Stars
    star
    330
  • Rank 127,657 (Top 3 %)
  • Language
    Java
  • License
    MIT License
  • Created almost 10 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Simple JVM Profiler Using StatsD and Other Metrics Backends

statsd-jvm-profiler Build Status

statsd-jvm-profiler is a JVM agent profiler that sends profiling data to StatsD. Inspired by riemann-jvm-profiler, it was primarily built for profiling Hadoop jobs, but can be used with any JVM process.

Read the blog post that introduced statsd-jvm-profiler on Code as Craft, Etsy's engineering blog.

Also check out the blog post reflecting on the experience of open-sourcing the project.

Mailing List

There is a mailing list for this project at https://groups.google.com/forum/#!forum/statsd-jvm-profiler. If you have questions or suggestions for the project send them here!

Installation

You will need the statsd-jvm-profiler JAR on the machine where the JVM will be running. If you are profiling Hadoop jobs, that means the JAR will need to be on all of the datanodes.

The JAR can be built with mvn package. You will need a relatively recent Maven (at least Maven 3).

statsd-jvm-profiler is available in Maven Central:

<dependency>
  <groupId>com.etsy</groupId>
  <artifactId>statsd-jvm-profiler</artifactId>
  <version>2.0.0</version>
</dependency>

If you would like an uberjar containing all of the dependencies instead of the standard JAR, use the jar-with-dependencies classifier:

<dependency>
  <groupId>com.etsy</groupId>
  <artifactId>statsd-jvm-profiler</artifactId>
  <version>2.0.0</version>
  <classifier>jar-with-dependencies</classifier>
</dependency>

Usage

The profiler is enabled using the JVM's -javaagent argument. You are required to specify at least the StatsD host and port number to use. You can also specify the prefix for metrics and a whitelist of packages to be included in the CPU profiling. Arguments can be specified like so:

-javaagent:/usr/etsy/statsd-jvm-profiler/statsd-jvm-profiler.jar=server=hostname,port=num

You should use the uberjar when starting the profiler in this manner so that all the profiler's dependencies are available.

The profiler can also be loaded dynamically (after the JVM has already started), but this technique requires relying on Sun's tools.jar, meaning it's an implementation-specific solution that might not work for all JVMs. For more information see the Dynamic Loading section.

An example of setting up Cascading/Scalding jobs to use the profiler can be found in the example directory.

Global Options

Name Meaning
server The hostname to which the reporter should send data (required)
port The port number for the server to which the reporter should send data (required)
prefix The prefix for metrics (optional, defaults to statsd-jvm-profiler)
packageWhitelist Colon-delimited whitelist for packages to include (optional, defaults to include everything)
packageBlacklist Colon-delimited whitelist for packages to exclude (optional, defaults to exclude nothing)
profilers Colon-delimited list of profiler class names (optional, defaults to CPUTracingProfiler and MemoryProfiler)
reporter Class name of the reporter to use (optional, defaults to StatsDReporter)
httpServerEnabled Determines if the embedded HTTP server should be started. (optional, defaults to true)
httpPort The port on which to bind the embedded HTTP server (optional, defaults to 5005). If this port is already in use, the next free port will be taken.

Embedded HTTP Server

statsd-jvm-profiler embeds an HTTP server to support simple interactions with the profiler while it is in operation. You can configure the port on which this server runs with the httpPort option. You can disable it altogether using the httpServerEnabled=false argument.

Endpoint Usage
/profilers List the currently enabled profilers
/isRunning List the running profilers. This should be the same as /profilers.
/disable/:profiler Disable the profiler specified by :profiler. The name must match what is returned by /profilers.
/errors List the past 10 errors from the running profilers and reporters.
/status/profiler/:profiler Displays a status message with the number of recorded stats for the requested profiler.

Reporters

statsd-jvm-profiler supports multiple backends. StatsD is the default, but InfluxDB is also supported. You can select the backend to use by passing the reporter argument to the profiler; StatsDReporter and InfluxDBReporter are the supported values.

Some reporters may require additional arguments.

StatsDReporter

This reporter does not have any additional arguments.

InfluxDBReporter

Name Meaning
username The username with which to connect to InfluxDB (required)
password The password with which to connect to InfluxDB (required)
database The database to which to write metrics (required)
tagMapping A mapping of tag names from the metric prefix (optional, defaults to no mapping)
useHttps A flag indicating if https connecition should be used (optional, defaults to false)
Tag Mapping

InfluxDB 0.9 supports tagging measurements and querying based on those tags. statsd-jvm-profilers uses these tags to support richer querying of the produced data. For compatibility with other metric backends, the tags are extracted from the metric prefix.

If the tagMapping argument is not defined, only the prefix tag will be added, with the value of the entire prefix.

tagMapping should be a period-delimited set of tag names. It must have the same number of components as prefix, or else an exception would be thrown. Each component of tagMapping is the name of the tag. The component in the corresponding position of prefix will be the value.

If you do not want to include a component of prefix as a tag, use the special name SKIP in tagMapping for that position.

Profilers

statsd-jvm-profiler offers 3 profilers: MemoryProfiler, CPUTracingProfiler and CPULoadProfiler.

The metrics for all these profilers will prefixed with the value from the prefix argument or it's default value: statsd-jvm-profiler.

You can enable specific profilers through the profilers argument like so:

  1. Memory metrics only: profilers=MemoryProfiler
  2. CPU Tracing metrics only: profilers=CPUTracingProfiler
  3. JVM/System CPU load metrics only: profilers=CPULoadProfiler

Default value: profilers=MemoryProfiler:CPUTracingProfiler

Garbage Collector and Memory Profiler: MemoryProfiler

This profiler will record:

  1. Heap and non-heap memory usage
  2. Number of GC pauses and GC time

Assuming you use the default prefix of statsd-jvm-profiler, the memory usage metrics will be under statsd-jvm-profiler.heap and statsd-jvm-profiler.nonheap, the GC metrics will be under statsd-jvm-profiler.gc.

Memory and GC metrics are reported once every 10 seconds.

CPU Tracing Profiler: CPUTracingProfiler

This profiler records the time spent in each function across all Threads.

Assuming you use the default prefix of statsd-jvm-profiler, the the CPU time metrics will be under statsd-jvm-profiler.cpu.trace.

The CPU time is sampled every millisecond, but only reported every 10 seconds. The CPU time metrics represent the total time spent in that function.

Profiling a long-running process or a lot of processes simultaneously will produce a lot of data, so be careful with the capacity of your StatsD instance. The packageWhitelist and packageBlacklist arguments can be used to limit the number of functions that are reported. Any function whose stack trace contains a function in one of the whitelisted packages will be included.

The visualization directory contains some utilities for visualizing the output of this profiler.

JVM And System CPU Load Profiler: CPULoadProfiler

This profiler will record the JVM's and the overall system's CPU load, if the JVM is capable of providing this information.

Assuming you use the default prefix of statsd-jvm-profiler, the JVM CPU load metrics will be under statsd-jvm-profiler.cpu.jvm, and the System CPU load wil be under statsd-jvm-profiler.cpu.system.

The reported metrics will be percentages in the range of [0, 100] with 1 decimal precision.

CPU load metrics are sampled and reported once every 10 seconds.

Important notes:

  • This Profiler is not enabled by default. To enable use the argument profilers=CPULoadProfiler
  • This Profiler relies on Sun/Oracle-specific JVM implementations that offer a JMX bean that might not be available in other JVMs. Even if you are using the right JVM, there's no guarantee this JMX bean will remain there in the future.
  • The minimum required JVM version that offers support for this is for Java 7.
  • See com.sun.management.OperatingSystemMXBean for more information.
  • If the JVM doesn't support the required operations, the metrics above won't be reported at all.

Dynamic Loading of Agent

  1. Make sure you have the tools.jar available in your classpath during compilation and runtime. This JAR is usually found in the JAVA_HOME directory under the /lib folder for Oracle Java installations.
  2. Make sure the jvm-profiler JAR is available during runtime.
  3. During your application boostrap process, do the following:
  val jarPath: String = s"$ABSOLUTE_PATH_TO/com.etsy.statsd-jvm-profiler-$VERSION.jar"
  val agentArgs: String = s"server=$SERVER,port=$PORT"
  attachJvmAgent(jarPath, agentArgs)

  def attachJvmAgent(profilerJarPath: String, agentArgs: String): Unit = {
    val nameOfRunningVM: String = java.lang.management.ManagementFactory.getRuntimeMXBean.getName
    val p: Integer = nameOfRunningVM.indexOf('@')
    val pid: String = nameOfRunningVM.substring(0, p)

    try {
      val vm: com.sun.tools.attach.VirtualMachine = com.sun.tools.attach.VirtualMachine.attach(pid)
      vm.loadAgent(profilerJarPath, agentArgs)
      vm.detach()
      LOGGER.info("Dynamically loaded StatsD JVM Profiler Agent...");
    } catch {
      case e: Exception => LOGGER.warn(s"Could not dynamically load StatsD JVM Profiler Agent ($profilerJarPath)", e);
    }
  }

Contributing

Contributions are highly encouraged! Check out the contribution guidlines.

Any ideas you have are welcome, but check out some ideas for contributions.

More Repositories

1

AndroidStaggeredGrid

An Android staggered grid view which supports multiple columns with rows of varying sizes.
Java
4,756
star
2

skyline

It'll detect your anomalies! Part of the Kale stack.
Python
2,135
star
3

logster

Parse log files, generate metrics for Graphite and Ganglia
Python
1,968
star
4

deployinator

Deployinate!
Ruby
1,878
star
5

morgue

post mortem tracker
PHP
1,017
star
6

411

An Alert Management Web Application
PHP
971
star
7

feature

Etsy's Feature flagging API used for operational rampups and A/B testing.
PHP
869
star
8

MIDAS

Mac Intrusion Detection Analysis System
833
star
9

opsweekly

On call alert classification and reporting
JavaScript
761
star
10

oculus

The metric correlation component of Etsy's Kale system
Java
707
star
11

mctop

a top like tool for inspecting memcache key values in realtime
Ruby
507
star
12

supergrep

realtime log streamer
JavaScript
411
star
13

Conjecture

Scalable Machine Learning in Scalding
Java
361
star
14

nagios-herald

Add context to Nagios alerts
Ruby
322
star
15

dashboard

JavaScript
308
star
16

boundary-layer

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform
Python
262
star
17

Testing101

Etsy's educational materials on testing and design
PHP
262
star
18

DebriefingFacilitationGuide

Leading Groups at Etsy to Learn From Accidents
247
star
19

phpunit-extensions

Etsy PHPUnit Extensions
PHP
228
star
20

nagios_tools

Tools for use with Nagios
Python
173
star
21

open-api

We are working on a new version of Etsy’s Open API and want feedback from developers like you.
166
star
22

TryLib

TryLib is a simple php library that helps you generate a diff of your working copy and send it to Jenkins to run the test suite(s) on the latest code patched with your changes.
PHP
155
star
23

BugHunt-iOS

Objective-C
148
star
24

mod_realdoc

Apache module to support atomic deploys - http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/
C
128
star
25

ab

Etsy's little framework for A/B testing, feature ramp up, and more.
128
star
26

wpt-script

Scripts to generate WebPagetest tests and download results
PHP
121
star
27

applepay-php

A PHP extension that verifies and decrypts Apple Pay payment tokens
C
118
star
28

foodcritic-rules

Etsy's foodcritic rules
Ruby
115
star
29

kevin-middleware

This is an Express middleware that makes developing javascript in a monorepo easier.
JavaScript
110
star
30

mixer

a tool to initiate meetings by randomly pairing individuals
Go
100
star
31

cloud-jewels

Estimate energy consumption using GCP Billing Data
TSQL
96
star
32

jenkins-master-project

Jenkins Plugin: Master Project. Jenkins project type that allows for selection of sub-jobs to execute, watch, and report worst status of all sub-projects.
Java
83
star
33

Sahale

A Cascading Workflow Visualizer
JavaScript
83
star
34

PushBot

An IRC Bot for organizing code pushes
Java
79
star
35

cdncontrol

CLI tool for working with multiple CDNs
Ruby
79
star
36

rules_grafana

Bazel rules for building Grafana dashboards
Starlark
70
star
37

chef-whitelist

Simple library to enable host based rollouts of changes
Ruby
68
star
38

rfid-checkout

Low Frequency RFID check out/in client for Raspberry Pi
Python
64
star
39

Etsy-Engineering-Career-Ladder

Etsy's Engineering Career Ladder
HTML
61
star
40

Evokit

Rust
60
star
41

ELK-utils

Utilities for working with the ELK (Elasticsearch, Logstash, Kibana) stack
Ruby
59
star
42

incpath

PHP extension to support atomic deploys
C
52
star
43

arbiter

A utility for generating Oozie workflows from a YAML definition
Java
48
star
44

VIPERBuilder

Scaffolding for building apps in a clean way with VIPER architecture
Swift
41
star
45

chef-handlers

Chef handlers we use at Etsy
Ruby
40
star
46

sbt-checkstyle-plugin

SBT Plugin for Running Checkstyle on Java Sources
Scala
32
star
47

es-restlog

Plugin for logging Elasticsearch REST requests
Java
29
star
48

yubigpgkeyer

Script to make RSA authentication key generation on Yubikeys differently painful
Python
28
star
49

Apotheosis

Python
28
star
50

jenkins-deployinator

Jenkins Plugin: Deployinator. Links key deployinator information to Jenkins builds via the CLI.
Java
25
star
51

sbt-compile-quick-plugin

SBT Plugin for Compiling a Single File
Scala
25
star
52

geonames

Scripts for using Geonames
PHP
24
star
53

jading

cascading.jruby build and execution tool
16
star
54

etsy.github.com

Etsy! on Github!
HTML
16
star
55

divertsy-client

The Android client for running DIVERTsy, a waste stream recording tool to help track diversion rates.
Java
13
star
56

cdncontrol_ui

A web UI for Etsy's cdncontrol tool
CSS
13
star
57

terraform-demux

A user-friendly launcher (Γ  la bazelisk) for Terraform.
Go
12
star
58

logstash-plugins

Ruby
11
star
59

jenkins-triggering-user

Jenkins Plugin: Triggering User. Populates a $TRIGGERING_USER environment variable from the build cause and other sources, a best guess.
10
star
60

EtsyCompositionalLayoutBridge

iOS framework that allows for simultaneously leveraging flow layout and compositional layout in collection views
Swift
3
star
61

consulkit

Ruby API for interacting with HashiCorp's Consul.
Ruby
1
star
62

soft-circuits-workshop

Etsy Soft Circuits Workshop
Arduino
1
star