• Stars
    star
    120
  • Rank 295,983 (Top 6 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created almost 8 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

The SignalFx Smart Agent

โ„น๏ธย ย SignalFx was acquired by Splunk in October 2019. See Splunk SignalFx for more information.

โš ๏ธ End of Support (EoS) Notice

The SignalFx Smart Agent has reached End of Support.

The Splunk Distribution of OpenTelemetry Collector is the successor. Smart Agent monitors are available and supported through the Smart Agent receiver in the Splunk Distribution of OpenTelemetry Collector.

To learn how to migrate, see Migrate from SignalFx Smart Agent to the Splunk Distribution of OpenTelemetry Collector.

SignalFx Smart Agent

GoDoc CircleCI

The SignalFx Smart Agent is a metric agent written in Go for monitoring infrastructure and application services in a variety of different environments. It is meant as a successor to our previous collectd agent, but still uses that internally on Linux -- so any existing Python or C-based collectd plugins will still work without modification. On Windows collectd is not included, but the agent is capable of running python based collectd plugins without collectd. C-based collectd plugins are not available on Windows.

Concepts

The agent has three main components:

  1. Observers that discover applications and services running on the host
  2. Monitors that collect metrics, events, and dimension properties the host and applications
  3. The Writer that sends the metrics, events, and dimension updates collected by monitors to SignalFx.

Observers

Observers watch the various environments that we support to discover running services and automatically configure the agent to send metrics for those services.

For a list of supported observers and their configurations, see Observer Config.

Monitors

Monitors collect metrics from the host system and services. They are configured under the monitors list in the agent config. For application-specific monitors, you can define discovery rules in your monitor configuration. A separate monitor instance is created for each discovered instance of applications that match a discovery rule. See Auto Discovery for more information.

Many of the monitors are built around collectd, an open source third-party monitor, and use it to collect metrics. Some other monitors do not use collectd. However, either type is configured in the same way.

For a list of supported monitors and their configurations, see Monitor Config.

The agent is primarily intended to monitor services/applications running on the same host as the agent. This is in keeping with the collectd model. The main issue with monitoring services on other hosts is that the host dimension that collectd sets on all metrics will currently get set to the hostname of the machine that the agent is running on. This allows everything to have a consistent host dimension so that metrics can be matched to a specific machine during metric analysis.

Writer

The writer collects metrics emitted by the configured monitors and sends them to SignalFx on a regular basis. There are a few things that can be configured in the writer, but this is generally only necessary if you have a very large number of metrics flowing through a single agent.

Installation

The agent is available for Linux in both a containerized and standalone form. Whatever form you use, the dependencies are completely bundled along with the agent, including a Java JRE runtime and a Python runtime, so there are no additional dependencies required. This means that the agent should work on any relatively modern Linux distribution (kernel version 2.6+).

Note: The agent is incompatible on Linux systems with SELinux enabled. Check the documentation for your distribution to learn how to disable SELinux.

The agent is also available on Windows in standalone form. It contains its own Python runtime. The agent supports Windows Server 2012 and above.

To get started deploying the Smart Agent directly on a host, see the Smart Agent Quick Install guide.

Deployment

We support the following deployment/configuration management tools to automate the installation process. See Bundles for a list of underlying packages for the agent.

Installer Script

For non-containerized environments, there is a convenience script that you can run on your host to install the agent package. This is useful for testing and trials, but for full-scale deployments you will probably want to use a configuration management system like Chef or Puppet.

Linux

You can view the source for the installer script and use it on your hosts by running:

curl -sSL https://dl.signalfx.com/signalfx-agent.sh > /tmp/signalfx-agent.sh;
sudo sh /tmp/signalfx-agent.sh --realm YOUR_SIGNALFX_REALM -- YOUR_SIGNALFX_API_TOKEN
Windows

The Agent has one dependency on Windows which must be satisfied before running the installer script.

The installer script is written for PowerShell v3.0 and above and will not function correctly on earlier versions.

Once the dependencies have been installed, please run the installer script below. You can view the source for the installer script and use it on your hosts in PowerShell by running:

& {Set-ExecutionPolicy Bypass -Scope Process -Force; $script = ((New-Object System.Net.WebClient).DownloadString('https://dl.signalfx.com/signalfx-agent.ps1')); $params = @{access_token = "YOUR_SIGNALFX_ACCESS_TOKEN"; ingest_url = "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com"; api_url = "https://api.YOUR_SIGNALFX_REALM.signalfx.com"}; Invoke-Command -ScriptBlock ([scriptblock]::Create(". {$script} $(&{$args} @params)"))}

The agent files are installed to \Program Files\SignalFx\SignalFxAgent, and the default configuration file is installed at \ProgramData\SignalFxAgent.

You can also use Chocolatey to install the agent. See the section Windows Chocolatey Package.

Chef

We offer a Chef cookbook to install and configure the agent. See the cookbook source and on the Chef Supermarket.

Puppet

We also offer a Puppet manifest to install and configure the agent on Linux. See the manifest source and on the Puppet Forge.

Ansible

We also offer an Ansible Role to install and configure the Smart Agent on Linux. See the role source.

Salt

We also offer a Salt Formula to install and configure the Smart Agent on Linux. See the formula source.

Docker Image

See Docker Deployment for more information.

Kubernetes

See our Kubernetes setup instructions and Monitor Kubernetes for more information. Helm version 3 or higher is supported.

AWS Elastic Container Service (ECS)

See the ECS directory, which includes a sample config and task definition for the agent.

Bundles

We offer the agent in the following forms:

Debian Package

We provide a Debian package repository that you can make use of with the following commands:

curl -sSL https://splunk.jfrog.io/splunk/signalfx-agent-deb/splunk-B3CD4420.gpg > /etc/apt/trusted.gpg.d/splunk.gpg
echo 'deb https://splunk.jfrog.io/splunk/signalfx-agent-deb release main' > /etc/apt/sources.list.d/signalfx-agent.list
apt-get update
apt-get install -y signalfx-agent

RPM Package

We provide a RHEL/RPM package repository that you can make use of with the following commands:

cat <<EOH > /etc/yum.repos.d/signalfx-agent.repo
[signalfx-agent]
name=SignalFx Agent Repository
baseurl=https://splunk.jfrog.io/splunk/signalfx-agent-rpm/release
gpgcheck=1
gpgkey=https://splunk.jfrog.io/splunk/signalfx-agent-rpm/splunk-B3CD4420.pub
enabled=1
EOH

yum install -y signalfx-agent

Linux Standalone tar.gz

If you don't want to use a distro package, we offer a .tar.gz that can be deployed to the target host. This bundle is available for download on the Github Releases Page for each new release.

To use the bundle:

  1. Unarchive it to a directory of your choice on the target system.

  2. Go into the unarchived signalfx-agent directory and run bin/patch-interpreter $(pwd). This ensures that the binaries in the bundle have the right loader set on them since your host's loader may not be compatible.

  3. Ensure a valid configuration file is available somewhere on the target system. The main thing that the distro packages provide -- but that you will have to provide manually with the bundle -- is a run directory for the agent to use. Since you aren't installing from a package, there are three config options that you will especially want to consider:

  • internalStatusHost - This is the host name that the agent will listen on so that the signalfx-agent status command can read diagnostic information from a running agent. This is also the host name the agent will listen on to serve internal metrics about the agent. These metrics can can be scraped by the internal-metrics monitor. This will default to localhost if left blank.

  • internalStatusPort - This is the port that the agent will listen on so that the signalfx-agent status command can read diagnostic information from a running agent. This is also the host name the agent will listen on to serve internal metrics about the agent. These metrics can can be scraped by the internal-metrics monitor. This will default to 8095.

  • collectd.configDir - This is where the agent writes the managed collectd config, since collectd can only be configured by files. Note that this entire dir will be wiped by the agent upon startup so that it doesn't pick up stale collectd config, so be sure that it is not used for anything else. Also note that these files could have sensitive information in them if you have passwords configured for collectd monitors, so you might want to place this dir on a tmpfs mount to avoid credentials persisting on disk.

See the section on Privileges for information on the capabilities the agent requires.

  1. Run the agent by invoking the archive path signalfx-agent/bin/signalfx-agent -config <path to config.yaml>. By default, the agent logs only to stdout/err. If you want to persist logs, you must direct the output to a log file or other log management system. See the signalfx-agent command doc for more information on supported command flags.

Windows Chocolatey Package

Only available for Smart Agent v5.3.0 and higher.

To install the Smart Agent using Chocolatey, run the following PowerShell command as an administrator:

choco install signalfx-agent [choco options] --params="'/access_token:YOUR_SIGNALFX_ACCESS_TOKEN /ingest_url:https://ingest.YOUR_SIGNALFX_REALM.signalfx.com /api_url:https://api.YOUR_SIGNALFX_REALM.signalfx.com'"

The Smart Agent looks for a configuration file at \ProgramData\SignalFxAgent\agent.yaml. If this file does not already exist during installation, a default config file will be copied into place by the installer.

The following package parameters are available:

  • /access_token - The access token (org token) used to send metric data to SignalFx. If the parameter is specified, the token will be saved to the \ProgramData\SignalFxAgent\token file. If the parameter is not specified and \ProgramData\SignalFxAgent\token does not exist or is empty, the Smart Agent service is not started after installation or upgrade. To start the service, add or update \ProgramData\SignalFxAgent\token with a valid token, and then either restart Windows or run the following PowerShell command: & "\Program Files\SignalFx\SignalFxAgent\bin\signalfx-agent.exe" -service "start"
  • /ingest_url - URL of the SignalFx ingest endpoint (e.g. https://ingest.YOUR_SIGNALFX_REALM.signalfx.com). The URL will be saved to the \ProgramData\SignalFxAgent\ingest_url file. If the parameter is not specified, the value found in \ProgramData\SignalFxAgent\ingest_url (if it exists) will be used. Otherwise, defaults to https://ingest.us0.signalfx.com.
  • /api_url - URL of the SignalFx API endpoint (e.g. https://api.YOUR_SIGNALFX_REALM.signalfx.com). The URL will be saved to the \ProgramData\SignalFxAgent\api_url file. If the parameter is not specified, the value found in \ProgramData\SignalFxAgent\api_url (if it exists) will be used. Otherwise, defaults to https://api.us0.signalfx.com.
  • /install_dir - Installation directory. Defaults to \Program Files\SignalFx\SignalFxAgent.

To learn more, see the Chocolatey SignalFx Smart Agent page.

Windows Standalone .zip

A .zip bundle is also available that can be deployed to the target host. To obtain the bundle, go to Github Releases Page and download the most recent release.

To learn more, see Install to Windows using a ZIP file.

Privileges

Linux

When using the host observer, the agent requires the Linux capabilities DAC_READ_SEARCH and SYS_PTRACE, both of which are necessary to allow the agent to determine which processes are listening on network ports on the host. Otherwise, there is nothing built into the agent that requires privileges. When using a package to install the agent, the agent binary is given those capabilities in the package post-install script, but the agent is run as the signalfx-agent user. If you are not using the host observer, then you can strip those capabilities from the agent binary if so desired.

You should generally not run the agent as root unless you can't use capabilities for some reason.

Windows

On Windows, the Smart Agent can be installed and run using an Administrator account. You can also run the Smart Agent in non-Administrator mode, See Configure user privileges.

Configuration

The agent is configured primarily from a YAML file. By default, the agent config is installed at and looked for at /etc/signalfx/agent.yaml on Linux and \ProgramData\SignalFxAgent\agent.yaml on Windows. This can be overridden by the -config command line flag.

For the full schema of the config, see Config Schema.

For information on how to configure the agent from remote sources, such as other files on the filesystem or KV stores such as Etcd, see Remote Configuration.

Logging

The default log level is info, which will log anything noteworthy in the agent without spamming the logs too much. Most of the info level logs are on startup and upon service discovery changes. debug will create very verbose log output and should only be used when trying to resolve a problem with the agent. You can change the log level with the logging: {level: info} YAML config option.

The agent will emit logs in either an unstructed text (default) or JSON format. You can configure it to emit JSON logs with the logging: {format: json} YAML config option.

Linux

Currently the agent only supports logging to stdout/stderr, which will generally be redirected by the init scripts we provide to either a file at /var/log/signalfx-agent.log or to the systemd journal on newer distros.

Windows

On Windows, the agent will log to the console when executed directly in a shell. If the agent is configured as a windows service, log events will be logged to the Windows Event Log. Use the Event Viewer application to read the logs. The Event Viewer is located under Start > Administrative Tools > Event Viewer. You can see logged events from the agent service under Windows Logs > Application.

Proxy Support

To use an HTTP(S) proxy, set the environment variable HTTP_PROXY and/or HTTPS_PROXY in the container configuration to proxy either protocol. The SignalFx ingest and API servers both use HTTPS. If the NO_PROXY envvar exists, the agent will automatically append the local services to the envvar to not use the proxy.

If the agent is running as a local service on the host, refer to the host's service management documentation for how to pass environment variables to the agent service in order to enable proxy support when the agent service is started.

For example, if the host services are managed by systemd, create the /etc/systemd/system/signalfx-agent.service.d/myproxy.conf file and add the following to the file:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080/"
Environment="HTTPS_PROXY=https://proxy.example.com:8081/"

Then execute systemctl daemon-reload and systemctl restart signalfx-agent.service to restart the agent service with proxy support.

Sys-V based init.d systems: Debian * RHEL

Create /etc/default/signalfx-agent with the following contents:

HTTP_PROXY="http://proxy.example.com:8080/"
HTTPS_PROXY="https://proxy.example.com:8081/"

Diagnostics

The agent serves diagnostic information on an HTTP server at the address configured by the internalStatusHost and internalStatusPort option. As a convenience, the command signalfx-agent status will read this server and dump out its contents. That command will also explain how to get further diagnostic information.

Also see our FAQ for more troubleshooting help.

Development

If you wish to contribute to the agent, see the Developer's Guide.

More Repositories

1

maestro-ng

Orchestration of Docker-based, multi-host environments
Python
682
star
2

splunk-otel-collector

Go
191
star
3

integrations

SignalFx integrations metadata
Python
144
star
4

splunk-otel-collector-chart

Splunk OpenTelemetry Collector for Kubernetes
Smarty
119
star
5

gateway

A proxy to buffer and forward metrics, events, and traces.
Go
97
star
6

swagger-angular-client

AngularJS service for communicating with endpoints described by swagger.
JavaScript
78
star
7

docker-zookeeper

Docker image for ZooKeeper (Maestro orchestration)
Python
69
star
8

golib

Golib contains various useful golang libraries
Go
66
star
9

splunk-otel-java

Splunk Distribution of OpenTelemetry Java
Java
63
star
10

docker-collectd

Collectd within a Docker image
Shell
58
star
11

tracing-examples

Examples of using third-party tracers with SignalFx
PHP
53
star
12

splunk-otel-js-web

Splunk distribution of Open Telemetry for browser environment.
TypeScript
49
star
13

splunk-otel-go

Splunk Distribution of OpenTelemetry Go
Go
47
star
14

signalfx-python

Python client library and instrumentation bindings for SignalFx
Python
43
star
15

splunk-otel-python

Splunk distribution of OpenTelemetry Python
Python
39
star
16

signalfx-java

Java client libraries for SignalFx
Java
37
star
17

splunk-otel-android

Android RUM library and instrumentation
Java
37
star
18

swagger-cli-client

Generates a command-line interface for any Swagger Specification
JavaScript
36
star
19

swagger-client-generator

Automatically generate an api object from a given swagger 1.2 schema
JavaScript
35
star
20

signalflow-library

Modules and Examples of SignalFlow usage
Shell
32
star
21

signalfx-dotnet-tracing

Automatic instrumentations for popular .NET libraries and frameworks. (Deprecated)
C#
32
star
22

fetch-swagger-schema

Given a url to a swagger api-docs spec, loads it and all related resource declarations as one JSON object.
JavaScript
30
star
23

splunk-otel-ios

iOS RUM library and instrumentation
Swift
23
star
24

swagger-validate

Detailed validation of objects against a given swagger model
JavaScript
22
star
25

signalfx-nodejs-tracing

OpenTracing-compatible Auto-Instrumentor for Node.js
JavaScript
21
star
26

splunk-otel-js

Splunk Distribution of OpenTelemetry JavaScript
TypeScript
21
star
27

signalfx-nodejs

Node.js client library for SignalFx
JavaScript
21
star
28

signalfx-python-tracing

OpenTracing-compatible Auto-Instrumentor for Python
Python
20
star
29

apmworkshop

19
star
30

signalfx-php-tracing

OpenTracing-compatible Auto-Instrumentor for PHP (Deprecated)
PHP
18
star
31

splunk-otel-react-native

Java
17
star
32

signalfx-jumpstart

POC in a Box
HCL
17
star
33

swagger-ajax-client

Create XHR clients for Swagger API Specifications
JavaScript
16
star
34

terraform-signalfx-slx

Best practices SLI, SLO and error budget monitoring, visualization and more for organizations using SignalFx
HCL
16
star
35

lambda-python

SignalFx AWS Lambda Python Wrapper
Python
15
star
36

signalflow-cli

Python command-line client for SignalFx SignalFlowโ„ข Analytics
Python
14
star
37

signalfx-go

Go client library and instrumentation bindings for SignalFx
Go
14
star
38

signalfx-go-tracing

Go
13
star
39

collectd-zookeeper

CollectD plugin for ZooKeeper metrics
Python
13
star
40

otelworkshop

Shell
13
star
41

swagger-node-client

Create clients for Swagger API Specifications.
JavaScript
10
star
42

lambda-nodejs

SignalFx AWS Lambda Node.js Wrapper
JavaScript
10
star
43

ruby-rails-instrumentation

OpenTracing instrumentation for Rails
Ruby
10
star
44

signalfx-k8s-metrics-adapter

Horizontal Pod Autoscaler custom/external metrics provider for Kubernetes that uses SignalFx as the backend
Go
10
star
45

splunk-otel-collector-operator

Kubernetes Operator for OpenTelemetry Collector
Go
10
star
46

python-celery

OpenTracing instrumentation for Celery
Python
9
star
47

splunk-forwarder

a custom search command to metricize Splunk logs
Python
9
star
48

signalfx-java-tracing

The SignalFx Java Agent
Java
9
star
49

lambda-layer-versions

8
star
50

app-dev-workshop

Files used for creation of workshop instances, useful tools and anything else!
8
star
51

grafana-signalfx-datasource

JavaScript
7
star
52

signalfx-istio-adapter

An Istio Mixer adapter for sending metrics and trace spans to SignalFx
Go
7
star
53

signalfx-collectd-plugin

SignalFx CollectD plugin
Python
7
star
54

splunk-otel-dotnet

Splunk Distribution of OpenTelemetry .NET
C#
7
star
55

enhanced-rds-monitoring

Lambda for enhanced AWS/RDS monitoring
Python
6
star
56

splunk-otel-collector-heroku

Shell
6
star
57

signalfx-ruby

Ruby client library for SignalFx
Ruby
6
star
58

collectd-build-ubuntu

Shell
5
star
59

splunk-otel-java-lambda

Splunk custom OTEL Java wrapper for AWS lambdas
Java
5
star
60

collectd-rabbitmq

collectd plugin for integration with rabbitmq
Python
5
star
61

splunk-otel-lambda

Splunk distribution of OpenTelemetry Lambda
Shell
5
star
62

s3-util

High-level S3 utilities
JavaScript
5
star
63

pyfilewatch

Watch files and directories in python. Also supports tailing and glob file patterns.
Python
4
star
64

terraform-workshop

Monitoring as code workshop
HCL
4
star
65

maestro-base

Base Docker image for Maestro-enabled components
Shell
4
star
66

signalfx-ruby-tracing

Automatic OpenTracing instrumentation for Ruby
Ruby
4
star
67

circleutil

Utility scripts for circleci
Go
4
star
68

kubectl-splunk

kubectl plugin for interacting with OpenTelemetry deployments
Go
4
star
69

collectd-jenkins

collectd based plugin for Jenkins
Python
3
star
70

aws-log-collector

AWS Lambda function which forwards AWS logs to Splunk Observability.
Python
3
star
71

kong-plugin-signalfx

Kong API Gateway metric aggregator for SignalFx
Lua
3
star
72

call-conduit

Programmatically call phabricator/conduit
JavaScript
3
star
73

signalfx-nodejs-collect

JavaScript
3
star
74

splunk-otel-java-overhead-test

Test tools to help quantify the overhead of the Java instrumentation agents.
Java
3
star
75

gdi-specification

Splunk GDI specification for cross-repository compatibility
Makefile
3
star
76

signalflow-grammar

SignalFx SignalFlow language grammar
ANTLR
3
star
77

sapm-proto

SAPM ProtoBuf
Go
3
star
78

collectd-kong

CollectD plugin for Kong API Gateway
Python
3
star
79

ruby-sequel-instrumentation

Ruby
3
star
80

PerfCounterReporter

Windows service for reporting Windows Performance Counters to SignalFx
C#
3
star
81

alert-assessor

SignalFx Alert Evaluation Tool
Python
3
star
82

puppet_collectd

A Puppet module to install collectd, configure plugins and send metrics
Python
2
star
83

ansible-signalfx-agent

Install and Configure the SignalFx Smart Agent. (Partial mirror of https://github.com/signalfx/signalfx-agent. Do not edit manually.)
2
star
84

aws-cloudformation-templates

CloudFormation templates describing AWS resources to deploy in user's AWS account when setting up AWS integration
Makefile
2
star
85

victorops-field-scripts

Scripts created by the field team utilizing the VictorOps API
Python
2
star
86

collectd-example

Python
2
star
87

Metrics.NET.SignalFx

SignalFx Reporter for Metrics.NET
C#
2
star
88

splunk-otel-ios-crashreporting

Swift
2
star
89

collectd-openstack

CollectdD plugin to monitor OpenStack cloud infrastructure
Python
2
star
90

cadvisor-integration

The SignalFx Metric Data Collector for Kubernetes
Go
2
star
91

replay

Python
2
star
92

splunk-extension-wrapper

Splunk Lambda Extension for Metrics
Go
2
star
93

collectd-couchbase

CollectD plugin for Couchbase metrics
Python
2
star
94

collectd-spark

collectd plugin for monitoring apache spark
Python
2
star
95

catchpoint-integration

How to integrate SignalFx and Catchpoint systems with regards to Synthetic Monitoring and RUM.
2
star
96

opentracing-python-patches

Python
2
star
97

lambda-java

SignalFx AWS Lambda Java Wrapper
Java
2
star
98

lambda-csharp

SignalFx AWS Lambda C# Wrapper
C#
2
star
99

signalfx-cloudfoundry-bridge-boshrelease

A BOSH release of our Cloud Foundry Metric Bridge
Makefile
2
star
100

splunk-otel-ruby

Splunk Distribution of OpenTelemetry Ruby [Deprecated]
Ruby
2
star