• Stars
    star
    114
  • Rank 308,031 (Top 7 %)
  • Language
    C#
  • License
    Apache License 2.0
  • Created over 10 years ago
  • Updated over 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Monitoring system instrumentation for Akka.NET actor systems

Akka.Monitoring

Pluggable monitoring system instrumentation for Akka.NET actor systems.

Phobos vs. Akka.Monitoring

Phobos Akka.NET Monitoring, Tracing, and Observability Logo

In 2018 Petabridge, the makers of Akka.Monitoring, published a propreitary product for Akka.NET monitoring and tracing called Phobos.

Phobos is much more fully-featured than Akka.Monitoring, as it:

  1. Doesn't require users to decorate their actors with any code - monitoring and tracing happens automatically;
  2. Supports a much larger number of metric systems and types out of the box;
  3. Includes traceability with actors, including over the network with Akka.Remote, and other parts of your system such as ASP.NET Core; and
  4. Comes with 1 business day SLA support for Phobos and installing it into your Akka.NET applications.

That said, a Phobos license costs your company $4,000 per year.

It's the belief of Petabridge that there should be a low-cost alternative to that in order to make Akka.NET accessible to students, startups, DIY developers, and anyone who doesn't want to pay for that license. That's what Akka.Monitoring is - a simple, straightforward alternative that is free and open source. We will continue to support Akka.Monitoring and respond to end-user requests, but you should expect the majority of our development efforts to go into Phobos.

Please open an issue or contact Petabridge if you have any questions.

What it is

Akka.Monitoring is an ActorSystem extension for Akka.NET that exposes a pluggable layer for reporting performance metrics from actors back to a monitoring system such as Etsy's StatsD or Graphite or Microsoft AppInsights.

Akka.Monitoring can report to multiple monitoring systems simultaneously and it can report different metrics for different discrete ActorSystems inside the same process without any collisions. It offers high-performance, a simple interface, and the ability to be easily extended in order to support new monitoring systems.

What does Akka.Monitoring monitor?

Akka.Monitoring collects the following pieces of data about your applications:

  1. Actor lifecycle - actor starts / stops / restarts per second.
  2. Akka log events - errors, warnings, debug messages, and info messages.
  3. Messaging metrics - messages received, unhandled messages, and dead letters.
  4. Custom counters, gauges, and timers - if you want to know how long it takes an actor to process a message inside its mailbox, Akka.Monitoring can time it.

Akka.Monitoring automatically breaks out all of these metrics by actor system totals AND metrics for individual actor types. That way you can identify overly chatty or unresponsive actors without having to dig through extensive logs.

Supported monitoring systems

Currently Akka.Monitoring supports the following monitoring systems out of the box:

  1. StatsD - via a lightweight dependency on NStatsD.
  2. Microsoft AppInsights
  3. Performance Counters

If you don't have any monitoring systems configured to run with Akka.Monitoring, no problem - the extension will no-op all stat collection calls by default.

How you use it

First thing you want to do is install the Akka.Monitoring packages via NuGet:

Install-Package Akka.Monitoring

And then install a specific implementation

Install-Package Akka.Monitoring.StatsD
Install-Package Akka.Monitoring.ApplicationInsights
Install-Package Akka.Monitoring.PerformanceCounters

Register monitors with your ActorSystem

Once that's done, programmatically register your monitoring agent with the ActorMonitoringExtension object like this:

using Akka;
using Akka.Monitoring;
using Akka.Monitoring.StatsD;

_system = ActorSystem.Create("akka-performance-demo");
var registeredMonitor = ActorMonitoringExtension.RegisterMonitor(_system, new ActorStatsDMonitor()); //new ActorAppInsightsMonitor(), new ActorPerformanceCountersMonitor ();

This will automatically register the AkkaStatsDMonitor monitoring agent with your ActorSystem, and it will be available for use immediately.

Record metrics inside your actors

Now that you have your monitor registered, you can begin recording metrics - there are two ways of doing this:

Record metrics via the ActorContext extension methods You can record all of your metrics directly off of the Context object inside each of your actors, like this:

class HelloActor : TypedActor, IHandle<string>
{
    protected override void PreStart()
    {
        Context.IncrementActorCreated();
    }

    protected override void PostStop()
    {
        Context.IncrementActorStopped();
    }

    public void Handle(string message)
    {
        Context.IncrementMessagesReceived();
        Console.WriteLine("Received: {0}", message);
        if (message == "Goodbye")
        {
            Context.Self.Stop();
            Program.ManualResetEvent.Set(); //allow the program to exit
        }
        else
            Sender.Tell("Hello!");
    }
}

Alternatively, if you need to be able to record some custom metrics without the Context object, you can use the ActorMonitoringExtension object directly.

Record metrics via the ActorMonitoringExtension methods

using Akka;
using Akka.Monitoring;
using Akka.Monitoring.StatsD;

_system = ActorSystem.Create("akka-performance-demo");
var registeredMonitor = ActorMonitoringExtension.RegisterMonitor(_system, new ActorStatsDMonitor());
ActorMonitoringExtension.Monitors(_system).IncrementDebugsLogged();
Console.WriteLine("Logging debug...");

Metric capture methods

Both of these techniques expose the following methods available for capturing metrics:

  • IncrementActorsCreated
  • IncrementActorsRestarted
  • IncrementActorsStopped
  • IncrementMessagesReceived
  • IncrementUnhandledMessages - logged automatically
  • IncrementDeadLetters - logged automatically
  • IncrementErrorsLogged - logged automatically
  • IncrementWarningsLogged - logged automatically
  • IncrementDebugsLogged - logged automatically
  • IncrementInfosLogged - logged automatically
  • IncrementCounter - for custom counters.
  • Timing - for timing custom events, such as the amount of time needed to process a mailbox.
  • Gauge - for recording arbitrary non-counter metrics, such as the average size of a message.

All of these methods have extensive Intellisense documentation.

Using Performance Counters

Akka.Monitoring.PerformanceCounters allows to track all metrics in perfmon. Once ActorPerformanceCountersMonitor is registered via ActorMonitoringExtension it automatically creates Performance Counters Category named Akka which contains all metrics related with agents events. There is one performance counter created for every metric (built-in or custom). Also, for every actor type there is performance counter instance which allows to track metrics per single actor class:

alt performance counters selection

An example of perfmon chart displaying metrics from Akka.Monitoring.PerformanceCounters.Demo:

alt perfmon chart

Custom metrics registration

In order to create custom counters, timing metrics and gauges, you need to provide CustomMetrics with metric names to ActorPerformanceCountersMonitor constructor:

var registeredMonitor = ActorMonitoringExtension.RegisterMonitor(_system,
    new ActorPerformanceCountersMonitor(
        new CustomMetrics
        {
            Counters = { "akka.custom.metric1", "akka.custom.metric2" },
            Gauges = { "akka.messageboxsize"},
            Timers = { "akka.handlertime" }
        }));

Metric names will become performance counter names, so make sure they are unique among Akka Pefmormance Counters Category.

One gotcha to be aware of is that .NET performance counters cannot be added to an existing category. If you have to add counters to categories that already exist, the only way you can do so is to delete the category and re-create it with all of its contents. So if you find that you are adding custom metrics and nothing is appearing in Perfmon then you may need to delete the Akka performance counter category. Additionally, performance counters don't always show up straightaway so you may need to give it some time. To delete the Counter category you can remove the following key in the windows registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Akka\Performance (not recommended in production code!)

Extending Akka.Monitoring

Have a different monitoring system you want to use? It's easy to integrate into Akka.Monitoring - just implement your own subclass from AbstractActorMonitoringClient (source) and follow the registration steps above.

Building Akka.Monitoring

Akka.Monitoring uses the F# FAKE build system. To build our NuGet packages locally, just execute build.cmd locally.

FAQ

What happens if I call Akka.Monitoring without having any monitors registered?

Nothing - all calls are automatically no-oped if no monitors have been registered, so you're safe to release instrumented code even if you don't have monitoring configured yet.

What happens if I call Akka.Monitoring with multiple monitors registered?

Akka.Monitoring will automatically pipe its counter / timer / gauge updates to all attached monitoring systems, so you get that data everywhere.

Any plans to automatically collect actor lifecycle and message received data?

That depends largely on how much traction Akka.Monitoring gets - we're considering subclassing UntypedActor and TypedActor to automatically provide lifecycle and receive instrumentation, but we want to see how other people use it first.

How in the hell do I configure StatsD and Graphite?

These may help you:

License

See License for details.

More Repositories

1

akka-bootcamp

Self-paced training course to learn Akka.NET fundamentals from scratch
C#
1,035
star
2

akkadotnet-code-samples

Akka.NET professional reference code samples
C#
546
star
3

NBench

Performance benchmarking and testing framework for .NET applications 📈
C#
532
star
4

lighthouse

Lighthouse - a simple service discovery platform for Akka.Cluster (Akka.NET)
F#
175
star
5

akkadotnet-cluster-workshop

Akka.NET + Kubernetes + Akka.Cluster Training Course
C#
99
star
6

akkadotnet-helpers

Akka.NET helper classes to assist with common production needs
C#
73
star
7

Incrementalist

Git-based incremental build and testing platform for .NET and .NET Core.
C#
67
star
8

petabridge-dotnet-new

.NET CLI template for Petabridge-style projects
F#
67
star
9

TurboMqtt

The fastest Message Queue Telemetry Transport (MQTT) client for .NET.
C#
65
star
10

Cluster.WebCrawler

K8s, DevOps-ified version of the Akka.Cluster WebCrawler code sample
JavaScript
62
star
11

DrawTogether.NET

C#
53
star
12

Petabridge.Tracing.Zipkin

Professionally supported Zipkin + OpenTracing driver in C#
C#
33
star
13

Petabridge.Tracing.ApplicationInsights

OpenTracing adapter for Microsoft Application Insights
C#
28
star
14

akkadotnet-bootstrap

Akka.Remote and Akka.Cluster Bootstrapping Tools for Akka.NET
C#
28
star
15

faker-csharp

Faker for C#
C#
20
star
16

Petabridge.App.Web

ASP.NET Core + Akka.NET Integrated Template
C#
20
star
17

akkadotnet-healthcheck

Healthchecks for Akka.NET Applications 🏥
C#
20
star
18

Akka.Persistence.Extras

Akka.NET Persistence extras and ideas
C#
19
star
19

http-playback

Capture, store, and reuse live HTTP requests for QA and load testing.
C#
14
star
20

Akka.Persistence.Azure

Azure-powered Akka.Persistence for Akka.NET actors
C#
13
star
21

AkkaDotNet.LargeNetworkTests

Network stress tests designed to measure Akka.NET "background radiation" in large clusters (100+ nodes)
C#
13
star
22

azure-app-service-akkadotnet

Demo for Akka.NET on Azure App Service
C#
12
star
23

Petabridge.Collections

A set of specialized collections and data structures that probably should be in CoreFx
C#
11
star
24

petabridge.cmd-quickstart

Quickstart tutorial sample for Petabridge.Cmd
PowerShell
11
star
25

phobos-infrastructure

Infrastructure used for trying out Phobos® monitoring 👁️ and tracing 🎯
10
star
26

Petabridge.App

Akka.NET Template for Cluster Applications
C#
10
star
27

Petabridge.Phobos.Web

Phobos-enabled Akka.NET + ASP.NET Core application
PowerShell
8
star
28

azure-container-app-akkadotnet

C#
6
star
29

spark-installer-windows

Versioned Apache Spark installation scripts for Windows.
PowerShell
6
star
30

Petabridge.Library

Petabridge project template repository for libraries.
C#
5
star
31

kinesis-sample

Akka.NET Streams + Cluster + AWS Kinesis Sample
C#
5
star
32

Versionator

.NET library for programmatically generating assemblies and assembly versions
F#
4
star
33

akkadotnet-integration-samples

Samples for demonstrating how to integrate Akka.NET into other popular OSS and .NET technologies
Batchfile
4
star
34

Akka.Cluster.Sharding.RepairTool

An automated cleanup tool for purging Akka.Cluster.Sharding data from Akka.Persistence
C#
3
star
35

Akka.Cluster.Chunking

Akka.Remote transport adapter that incorporates Akka.Delivery to chunk large messages
C#
3
star
36

Phobos.Actor.Common

Common interfaces for use within Phobos applications
F#
3
star
37

phobos-dashboards

Shared Grafana / Datadog / Application Insights / etc dashboards for plotting Phobos Akka.NET metrics
3
star
38

phobos-samples

Sample projects for working with Phobos® and Akka.NET
C#
3
star
39

Petabridge.Phobos.Web.ApplicationInsights

Azure Application Insights + Phobos tutorial for monitoring and tracing Akka.NET Clusters + ASP.NET Core
PowerShell
3
star
40

Petabridge.Phobos.Web.InfluxDb

Phobos-enabled Akka.NET + ASP.NET Core application using InfluxDb
PowerShell
2
star
41

Petabridge.Phobos.Web.DataDog

DataDog + Phobos tutorial for monitoring and tracing Akka.NET Clusters + ASP.NET Core
PowerShell
2
star
42

phobos-issues

Public issues and bug tracker for Phobos®
2
star
43

petabridge.cmd-issues

Issue tracker for Petabridge.Cmd
2
star
44

Petabridge.Monitoring.PCF

Driver for the Metrics Forwarder in Pivotal Cloud Foundry (PCF)
C#
1
star
45

Phobos.Samples

Various samples for integrating Phobos, Akka.NET, and other technologies
C#
1
star
46

SocketLeakDetection

C#
1
star