• Stars
    star
    163
  • Rank 231,141 (Top 5 %)
  • Language
    Go
  • License
    MIT License
  • Created about 2 years ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

⚡️ Automatically add Trace Spans to Go methods and functions

⚡️ go-instrument

Automatically add Trace Spans to Go methods and functions

codecov Go Report Card Go Reference Mentioned in Awesome Go go-recipes OpenSSF Scorecard

This tool uses standard Go library to modify AST with instrumentation. You can add new instrumentations by defining your own Instrumenter and invoking Processor like it is done in main.

  • No dependencies
  • 500 LOC
  • OpenTelemetry (Datadog, NewRelic, etc.)
go install github.com/nikolaydubina/go-instrument@latest
find . -name "*.go" | xargs -I{} go-instrument -app my-service -w -filename {}

Functions and methods with ctx context.Context in arguments

func (s Cat) Name(ctx context.Context) (name string, err error) {
  ...

will be instrumented with span

func (s Cat) Name(ctx context.Context) (name string, err error) {
	ctx, span := otel.Trace("my-service").Start(ctx, "Cat.Name")
	defer span.End()
	defer func() {
		if err != nil {
			span.SetStatus(codes.Error, "error")
			span.RecordError(err)
		}
	}()
  ...

Example HTTP server go-instrument-example as it appears in Datadog.

Features

Excluding and Including

To avoid instrumentation of function add comment directive anywhere in the file.

//instrument:exclude SomeFunc|SomeOtherfunc|privateFunc
...

func (s Cat) Name(ctx context.Context) (name string, err error) {
  //instrument:exclude Name

To instrument only specific functions add comment directive anywhere in the file and pass -all=false in CLI.

//instrument:include SomeFunc|SomeOtherfunc|privateFunc
...

func (s Cat) Name(ctx context.Context) (name string, err error) {
  //instrument:include Name

Errors

Functions that have named return err error will get spans with appropriate status and error recorded.

func (s Cat) Walk(ctx context.Context) (err error) {
  ...

Comments

Comments are supported through patching source files bytes and fmt.

Go compiler directives

Standard Go compiler directives are recognized. More details go help buildconstraint and https://pkg.go.dev/cmd/go#hdr-Build_constraints.

  • //go:build exclude
  • // +build exclude
  • //go:build ignore
  • // +build ignore

In Development

  • Dynamic error variable name
  • Creating error when return is not named
  • Detection if function is already instrumented
  • Span Tags arguments
  • Span Tags returns
  • Assigning ctx to _ when ctx is not used in function (unused assignement linter checks issue)
  • Datadog native instrumenter

Motivation

It is laborious to add tracing code to every function manually. The code repeats 99% of time. Other languages can either modify code or have wrapper notations that makes even manual tracing much less laborious.

As of 2022-11-06, official Go does not support automatic function traces. https://go.dev/doc/diagnostics

Is there a way to automatically intercept each function call and create traces?

Go doesn’t provide a way to automatically intercept every function call and create trace spans. You need to manually instrument your code to create, end, and annotate spans.

Thus, providing automated version to add Trace Spans annotation.

Performance

Go Compiler Inlining

Since we are adding multiple functions calls, it affects Go compiler decisions on inlining. It is expected that Go will less likely inline.

For example, can inline function

$ go build -gcflags="-m -m" ./internal/testdata 2>&1 | grep OneLine
internal/testdata/basic.go:80:6: can inline OneLineTypical with cost 62 as: func(context.Context, int) (int, error) { return fib(n), nil }
go-instrument -w -filename internal/testdata/basic.go

Can not inline after instrumentation

$ go build -gcflags="-m -m" ./internal/testdata 2>&1 | grep OneLine
internal/testdata/basic.go:132:6: cannot inline OneLineTypical: unhandled op DEFER

Appendix A: Related Work

Appendix B: Other Languages

Java

Java runtime modifies bytecode of methods on load time that adds instrumentation calls. Pre-defined libraries are instrumented (http, mysql, etc).

✅ Very short single line decorator statement can be used to trace selected methods.

Datadog

import datadog.trace.api.Trace

public class BackupLedger {
  @Trace
  public void write(List<Transaction> transactions) {
    for (Transaction transaction : transactions) {
      ledger.put(transaction.getId(), transaction);
    }
  }
}

OpenTelemetry

import io.opentelemetry.instrumentation.annotations.WithSpan;

public class MyClass {
  @WithSpan
  public void myMethod() {
      <...>
  }
}

✅ Automatic instrumentation of all functions is also possible.

Datadog supports wildcard for list of methods to trace.

dd.trace.methods
Environment Variable: DD_TRACE_METHODS
Default: null
Example: package.ClassName[method1,method2,...];AnonymousClass$1[call];package.ClassName[]
List of class/interface and methods to trace. Similar to adding @Trace, but without changing code. Note: The wildcard method support ([
]) does not accommodate constructors, getters, setters, synthetic, toString, equals, hashcode, or finalizer method calls

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -Ddd.trace.methods="*" -jar path/to/application.jar

Python

Python monkeypatching of functions at runtime is used to add instrumentation calls. Pre-defined libraries are instrumented (http, mysql, etc).

✅ Very short single line decorator statement can be used to trace selected methods.

Datadog

from ddtrace import tracer

class BackupLedger:
    @tracer.wrap()
    def write(self, transactions):
        for transaction in transactions:
            self.ledger[transaction.id] = transaction

OpenTelemetry

@tracer.start_as_current_span("do_work")
def do_work():
    print("doing some work...")

⚠️ Automatic instrumentation of all functions is also possible via monkeypatching (fidning stable library is pending).

C++

❌ Only manual instrumentation.

Rust

✅ Very short single line decorator statement can be used to trace selected functions with well-establisehd tokio framework.

#[tracing::instrument]
pub fn shave(yak: usize) -> Result<(), Box<dyn Error + 'static>> {
#[instrument]
async fn write(stream: &mut TcpStream) -> io::Result<usize> {

Appendix C: Paths Not Taken

eBPF

With eBPF we can track latency, but we would not be able to assign errors to spans. Some platforms may not have access to eBPF.

Wrapping internal functions

Benefit of wrapping is to keep original code without modifications. However, manual step for switching would still be requied. Given every single function is duplciated and is within same package, code will quickly become messy and hard to maintain by user.

Wrapping exported functions

Typically, packages are failry big and performs lots of logic. Oftencase, business domains are split only in few large packages. Low level packages are already likely to be traced with standard tracing (MySQL, het/http, etc). Thus, it is doubtful how much benefit would be from tracing only exported functions and only on import.

Wrapping exported functions with separate package

This would lead to circular dependency failure, since some even exported functions in original package may be called withing same package. Thus, we would either skip those calls, or fail with circular dependency while trying to wrap those.

Appendix D: Generating Many Spans

1.97K spans, fibbonaci

3.7K spans, go cover treemap

Appendix E: Directives

Orignal version was using go:instrument directive. However, many members of Go community raised concern that it takes over reserved core Go toolchain directives (eg, //go:norace). Even though as of 2022-11-25 Go core does not use go:instrument, to respect community and Go core, leaving using //instrument: directive instead.

Appendix F: Selectors

One of proposed solutions for selectors was to use regex.

Specifically, first usecase proposed was to use

//instrument:exclude .*
//instrument:include ^API.*$

The issue with this is collision of two functions:

  • A) exlude all and select specific
  • B) include all and exclude specific

Similarly, there is collision of subspace of functions for exclusion and inclusion.

As of 2022-11-25, @nikolaydubina does not know how to resolve this better. Thus, keeping simple map matching wiht and condition of overlaps.

More Repositories

1

go-recipes

🦩 Tools for Go projects
Go
4,036
star
2

go-cover-treemap

🎄 Go code coverage to SVG treemap
Go
484
star
3

go-binsize-treemap

🔍 Go binary size SVG treemap
Go
444
star
4

calendarheatmap

📅 Calendar heatmap inspired by GitHub contribution activity
Go
393
star
5

llama2.go

LLaMA-2 in native Go
Go
187
star
6

treemap

🍬 Pretty Treemaps
Go
149
star
7

go-featureprocessing

🔥 Fast, simple sklearn-like feature processing for Go
Go
116
star
8

go-hackers-delight

"Hacker's Delight" in Go
Go
85
star
9

go-graph-layout

🔮 Graph Layout Algorithms in Go
Go
85
star
10

go-cover-treemap-web

Go
78
star
11

jsonl-graph

🏝 JSONL Graph Tools
Go
72
star
12

import-graph

Collect data about your dependencies
Go
40
star
13

twitter-remover

Remove twitter likes, posts, retweets, replies, followers
38
star
14

watchhttp

🌺 Run command periodically and expose latest STDOUT as HTTP endpoint
Go
32
star
15

go-ml-benchmarks

⏱ Benchmarks of machine learning inference for Go
Go
30
star
16

fpdecimal

🛫 Fixed-Point Decimals
Go
29
star
17

fpmoney

🧧 Fixed-Point Decimal Money
Go
25
star
18

validate

🥬 validate. simply.
Go
19
star
19

hq

🐁 happy little queue
Go
16
star
20

smrcptr

detect mixing pointer and value method receivers
Go
13
star
21

neuroscience-landscape

🌌 Resources on Neuroscience
12
star
22

vertfn

Go linter for Vertical Function Ordering
Go
12
star
23

go-enum-encoding

Generate Go enum encoding
Go
12
star
24

go-enum-example

Go Enum: benchmarks, examples, analysis
Go
8
star
25

htmljson

🫐 Rich rendering of JSON as HTML in Go
Go
7
star
26

multiline-jsonl

Read and write multiline JSONL in Go
Go
6
star
27

openapi-inline-examples

🌏 Inline OpenAPI JSON examples from filenames
Go
6
star
28

htmlyaml

🐹 render YAML as HTML in Go
Go
4
star
29

go-commentage

How far Go comments drifting behind?
Go
4
star
30

rchan

Go channel through Redis List
Go
4
star
31

mini-awesome-cv

📝 LaTeX Awesome-CV under 200LOC
TeX
4
star
32

go-instrument-example

Go
3
star
33

svgpan

Pan and Zoom of SVG in your Go front-end app in browser.
Go
3
star
34

go-bench-errors

Benchmarking Go errors
Go
3
star
35

mdpage

one-pager Markdown list from YAML
Go
3
star
36

consistentimports

Detect inconsistent import aliases
Go
2
star
37

go-callsite-stats

analyse function callsites
Go
2
star
38

read-seek-peeker

Go Reader that can Seek() and Peek()
Go
2
star
39

go-bench-stream

🌊 Go Benchmarks for Stream Processing
Go
2
star
40

go-bitset-example

Go Bitset: benchmarks, examples, analysis
Go
1
star
41

aws-s3-reader

Efficient Go Reader for large AWS S3 Objects
Go
1
star
42

go-bench-receiver

Which is more efficient value or pointer method receivers in Go?
Go
1
star
43

totp

TOTP (RFC-6238) and HOTP (RFC-4226)
Go
1
star
44

dotfiles

macOS, Linux
Vim Script
1
star
45

presentations

1
star
46

PINTOS

C
1
star
47

ARIA

C++
1
star
48

mini-blog

JavaScript
1
star
49

graph-tools-gallery

Inspiration gallery of graph tools
1
star