• Stars
    star
    120
  • Rank 295,983 (Top 6 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created over 7 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

OpenTracing Instrumentation for Apache Kafka Client

Build Status Coverage Status Released Version Apache-2.0 license

OpenTracing Apache Kafka Client Instrumentation

OpenTracing instrumentation for Apache Kafka Client.
Two solutions are provided:

  1. Based on decorated Producer and Consumer
  2. Based on Interceptors

Requirements

  • Java 8
  • Kafka 2.2.0

Installation

Kafka Client

pom.xml

<dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>opentracing-kafka-client</artifactId>
    <version>VERSION</version>
</dependency>

Kafka Streams

pom.xml

<dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>opentracing-kafka-streams</artifactId>
    <version>VERSION</version>
</dependency>

Spring Kafka

pom.xml

<dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>opentracing-kafka-spring</artifactId>
    <version>VERSION</version>
</dependency>

Usage

// Instantiate tracer
Tracer tracer = ...

// Optionally register tracer with GlobalTracer
GlobalTracer.register(tracer);

Kafka Client

Decorators based solution

// Instantiate KafkaProducer
KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps);

//Decorate KafkaProducer with TracingKafkaProducer
TracingKafkaProducer<Integer, String> tracingProducer = new TracingKafkaProducer<>(producer, 
        tracer);

// Send
tracingProducer.send(...);

// Instantiate KafkaConsumer
KafkaConsumer<Integer, String> consumer = new KafkaConsumer<>(consumerProps);

// Decorate KafkaConsumer with TracingKafkaConsumer
TracingKafkaConsumer<Integer, String> tracingConsumer = new TracingKafkaConsumer<>(consumer, 
        tracer);

//Subscribe
tracingConsumer.subscribe(Collections.singletonList("messages"));

// Get records
ConsumerRecords<Integer, String> records = tracingConsumer.poll(1000);

// To retrieve SpanContext from polled record (Consumer side)
ConsumerRecord<Integer, String> record = ...
SpanContext spanContext = TracingKafkaUtils.extractSpanContext(record.headers(), tracer);
Custom Span Names for Decorators based solution

The decorator-based solution includes support for custom span names by passing in a BiFunction object as an additional argument to the TracingKafkaConsumer or TracingKafkaProducer constructors, either one of the provided BiFunctions or your own custom one.

// Create BiFunction for the KafkaProducer that operates on
// (String operationName, ProducerRecord consumerRecord) and
// returns a String to be used as the name
BiFunction<String, ProducerRecord, String> producerSpanNameProvider =
    (operationName, producerRecord) -> "CUSTOM_PRODUCER_NAME";

// Instantiate KafkaProducer
KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps);

//Decorate KafkaProducer with TracingKafkaProducer
TracingKafkaProducer<Integer, String> tracingProducer = new TracingKafkaProducer<>(producer, 
        tracer,
        producerSpanNameProvider);
// Spans created by the tracingProducer will now have "CUSTOM_PRODUCER_NAME" as the span name.


// Create BiFunction for the KafkaConsumer that operates on
// (String operationName, ConsumerRecord consumerRecord) and
// returns a String to be used as the name
BiFunction<String, ConsumerRecord, String> consumerSpanNameProvider =
    (operationName, consumerRecord) -> operationName.toUpperCase();
// Instantiate KafkaConsumer
KafkaConsumer<Integer, String> consumer = new KafkaConsumer<>(consumerProps);
// Decorate KafkaConsumer with TracingKafkaConsumer, passing in the consumerSpanNameProvider BiFunction
TracingKafkaConsumer<Integer, String> tracingConsumer = new TracingKafkaConsumer<>(consumer, 
        tracer,
        consumerSpanNameProvider);
// Spans created by the tracingConsumer will now have the capitalized operation name as the span name.
// "receive" -> "RECEIVE"

Interceptors based solution

// Register tracer with GlobalTracer:
GlobalTracer.register(tracer);

// Add TracingProducerInterceptor to sender properties:
senderProps.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, 
          TracingProducerInterceptor.class.getName());

// Instantiate KafkaProducer
KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps);

// Send
producer.send(...);

// Add TracingConsumerInterceptor to consumer properties:
consumerProps.put(ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG,
          TracingConsumerInterceptor.class.getName());

// Instantiate KafkaConsumer
KafkaConsumer<Integer, String> consumer = new KafkaConsumer<>(consumerProps);

//Subscribe
consumer.subscribe(Collections.singletonList("messages"));

// Get records
ConsumerRecords<Integer, String> records = consumer.poll(1000);

// To retrieve SpanContext from polled record (Consumer side)
ConsumerRecord<Integer, String> record = ...
SpanContext spanContext = TracingKafkaUtils.extractSpanContext(record.headers(), tracer);

Kafka Streams

// Instantiate TracingKafkaClientSupplier
KafkaClientSupplier supplier = new TracingKafkaClientSupplier(tracer);

// Provide supplier to KafkaStreams
KafkaStreams streams = new KafkaStreams(builder.build(), new StreamsConfig(config), supplier);
streams.start();

Spring Kafka

// Declare Tracer bean
@Bean
public Tracer tracer() {
  return ...
}


// Decorate ConsumerFactory with TracingConsumerFactory
@Bean
public ConsumerFactory<Integer, String> consumerFactory() {
  return new TracingConsumerFactory<>(new DefaultKafkaConsumerFactory<>(consumerProps()), tracer());
}

// Decorate ProducerFactory with TracingProducerFactory
@Bean
public ProducerFactory<Integer, String> producerFactory() {
  return new TracingProducerFactory<>(new DefaultKafkaProducerFactory<>(producerProps()), tracer());
}

// Use decorated ProducerFactory in KafkaTemplate 
@Bean
public KafkaTemplate<Integer, String> kafkaTemplate() {
  return new KafkaTemplate<>(producerFactory());
}

// Use an aspect to decorate @KafkaListeners
@Bean
public TracingKafkaAspect tracingKafkaAspect() {
  return new TracingKafkaAspect(tracer());
}
Custom Span Names for Spring Kafka

The Spring Kafka factory implementations include support for custom span names by passing in a BiFunction object as an additional argument to the TracingConsumerFactory or TracingProducerFactory constructors, either one of the provided BiFunctions or your own custom one.

// Create BiFunction for the KafkaProducerFactory that operates on
// (String operationName, ProducerRecord consumerRecord) and
// returns a String to be used as the name
BiFunction<String, ProducerRecord, String> producerSpanNameProvider =
    (operationName, producerRecord) -> "CUSTOM_PRODUCER_NAME";

// Decorate ProducerFactory with TracingProducerFactory
@Bean
public ProducerFactory<Integer, String> producerFactory() {
  return new TracingProducerFactory<>(new DefaultKafkaProducerFactory<>(producerProps()), tracer());
}
// Spans created by the tracingProducer will now have "CUSTOM_PRODUCER_NAME" as the span name.


// Create BiFunction for the KafkaConsumerFactory that operates on
// (String operationName, ConsumerRecord consumerRecord) and
// returns a String to be used as the name
BiFunction<String, ConsumerRecord, String> consumerSpanNameProvider =
    (operationName, consumerRecord) -> operationName.toUpperCase();

// Decorate ConsumerFactory with TracingConsumerFactory
@Bean
public ConsumerFactory<Integer, String> consumerFactory() {
  return new TracingConsumerFactory<>(new DefaultKafkaConsumerFactory<>(consumerProps()), tracer());
}
// Consumers produced by the traced consumerFactory

Pre-made Span Name Providers

The following BiFunctions are already included in the ClientSpanNameProvider class, with CONSUMER_OPERATION_NAME and PRODUCER_OPERATION_NAME being the default should no spanNameProvider be provided:

  • CONSUMER_OPERATION_NAME and PRODUCER_OPERATION_NAME : Returns the operationName as the span name ("receive" for Consumer, "send" for producer).
  • CONSUMER_PREFIXED_OPERATION_NAME(String prefix) and PRODUCER_PREFIXED_OPERATION_NAME(String prefix) : Returns a String concatenation of prefix and operatioName.
  • CONSUMER_TOPIC and PRODUCER_TOPIC : Returns the Kafka topic name that the record was pushed to/pulled from (record.topic()).
  • PREFIXED_CONSUMER_TOPIC(String prefix) and PREFIXED_PRODUCER_TOPIC(String prefix) : Returns a String concatenation of prefix and the Kafka topic name (record.topic()).
  • CONSUMER_OPERATION_NAME_TOPIC and PRODUCER_OPERATION_NAME_TOPIC : Returns "operationName - record.topic()".
  • CONSUMER_PREFIXED_OPERATION_NAME_TOPIC(String prefix) and PRODUCER_PREFIXED_OPERATION_NAME_TOPIC(String prefix) : Returns a String concatenation of prefix and "operationName - record.topic()".

License

Apache 2.0 License.

More Repositories

1

opentracing-specification-zh

OpenTracingζ ‡ε‡†οΌˆδΈ­ζ–‡η‰ˆοΌ‰ `zh` (Chinese) translation of the opentracing/specification
929
star
2

nginx-opentracing

NGINX plugin for OpenTracing
C++
499
star
3

java-spring-cloud

Distributed tracing for Spring Boot, Cloud and other Spring projects
Java
386
star
4

csharp-netcore

OpenTracing instrumentation for .NET Core 3.1 & .NET 6+ apps
C#
261
star
5

java-spring-jaeger

Java
243
star
6

go-stdlib

OpenTracing instrumentation for packages in the Go stdlib
Go
217
star
7

java-specialagent

Automatic instrumentation for 3rd-party libraries in Java applications with OpenTracing.
Java
185
star
8

python-flask

OpenTracing instrumentation for the Flask microframework
Python
135
star
9

java-spring-web

OpenTracing Spring Web instrumentation
Java
106
star
10

java-opentracing-walkthrough

A self-guided OpenTracing walkthrough / demo project
JavaScript
106
star
11

python-django

OpenTracing instrumentation for the Django framework
Python
105
star
12

java-jdbc

OpenTracing Instrumentation for JDBC
Java
81
star
13

java-jfr-tracer

This is a delegating tracer to be used with OpenTracing. It records span information into the JFR, allowing very deep tracing.
Java
77
star
14

go-grpc

Package otgrpc provides OpenTracing support for any gRPC client or server.
Go
71
star
15

go-gin

OpenTracing middleware for gin-gonic
Go
64
star
16

python-grpc

Python
53
star
17

java-grpc

OpenTracing Instrumentation for gRPC
Java
52
star
18

csharp-grpc

OpenTracing Instrumentation for gRPC
C#
43
star
19

java-agent

Agent-based OpenTracing instrumentation in Java
Java
40
star
20

javascript-express

OpenTracing middleware for express
JavaScript
39
star
21

java-jaxrs

OpenTracing Java JAX-RS instrumentation
Java
37
star
22

java-concurrent

OpenTracing-aware helpers related to java.util.concurrent
Java
36
star
23

meta

A meta-repository for OpenTracing contributions
34
star
24

java-redis-client

OpenTracing Instrumentation for Redis Client
Java
33
star
25

go-zap

Integration with go.uber.org/zap
Go
33
star
26

java-metrics

Java
31
star
27

go-amqp

AMQP instrumentation in Go
Go
29
star
28

scala-concurrent

OpenTracing instrumentation for scala.concurrent package
Scala
24
star
29

opentracing-erlang

Open Tracing Toolkit for ERlang
Erlang
24
star
30

java-web-servlet-filter

OpenTracing Java Web Servlet Filter Instrumentation
Java
23
star
31

java-xray-tracer

Java OpenTracing implementation backed by AWS X-Ray
Java
23
star
32

java-okhttp

OpenTracing Okhttp client instrumentation
Java
23
star
33

java-vertx-web

OpenTracing instrumentation for Vert.x web package
Java
21
star
34

python-redis

OpenTracing instrumentation for the Redis client.
Python
20
star
35

ruby-rack-tracer

Ruby
20
star
36

java-span-reporter

A Tracer implementation that writes all instrumented data to a conventional Logger
Java
20
star
37

goredis

a middleware for go-redis/redis to use opentracing
Go
18
star
38

java-spring-rabbitmq

OpenTracing RabbitMQ instrumentation
Java
18
star
39

java-spring-zipkin

Java
16
star
40

python-sqlalchemy

OpenTracing instrumentation for SQLAlchemy
Python
16
star
41

scala-akka

OpenTracing instrumentation for Scala Akka
Scala
16
star
42

java-dropwizard

(deprecated) OpenTracing instrumentation for the Dropwizard framework
Java
15
star
43

java-aws-sdk

OpenTracing instrumentation for AWS SDK
Java
15
star
44

java-specialagent-demo

A SpecialAgent Demo
Java
13
star
45

java-mongo-driver

OpenTracing Instrumentation for Mongo Driver
Java
12
star
46

java-reactor

OpenTracing instrumentation for Reactor
Java
12
star
47

javascript-tracedpromise

JavaScript
12
star
48

python-elasticsearch

OpenTracing instrumentation for the the Python's Elasticsearch clients
Python
11
star
49

java-apache-httpclient

OpenTracing Apache HttpClient instrumentation
Java
11
star
50

python-requests

OpenTracing instrumentation for Requests
Python
10
star
51

echo

a middleware for the echov4 web framework to use opentracing
Go
10
star
52

go-aws-sdk

OpenTracing support for AWS SDK in Go
Go
10
star
53

java-thrift

OpenTracing instrumentation for Apache Thrift
Java
10
star
54

java-rabbitmq-client

OpenTracing Instrumentation for RabbitMQ Client
Java
9
star
55

python-tornado

OpenTracing instrumentation for Tornado
Python
9
star
56

java-rxjava

OpenTracing Instrumentation for RxJava
Java
9
star
57

examples

Examples (some trivial) for opentracing.io
Go
9
star
58

java-elasticsearch-client

OpenTracing Instrumentation for Elasticsearch Client
Java
8
star
59

go-observer

an Observer API for OpenTracing-Go Tracers
Go
8
star
60

java-examples

tested examples of common instrumentation patterns
Java
7
star
61

ruby-faraday-tracer

Ruby
7
star
62

java-tracerresolver

Resolver API for OpenTracing Tracer implementations.
Java
6
star
63

java-akka

Java
6
star
64

go-gorilla

OpenTracing instrumentation for Gorilla framework (github.com/gorilla)
Go
6
star
65

java-spanmanager

Current span management for Java
Java
6
star
66

java-jms

OpenTracing Instrumentation for JMS API
Java
5
star
67

csharp-decorators

C#
5
star
68

perfevents

Perf metrics library for go and java applications
Go
5
star
69

beego

a middleware for the beego web framework to use opentracing
Go
5
star
70

java-hprose

OpenTracing instrumentation for the Hprose Remote Object Service Engine
Java
4
star
71

python-pyramid

OpenTracing instrumentation for the Pyramid framework
Python
4
star
72

java-cassandra-driver

OpenTracing instrumentation for Cassandra Driver
Java
4
star
73

java-globaltracer

(deprecated) Global OpenTracing Tracer resolution for Java
Java
4
star
74

java-asynchttpclient

OpenTracing Instrumentation for https://github.com/AsyncHttpClient/async-http-client
Java
4
star
75

cpp-grpc

OpenTracing Instrumentation for gRPC
4
star
76

python-cassandra

OpenTracing instrumentation for cassandra-driver
Python
3
star
77

java-p6spy

OpenTracing Instrumentation for P6Spy
Java
3
star
78

java-redisson

OpenTracing Instrumentation for Redisson
Java
3
star
79

java-spring-messaging

OpenTracing Spring Messaging instrumentation
Java
3
star
80

java-jdbi

OpenTracing instrumentation for the JDBI database framework
Java
3
star
81

java-benchmarks

Set of benchmarks to assess the performance of different OpenTracing components and/or libraries
Java
3
star
82

python-examples

tester examples of common instrumentation patterns
Python
3
star
83

java-spring-tracer-configuration

Java
2
star
84

python-pymongo

OpenTracing instrumentation for PyMongo
Python
2
star
85

java-cdi

Java
2
star
86

go-restful

OpenTracing instrumentation for the go-restful framework
Go
2
star
87

java-neo4j-driver

OpenTracing instrumentation for Neo4j Driver
Java
2
star
88

java-ejb

Java
2
star
89

java-solr-client

OpenTracing Instrumentation for Solr Client
Java
2
star
90

python-gevent

OpenTracing instrumentation for gevent
Python
2
star
91

java-grizzly-http-server

OpenTracing instrumentation for Grizzly HttpServer
Java
1
star
92

ruby-redis-instrumentation

Ruby
1
star
93

java-grizzly-ahc

OpenTracing instrumentation for Grizzly AsyncHttpClient
Java
1
star
94

java-memcached-client

OpenTracing Instrumentation for Memcached Client
Java
1
star
95

ruby-mongodb-instrumentation

Ruby
1
star
96

java-opentelemetry-bridge

OpenTracing-OpenTelemetry Tracer Bridge
Java
1
star
97

java-common

Common utilities for OpenTracing Instrumentation Plugins.
Java
1
star
98

java-hazelcast

OpenTracing Instrumentation for Hazelcast
Java
1
star
99

java-api-extensions

This repository contains API extensions for use with the core Tracer and Span APIs.
Java
1
star
100

java-opentracing-jaeger-bundle

Java
1
star