• Stars
    star
    320
  • Rank 128,718 (Top 3 %)
  • Language
    Scala
  • License
    Apache License 2.0
  • Created over 7 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Kafka-based message queue

Kafka Message Queue

Join the chat at https://gitter.im/softwaremill/kmq Maven Central

Using kmq you can acknowledge processing of individual messages in Kafka, and have unacknowledged messages re-delivered after a timeout.

This is in contrast to the usual Kafka offset-committing mechanism, using which you can acknowledge all messages up to a given offset only.

If you are familiar with Amazon SQS, kmq implements a similar message processing model.

How does this work?

For a more in-depth overview see the blog: Using Kafka as a message queue, and for performance benchmarks: Kafka with selective acknowledgments (kmq) performance & latency benchmark

The acknowledgment mechanism uses a marker topic, which should have the same number of partitions as the "main" data topic (called the queue topic). The marker topic is used to track which messages have been processed, by writing start/end markers for every message.

message flow diagram

Using kmq

An application using kmq should consist of the following components:

  • a number of RedeliveryTrackers. This components consumes the marker topic and redelivers messages if appropriate. Multiple copies should be started in a cluster for fail-over. Uses automatic partition assignment.
  • components which send data to the queue topic to be processed
  • queue clients, either custom or using the KmqClient

Maven/SBT dependency

SBT:

"com.softwaremill.kmq" %% "core" % "0.3.1"

Maven:

<dependency>
    <groupId>com.softwaremill.kmq</groupId>
    <artifactId>core_2.13</artifactId>
    <version>0.3.1</version>
</dependency>

Note: The supported Scala versions are: 2.12, 2.13.

Client flow

The flow of processing a message is as follows:

  1. read messages from the queue topic, in batches
  2. write a start marker to the markers topic for each message, wait until the markers are written
  3. commit the biggest message offset to the queue topic
  4. process messages
  5. for each message, write an end marker. No need to wait until the markers are written.

This ensures at-least-once processing of each message. Note that the acknowledgment of each message (writing the end marker) can be done for each message separately, out-of-order, from a different thread, server or application.

Example code

There are three example applications:

  • example-java/embedded: a single java application that starts all three components (sender, client, redelivery tracker)
  • example-java/standalone: three separate runnable classes to start the different components
  • example-scala: an implementation of the client using reactive-kafka

Time & timestamps

How time is handled is crucial for message redelivery, as messages are redelivered after a given amount of time passes since the start marker was sent.

To track what was sent when, kmq uses Kafka's message timestamp. By default, this is messages create time (message.timestamp.type=CreateTime), but for the markers topic, it is advisable to switch this to LogAppendTime. That way, the timestamps more closely reflect when the markers are really written to the log, and are guaranteed to be monotonic in each partition (which is important for redelivery - see below).

To calculate which messages should be redelivered, we need to know the value of "now", to check which start markers have been sent later than the configured timeout. When a marker has been received from a partition recently, the maximum such timestamp is used as the value of "now" - as it indicates exactly how far we are in processing the partition. What "recently" means depends on the useNowForRedeliverDespiteNoMarkerSeenForMs config setting. Otherwise, the current system time is used, as we assume that all markers from the partition have been processed.

Dead letter queue (DMQ)

The redelivery of the message is attempted only a configured number of times. By default, it's 3. You can change that number by setting maxRedeliveryCount value in KmqConfig. After that number is exceeded messages will be forwarded to a topic working as a dead letter queue. By default, the name of that topic is name of the message topic concatenated with the suffix __undelivered. You can configure the name by setting deadLetterTopic in KmqConfig. The number of redeliveries is tracked by kmq with a special header. The default the name of that header is kmq-redelivery-count. You can change it by setting redeliveryCountHeader in KmqConfig.

More Repositories

1

elasticmq

In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.
Scala
2,379
star
2

sttp

The Scala HTTP client you always wanted!
Scala
1,418
star
3

tapir

Declarative, type-safe web endpoints library
Scala
1,294
star
4

macwire

Lightweight and Nonintrusive Scala Dependency Injection Library
Scala
1,258
star
5

quicklens

Modify deeply nested case class fields
Scala
815
star
6

magnolia

Easy, fast, transparent generic derivation of typeclass instances
Scala
755
star
7

bootzooka

Simple project to quickly start developing a Scala-based microservice or web application, without the need to write login, user registration etc.
Scala
695
star
8

codebrag

Your daily code review tool
Scala
651
star
9

akka-http-session

Web & mobile client-side akka-http sessions, with optional JWT support
Scala
440
star
10

it-cfp-list

List of Call For Papers for IT conferences
374
star
11

retry

because you should never give up, at least not on the first try
Scala
347
star
12

diffx

Pretty diffs for scala case classes
Scala
343
star
13

scala-clippy

Good advice for Scala compiler errors
Scala
315
star
14

ox

Safe direct style concurrency and resiliency for Scala on the JVM
Scala
305
star
15

supler

Rapid Form Development library. Use your favourite JS frontend & Scala backend frameworks.
Scala
286
star
16

jox

Fast and Scalable Channels in Java
Java
149
star
17

mqperf

Scala
143
star
18

scala-common

Tiny independent libraries with a single purpose, often a single class
Scala
120
star
19

slick-eventsourcing

Example for "Entry level event-sourcing" blog
Scala
118
star
20

lemon-dataset

Lemons quality control dataset
99
star
21

maven-badges

A node.js implementation of https://github.com/jirutka/maven-badges, originally created in ruby.
TypeScript
85
star
22

sbt-softwaremill

A sane set of default build settings
Scala
72
star
23

akka-vs-scalaz

Scala
63
star
24

recursion-training

Recursion schemes training examples and exercises
HTML
59
star
25

livestub

The HTTP server stub you always wanted!
Scala
51
star
26

scala-sql-compare

Scala
50
star
27

stringmask

A micro-library for macro-based case class field masking in .toString
Scala
48
star
28

scala-id-generator

Scala
48
star
29

confluent-playground

Java
44
star
30

odelay

delayed reactions
Scala
42
star
31

sttp-model

Simple Scala HTTP model
Scala
42
star
32

saft

Scala
41
star
33

akka-simple-cluster-k8s

Scala
39
star
34

softwaremill-common

SoftwareMill Common library
Java
37
star
35

walk-that-type

A tool for evaluating TypeScript types step by step.
TypeScript
36
star
36

sttp-openai

Scala
34
star
37

FoXAI

The open-source library for explainable AI. Generic and easy to integrate with PyTorch.
Python
30
star
38

neme-plugin

Scala compiler plugin for turning non exhaustive match warnings into errors
Scala
29
star
39

scala-pre-commit-hooks

Pre-commit/Pre-push hooks for Scala
Python
29
star
40

reactive-event-sourcing-java

Java
28
star
41

zio2-structure

Scala
26
star
42

helisa

Scala API for jenetics
Scala
26
star
43

streams-tests

Scala
25
star
44

tapir-loom

Scala
24
star
45

node-typescript-starter

A basic boilerplate for node + TypeScript development with debugger source maps support.
TypeScript
22
star
46

free-tagless-compare

Free monads compared to tagless final
Scala
22
star
47

akka-http-session-faq

Java
21
star
48

sttp-apispec

OpenAPI, AsyncAPI and JSON Schema Scala models.
Scala
21
star
49

activator-reactive-kafka-scala

Activator template for Reactive Kafka
Scala
20
star
50

scala3-macro-debug

Scala
17
star
51

reactive-streams-for-java-developers

Java
17
star
52

meerkat

Observability Starter Kit for JVM Applications
JavaScript
17
star
53

resilience4s

Scala
16
star
54

react-use-promise-matcher

React hooks allowing you to handle promises in a stateful way
TypeScript
16
star
55

adopt-tapir

A quickstart generator for Tapir projects
Scala
15
star
56

simple-http-server

Simple JVM based HTTP server with no dependencies
Scala
15
star
57

correlator

Scala
15
star
58

detectnet-tests

Python scripts and other resources for tesing DetectNet on Nvidia DIGITS
Python
14
star
59

blockchain-schedule

An experimental collaborative planning app based on Ethereum ("Decentralized Doodle")
TypeScript
14
star
60

blog-scala-structure-lifecycle

Scala
12
star
61

akka-sandbox

Training ground for experiments with Akka framework.
Scala
12
star
62

broadway-pipelines-blog

Constructing effective data processing workflows using Elixir and Broadway
Elixir
12
star
63

undelay

Satisfy Scala Futures quickly
Scala
11
star
64

monix-correlation-id

Scala
10
star
65

cassandra-monitoring

Scripts for the Cassandra Monitoring blog miniseries
10
star
66

reason-companies-example

Reason example application
OCaml
10
star
67

jvmbot

Scala
9
star
68

botarium

A simple starter kit for building bots using Node + TypeScript + BotKit.
TypeScript
8
star
69

sbt-template

Scala
8
star
70

asamal

POC for a CDI-based web lightweight framework
Java
8
star
71

boot-scala-microservice

Bootstrap microservice template that uses micro-deps library https://github.com/4finance/micro-deps
Scala
8
star
72

sttp-shared

Scala
7
star
73

modem-connector

Modulator and Demodulator for HAM Radio AX.25 audio signals
Scala
7
star
74

gatling-zeromq

A Gatling stress test plugin for ZeroMQ protocol
Scala
5
star
75

trqbox-demo

Ruby
5
star
76

idea-pastie-plugin

Plugin to post pastie.org pasties from IntelliJ Idea
Java
5
star
77

sentinel-cgan

Sentinel generative conditional adversarial network implementation
Python
5
star
78

scalatimes

Pug
5
star
79

scala-compiler-plugin-template

Scala
5
star
80

tapir-serverless

Scala
5
star
81

OtterJet

Java
5
star
82

slack-alphabet

Scala
4
star
83

scalar-conf-website

Scalar - Scala Conference in Central Europe
Python
4
star
84

try-them-off

Showcase service presenting possible usage of the Try monad from Vavr.
Java
4
star
85

sttp-openapi-example

Scala
4
star
86

cache-get-or-create

Java
4
star
87

bootzooka-react

Simple project to quickly start developing a web application using React and Akka HTTP, without the need to write login, user registration etc. https://softwaremill.com/open-source/
Scala
4
star
88

vehicle-routing-problem-java

Java
4
star
89

fabrica

Shell
3
star
90

akka-typed-workshop

Scala
3
star
91

kuberenetes-fundamentals

Training projects to explore k8s features
Scala
3
star
92

play-scala-slick-example-part2

Scala
3
star
93

ansible-bigbluebutton

Shell
3
star
94

kleisli-example

Scala
3
star
95

loom-protect

Java
3
star
96

supler-example

Example project for Supler http://supler.io
JavaScript
2
star
97

jekyll-softwaremill

SoftwareMill.com website written in Jekyll
PHP
2
star
98

demo-spring-boot-docker-compose

Demo application with comparison of spring-boot-docker-compose to testcontainers
Java
2
star
99

oauth_tutorial

Phoenix OAuth tutorial
Elixir
2
star
100

terraform-gke-bootstrap

HCL
2
star