• Stars
    star
    238
  • Rank 169,306 (Top 4 %)
  • Language
    Ruby
  • License
    Apache License 2.0
  • Created about 7 years ago
  • Updated 10 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A simple way to publish messages to Kafka from Ruby applications

DeliveryBoy

This library provides a dead easy way to start publishing messages to a Kafka cluster from your Ruby or Rails application!

Installation

Add this line to your application's Gemfile:

gem 'delivery_boy'

And then execute:

$ bundle

Or install it yourself as:

$ gem install delivery_boy

Usage

Once you've installed the gem, and assuming your Kafka broker is running on localhost, you can simply start publishing messages to Kafka directly from your Rails code:

# app/controllers/comments_controller.rb
class CommentsController < ApplicationController
  def create
    @comment = Comment.create!(params)

    # This will publish a JSON representation of the comment to the `comments` topic
    # in Kafka. Make sure to create the topic first, or this may fail.
    DeliveryBoy.deliver(comment.to_json, topic: "comments")
  end
end

The above example will block the server process until the message has been delivered. If you want deliveries to happen in the background in order to free up your server processes more quickly, call #deliver_async instead:

# app/controllers/comments_controller.rb
class CommentsController < ApplicationController
  def show
    @comment = Comment.find(params[:id])

    event = {
      name: "comment_viewed",
      data: {
        comment_id: @comment.id,
        user_id: current_user.id
      }
    }

    # By delivering messages asynchronously you free up your server processes faster.
    DeliveryBoy.deliver_async(event.to_json, topic: "activity")
  end
end

In addition to improving response time, delivering messages asynchronously also protects your application against Kafka availability issues -- if messages cannot be delivered, they'll be buffered for later and retried automatically.

A third method is to produce messages first (without delivering the messages to Kafka yet), and deliver them synchronously later.

 # app/controllers/comments_controller.rb
 class CommentsController < ApplicationController
   def create
     @comment = Comment.create!(params)

     event = {
       name: "comment_created",
       data: {
         comment_id: @comment.id
         user_id: current_user.id
       }
     }

     # This will queue the two messages in the internal buffer.
     DeliveryBoy.produce(comment.to_json, topic: "comments")
     DeliveryBoy.produce(event.to_json, topic: "activity")

     # This will deliver all messages in the buffer to Kafka.
     # This call is blocking.
     DeliveryBoy.deliver_messages
   end
 end

The methods deliver, deliver_async and produce take the following options:

  • topic – the Kafka topic that should be written to (required).
  • key – the key that should be set on the Kafka message (optional).
  • partition – a specific partition number that should be written to (optional).
  • partition_key – a string that can be used to deterministically select the partition that should be written to (optional).

Regarding partition and partition_key: if none are specified, DeliveryBoy will pick a partition at random. If you want to ensure that e.g. all messages related to a user always get written to the same partition, you can pass the user id to partition_key. Don't use partition directly unless you know what you're doing, since it requires you to know exactly how many partitions each topic has, which can change and cause you pain and misery. Just use partition_key or let DeliveryBoy choose at random.

Configuration

You configure DeliveryBoy in three different ways: in a YAML config file, in a Ruby config file, or by setting environment variables.

If you're using Rails, the fastest way to get started is to execute the following in your terminal:

$ bundle exec rails generate delivery_boy:install

This will create a config file at config/delivery_boy.yml with configurations for each of your Rails environments. Open that file in order to make changes.

Note that for all configuration variables, you can pass in an environment variable. These environment variables all take the form DELIVERY_BOY_X, where X is the upper-case configuration variable name, e.g. DELIVERY_BOY_CLIENT_ID.

You can also configure DeliveryBoy in Ruby if you prefer that. By default, the file config/delivery_boy.rb is loaded if present, but you can do this from anywhere – just call DeliveryBoy.configure like so:

DeliveryBoy.configure do |config|
  config.client_id = "yolo"
  # ...
end

The following configuration variables can be set:

Basic

brokers

A list of Kafka brokers that should be used to initialize the client. Defaults to just localhost:9092 in development and test, but in production you need to pass a list of hostname:port strings.

client_id

This is how the client will identify itself to the Kafka brokers. Default is delivery_boy.

log_level

The log level for the logger.

Message delivery

delivery_interval

The number of seconds between background message deliveries. Default is 10 seconds. Disable timer-based background deliveries by setting this to 0.

delivery_threshold

The number of buffered messages that will trigger a background message delivery. Default is 100 messages. Disable buffer size based background deliveries by setting this to 0.

required_acks

The number of Kafka replicas that must acknowledge messages before they're considered as successfully written. Default is all replicas.

See ruby-kafka for more information.

ack_timeout

A timeout executed by a broker when the client is sending messages to it. It defines the number of seconds the broker should wait for replicas to acknowledge the write before responding to the client with an error. As such, it relates to the required_acks setting. It should be set lower than socket_timeout.

max_retries

The number of retries when attempting to deliver messages. The default is 2, so 3 attempts in total, but you can configure a higher or lower number.

retry_backoff

The number of seconds to wait after a failed attempt to send messages to a Kafka broker before retrying. The max_retries setting defines the maximum number of retries to attempt, and so the total duration could be up to max_retries * retry_backoff seconds. The timeout can be arbitrarily long, and shouldn't be too short: if a broker goes down its partitions will be handed off to another broker, and that can take tens of seconds.

Compression

See ruby-kafka for more information.

compression_codec

The codec used to compress messages. Must be either snappy or gzip.

compression_threshold

The minimum number of messages that must be buffered before compression is attempted. By default only one message is required. Only relevant if compression_codec is set.

Network

connect_timeout

The number of seconds to wait while connecting to a broker for the first time. When the Kafka library is initialized, it needs to connect to at least one host in brokers in order to discover the Kafka cluster. Each host is tried until there's one that works. Usually that means the first one, but if your entire cluster is down, or there's a network partition, you could wait up to n * connect_timeout seconds, where n is the number of hostnames in brokers.

socket_timeout

Timeout when reading data from a socket connection to a Kafka broker. Must be larger than ack_timeout or you risk killing the socket before the broker has time to acknowledge your messages.

Buffering

When using the asynhronous API, messages are buffered in a background thread and delivered to Kafka based on the configured delivery policy. Because of this, problems that hinder the delivery of messages can cause the buffer to grow. In order to avoid unlimited buffer growth that would risk affecting the host application, some limits are put in place. After the buffer reaches the maximum size allowed, calling DeliveryBoy.deliver_async will raise Kafka::BufferOverflow.

max_buffer_bytesize

The maximum number of bytes allowed in the buffer before new messages are rejected.

max_buffer_size

The maximum number of messages allowed in the buffer before new messages are rejected.

max_queue_size

The maximum number of messages allowed in the queue before new messages are rejected. The queue is used to ferry messages from the foreground threads of your application to the background thread that buffers and delivers messages. You typically only want to increase this number if you have a very high throughput of messages and the background thread can't keep up with spikes in throughput.

SSL Authentication and authorization

See ruby-kafka for more information.

ssl_ca_cert

A PEM encoded CA cert, or an Array of PEM encoded CA certs, to use with an SSL connection.

ssl_ca_cert_file_path

The path to a valid SSL certificate authority file.

ssl_client_cert

A PEM encoded client cert to use with an SSL connection. Must be used in combination with ssl_client_cert_key.

ssl_client_cert_key

A PEM encoded client cert key to use with an SSL connection. Must be used in combination with ssl_client_cert.

ssl_client_cert_key_password

The password required to read the ssl_client_cert_key. Must be used in combination with ssl_client_cert_key.

SASL Authentication and authorization

See ruby-kafka for more information.

Use it through GSSAPI, PLAIN or OAUTHBEARER.

sasl_gssapi_principal

The GSSAPI principal.

sasl_gssapi_keytab

Optional GSSAPI keytab.

sasl_plain_authzid

The authorization identity to use.

sasl_plain_username

The username used to authenticate.

sasl_plain_password

The password used to authenticate.

sasl_oauth_token_provider

A instance of a class which implements the token method. As described in ruby-kafka

class TokenProvider
  def token
    "oauth-token"
  end
end

DeliveryBoy.configure do |config|
  config.sasl_oauth_token_provider = TokenProvider.new
  config.ssl_ca_certs_from_system = true
end

AWS MSK IAM Authentication and Authorization

sasl_aws_msk_iam_access_key_id

The AWS IAM access key. Required.

sasl_aws_msk_iam_secret_key_id

The AWS IAM secret access key. Required.

sasl_aws_msk_iam_aws_region

The AWS region. Required.

sasl_aws_msk_iam_session_token

The session token. This value can be optional.

Examples

Using a role arn and web identity token to generate temporary credentials:

require "aws-sdk-core"
require "delivery_boy"

role = Aws::AssumeRoleWebIdentityCredentials.new(
  role_arn: ENV["AWS_ROLE_ARN"],
  web_identity_token_file: ENV["AWS_WEB_IDENTITY_TOKEN_FILE"]
)

DeliveryBoy.configure do |c|
  c.sasl_aws_msk_iam_access_key_id = role.credentials.access_key_id
  c.sasl_aws_msk_iam_secret_key_id = role.credentials.secret_access_key
  c.sasl_aws_msk_iam_session_token = role.credentials.session_token
  c.sasl_aws_msk_iam_aws_region    = ENV["AWS_REGION"]
  c.ssl_ca_certs_from_system       = true
end

Testing

DeliveryBoy provides a test mode out of the box. When this mode is enabled, messages will be stored in memory rather than being sent to Kafka. If you use RSpec, enabling test mode is as easy as adding this to your spec helper:

# spec/spec_helper.rb
require "delivery_boy/rspec"

Now your application can use DeliveryBoy in tests without connecting to an actual Kafka cluster. Asserting that messages have been delivered is simple:

describe PostsController do
  describe "#show" do
    it "emits an event to Kafka" do
      post = Post.create!(body: "hello")

      get :show, id: post.id

      # Use this API to extract all messages written to a Kafka topic.
      messages = DeliveryBoy.testing.messages_for("post_views")

      expect(messages.count).to eq 1

      # In addition to #value, you can also pull out #key and #partition_key.
      event = JSON.parse(messages.first.value)

      expect(event["post_id"]).to eq post.id
    end
  end
end

This takes care of clearing messages after each example, as well.

If you're not using RSpec, you can easily replicate the functionality yourself. Call DeliveryBoy.test_mode! at load time, and make sure that DeliveryBoy.testing.clear is called after each test.

Instrumentation & monitoring

Since DeliveryBoy is just an opinionated API on top of ruby-kafka, you can use all the instrumentation made available by that library. You can also use the existing monitoring solutions that integrate with various monitoring services.

Contributing

Bug reports and pull requests are welcome on GitHub. Feel free to join our Slack team and ask how best to contribute!

Support and Discussion

If you've discovered a bug, please file a Github issue, and make sure to include all the relevant information, including the version of DeliveryBoy, ruby-kafka, and Kafka that you're using.

If you have other questions, or would like to discuss best practises, how to contribute to the project, or any other ruby-kafka related topic, join our Slack team!

Copyright and license

Copyright 2017 Zendesk, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.

You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

More Repositories

1

android-floating-action-button

Floating Action Button for Android based on Material Design specification
Java
6,374
star
2

maxwell

Maxwell's daemon, a mysql-to-json kafka producer
Java
3,989
star
3

cross-storage

Cross domain local storage, with permissions
JavaScript
2,190
star
4

samson

Web interface for deployments, with plugin architecture and kubernetes support
Ruby
1,446
star
5

ruby-kafka

A Ruby client library for Apache Kafka
Ruby
1,269
star
6

helm-secrets

DEPRECATED A helm plugin that help manage secrets with Git workflow and store them anywhere
Shell
1,159
star
7

curly

The Curly template language allows separating your logic from the structure of your HTML templates.
Ruby
594
star
8

biz

Time calculations using business hours.
Ruby
488
star
9

racecar

Racecar: a simple framework for Kafka consumers in Ruby
Ruby
486
star
10

zendesk_api_client_rb

Official Ruby Zendesk API Client
Ruby
387
star
11

dropbox-api

Dropbox API Ruby Client
Ruby
362
star
12

zendesk_api_client_php

Official Zendesk API v2 client library for PHP
PHP
335
star
13

stronger_parameters

Type checking and type casting of parameters for Action Pack
Ruby
297
star
14

active_record_shards

Support for sharded databases and replicas for ActiveRecord
Ruby
250
star
15

android-db-commons

Some common utilities for ContentProvider/ContentResolver/Cursor and other db-related android stuff
Java
223
star
16

radar

High level API and backend for writing web apps that use push messaging
JavaScript
221
star
17

sunshine-conversations-web

The Smooch Web SDK will add live web messaging to your website or web app.
212
star
18

demo_apps

HTML
179
star
19

arturo

Feature Sliders for Rails
Ruby
174
star
20

belvedere

An image picker library for Android
Java
146
star
21

zendesk_jwt_sso_examples

Examples using JWT for Zendesk SSO
Ruby
142
star
22

sunshine-conversations-ios

Smooch
Objective-C
122
star
23

laika

Log, test, intercept and modify Apollo Client's operations
TypeScript
122
star
24

prop

Puts a cork in their requests
Ruby
117
star
25

zendesk_sdk_ios

Zendesk Mobile SDK for iOS
Objective-C
117
star
26

zopim-chat-web-sdk-sample-app

Zendesk Chat Web SDK sample app developed using React
JavaScript
98
star
27

copenhagen_theme

The default theme for Zendesk Guide
Handlebars
98
star
28

app_scaffold

A scaffold for developers to build ZAF v2 apps
JavaScript
90
star
29

node-publisher

A zero-configuration release automation tool for Node packages inspired by create-react-app and Travis CI.
JavaScript
75
star
30

zendesk_apps_tools

Ruby
74
star
31

zendesk_app_framework_sdk

The Zendesk App Framework (ZAF) SDK is a JavaScript library that simplifies cross-frame communication between iframed apps and the Zendesk App Framework
JavaScript
72
star
32

ios_sdk_demo_apps

This repository contains sample iOS code and applications which use our SDKs
Swift
64
star
33

zendesk_sdk_chat_ios

Mobile Chat SDK for iOS
Objective-C
63
star
34

kamcaptcha

A captcha plugin for Rails
Ruby
63
star
35

zcli

A command-line tool for Zendesk
TypeScript
62
star
36

linksf

A mobile website to connect those in need in to services that can help them
JavaScript
62
star
37

sdk_demo_app_android

This is Remember The Date, an Android demo app for our Mobile SDK. All docs available on developer.zendesk.com
Java
61
star
38

property_sets

A way to store attributes in a side table.
Ruby
51
star
39

go-httpclerk

A simple HTTP request/response logger for Go supporting multiple formatters.
Go
51
star
40

android-schema-utils

Android library for simplifying database schema and migrations management.
Java
48
star
41

android_sdk_demo_apps

This repository contains sample android code and applications which use our SDKs
Java
46
star
42

statsd-logger

StatsD + Datadog APM logging server for development - standalone or embedded
Go
44
star
43

sunshine-conversations-android

Smooch Android SDK
42
star
44

support_sdk_ios

Zendesk Support SDK for iOS
Objective-C
37
star
45

react-native-sunshine-conversations

React Native wrapper for Smooch.io
Java
36
star
46

docker-logs-tail

Docker Logs Tail simultaneously tails logs for all running Docker containers, interleaving them in the command line output
JavaScript
35
star
47

call_center

Ruby
34
star
48

curlybars

Handlebars.js compatible templating library in Ruby
Ruby
34
star
49

kasket

A caching layer for ActiveRecord. Puts a cap on your queries!
Ruby
33
star
50

ruby_memprofiler_pprof

Experimental memory profiler for Ruby that emits pprof files.
C
33
star
51

method_struct

Ruby
33
star
52

samlr

Clean room implementation of SAML for Ruby
Ruby
31
star
53

sdk_demo_app_ios

This is Remember The Date, an iOS demo app for our Mobile SDK. All docs available on developer.zendesk.com
Objective-C
31
star
54

volunteer_portal

An event calendar focused on tracking and reporting volunteering opportunities
JavaScript
29
star
55

classic_asp_jwt

A JWT implementation in Classic ASP
ASP
29
star
56

basecrm-ruby

Base CRM API Client
Ruby
29
star
57

ultragrep

the grep that greps the hardest.
C
28
star
58

chariot-tooltips

A javascript library for creating on screen step by step tutorials.
JavaScript
26
star
59

clj-headlights

Clojure on Beam
Clojure
26
star
60

sunshine-conversations-javascript

Javascript API for Sunshine Conversations
JavaScript
26
star
61

android-autoprovider

Utility for creating ContentProviders without boilerplate and with heavy customization options.
Java
25
star
62

basecrm-php

Base CRM API client, PHP edition
PHP
24
star
63

migration_tools

Rake tasks for Rails that add groups to migrations
Ruby
23
star
64

large_object_store

Store large objects in memcache or others by slicing them.
Ruby
22
star
65

term-check

A GitHub app which runs checks for flagged terminology in GitHub repos
Go
22
star
66

basecrm-python

BaseCRM API Client for Python
Python
22
star
67

sdk_unity_plugin

This repository contains a unity plugin which wraps the Zendesk support SDKs
Objective-C
22
star
68

jazon

Test assertions on JSONs have never been easier
Java
21
star
69

ipcluster

Node.js master/worker clustering module for sticky session load balancing using IPTABLES
JavaScript
21
star
70

sunshine-conversations-python

Smooch API Library for Python
Python
21
star
71

cloudwatch-logger

Connects standard input to Amazon CloudWatch Logs
Go
20
star
72

forger

Android library for populating the ContentProvider with test data.
Java
19
star
73

radar_client

High level API and backend for writing web apps that use push messaging
JavaScript
19
star
74

double_doc

Write documentation with your code, to keep them in sync, ideal for public API docs.
Ruby
18
star
75

pakkr

Python pipeline utility library
Python
18
star
76

chat_sdk_ios

Zendesk Chat SDK
Objective-C
18
star
77

goship

Utility that helps find, connect and copy to particular cloud resources using configured providers
Go
18
star
78

url_builder_app

A Zendesk App to help you generate links for agents.
JavaScript
18
star
79

sunshine-conversations-api-quickstart-example

Sample code to get started with the Smooch REST APIs
JavaScript
17
star
80

sunshine-conversations-desk

A sample business system built with Meteor and the Smooch API
CSS
17
star
81

apt-s3

apt method for private S3 buckets
Go
17
star
82

api_client

HTTP API Client Builder
Ruby
16
star
83

zendesk_apps_support

Ruby
16
star
84

iron_bank

An opinionated Ruby interface to the Zuora REST API
Ruby
15
star
85

private_gem

Keeps your private gems private
Ruby
15
star
86

active_record_host_pool

Connect to multiple databases using one ActiveRecord connection
Ruby
15
star
87

sdk_messaging_ios

The Zendesk Messaging SDK
Objective-C
14
star
88

rate_my_app_ios

An open source version of the Rate My App feature from 1.x versions of the Support SDK.
Swift
14
star
89

input_sanitizer

A gem to sanitize hash of incoming data
Ruby
14
star
90

sunshine-conversations-conversation-extension-examples

A series of examples using Smooch conversation extensions
HTML
14
star
91

scala-flow

A lightweight library intended to make developing Google DataFlow jobs in Scala easier.
Scala
14
star
92

punchabunch

Punchabunch: A highly concurrent, easily configurable SSH local-forwarding proxy
Go
14
star
93

zendesk-jira-plugin

Java
13
star
94

sunshine-conversations-ruby

Smooch API Library for Ruby
Ruby
13
star
95

samson_secret_puller

kubernetes sidecar and app to publish secrets to a containerized app.
Ruby
13
star
96

sqlitemaster

Android library for getting existing db schema information from sqlite_master table.
Java
13
star
97

sunshine-conversations-wordpress

PHP
13
star
98

predictive_load

Ruby
12
star
99

sunshine-conversations-api-spec

Sunshine Conversations OpenAPI Specification for v2+
12
star
100

zombie_record

A soft-delete library
Ruby
12
star