• Stars
    star
    4,927
  • Rank 8,518 (Top 0.2 %)
  • Language
    Go
  • License
    MIT License
  • Created about 8 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

Jocko

ci gitter codecov

Kafka/distributed commit log service in Go.

Goals of this project:

  • Implement Kafka in Go
  • Protocol compatible with Kafka so Kafka clients and services work with Jocko
  • Make operating simpler
  • Distribute a single binary
  • Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
  • Smarter configuration settings
    • Able to use percentages of disk space for retention policies rather than only bytes and time kept
    • Handling size configs when you change the number of partitions or add topics
  • Learn a lot and have fun

TODO

  • Producing
  • Fetching
  • Partition consensus and distribution
  • Protocol
    • Produce
    • Fetch
    • Metadata
    • Create Topics
    • Delete Topics
    • Consumer group [current task]
  • Discovery
  • API versioning [more API versions to implement]
  • Replication [first draft done - testing heavily now]

Hiatus Writing Book

Iโ€™m writing a book for PragProg called Building Distributed Services with Go. You can sign up on this mailing list and get updated when the bookโ€™s available. It walks you through building a distributed commit log from scratch. I hope it will help Jocko contributors and people who want to work on distributed services.

Reading

Project Layout

โ”œโ”€โ”€ broker        broker subsystem
โ”œโ”€โ”€ cmd           commands
โ”‚ย ย  โ””โ”€โ”€ jocko     command to run a Jocko broker and manage topics
โ”œโ”€โ”€ commitlog     low-level commit log implementation
โ”œโ”€โ”€ examples      examples running/using Jocko
โ”‚ย ย  โ”œโ”€โ”€ cluster   example booting up a 3-broker Jocko cluster
โ”‚ย ย  โ””โ”€โ”€ sarama    example producing/consuming with Sarama
โ”œโ”€โ”€ protocol      golang implementation of Kafka's protocol
โ”œโ”€โ”€ prometheus    wrapper around Prometheus' client lib to handle metrics
โ”œโ”€โ”€ server        API subsystem
โ””โ”€โ”€ testutil      test utils
    โ””โ”€โ”€ mock      mocks of the various subsystems

Building

Local

  1. Clone Jocko

    $ go get github.com/travisjeffery/jocko
    
  2. Build Jocko

    $ cd $GOPATH/src/github.com/travisjeffery/jocko
    $ make build
    

    (If you see an error about dep not being found, ensure that $GOPATH/bin is in your PATH)

Docker

docker build -t travisjeffery/jocko:latest .

Contributing

See CONTRIBUTING for details on submitting patches and the contribution workflow.

License

Jocko is under the MIT license, see the LICENSE file for details.


More Repositories

1

timecop

A gem providing "time travel", "time freezing", and "time acceleration" capabilities, making it simple to test time-dependent code. It provides a unified method to mock Time.now, Date.today, and DateTime.now in a single call.
Ruby
3,364
star
2

ClangFormat-Xcode

Xcode plug-in to to use clang-format from in Xcode and consistently format your code with Clang
Objective-C
2,894
star
3

proglog

grpc log service example project
Go
187
star
4

TRVSNavigationControllerTransition

Push/pop transition entire UINavigationController views.
Objective-C
151
star
5

LingusticTaggerDemo

< 80 LOC Implementing Writer Pro's syntax control (with NSLinguisticTagger) that iA tried to patent
Objective-C
106
star
6

TRVSURLSessionOperation

NSURLSession and NSOperationQueue working together
Objective-C
103
star
7

TRVSEventSource

Server-sent events EventSource client using NSURLSession
Objective-C
80
star
8

TRVSMonitor

ObjC synchronization construct with the ability to wait until signalled that a condition was met.
Objective-C
79
star
9

proto-go-sql

Generate SQL Scanner and Valuer implementations for your Protobufs.
Go
77
star
10

ecs-deploy

CLI tool to easily update your ECS services.
Go
65
star
11

mocha-teamcity-reporter

JavaScript
40
star
12

kube-leaderelection

Leader election for your own services, building on Kubernetes' APIs/etcd.
Go
30
star
13

terraform-provider-kafka

Terraform provider for managing Kafka topics.
Go
27
star
14

kafka-proxy

Proxy requests to Kafka with middleware.
Go
27
star
15

go-dynaport

A Go library for getting free ports, dynamically.
Go
25
star
16

validates_phone_number

Adds validation methods to ActiveModel for validating phone numbers.
Ruby
23
star
17

certmagic-sqlstorage

SQL storage for CertMagic/Caddy TLS data.
Go
21
star
18

dotfiles

@travisjeffery's dotfiles. emacs, zsh
Shell
21
star
19

kafka-statsd

Send your Kafka offset lags to StatsD.
Go
20
star
20

sinon-rails

Sinon.js for Rails
Ruby
20
star
21

go-fastbqstreamer

A fast BigQuery streaming client for Go (Golang).
Go
18
star
22

go-bqloadbatcher

A BigQuery client to batch and load data with Go (Golang).
Go
17
star
23

mocker

Mock pkg/generating tool for Go.
Go
17
star
24

proced-narrow

Live-narrowing of search results for proced in Emacs.
Emacs Lisp
16
star
25

deets

Library to get browser details. Name, version, size, etc.
JavaScript
15
star
26

grpc-go-kit-error-example

Go
14
star
27

TRVSRainbowRoad

iOS lib to animate your labels like Mario Kart's Rainbow Road
Objective-C
14
star
28

awesome-wm

My configuration of the awesome window manager.
Lua
14
star
29

ios-how-to-mask-and-shadow

iOS-How-to-MaskAndShadow
Objective-C
14
star
30

jasmine-jquery-rails

Jasmine-jQuery for Rails
Ruby
12
star
31

snipmate-snippets

my snippets that I use with snipmate
Vim Script
12
star
32

datadog-terraform-example

Managing Datadog with Terraform example
HCL
11
star
33

ecs-exec

Run commands on your ECS container instances.
Go
10
star
34

vimperator-scripts

vimperator plug-ins, color schemes and etc I've written
JavaScript
8
star
35

go-batcher

Simple batch library for Golang.
Go
8
star
36

TRVSKit

a collection of useful apis for objc, ios, mac hacking
Objective-C
8
star
37

tumblr.tmbundle

TextMate bundle for interfacing with Tumblr.
7
star
38

ec2-tag-spots

Create tags for AWC EC2 spot fleets.
Go
7
star
39

eshell-z.el

quickly jump to directories in emacs's eshell
Emacs Lisp
7
star
40

NSDictionary-TRVSUnderscoreCamelCaseAdditions

Convert NSDictionary keys to/from under_score/camelCase. Useful when working with JSON APIs, etc.
Objective-C
7
star
41

hubot-socket.io

Socket IO adapter for Hubot.
CoffeeScript
7
star
42

emacs-tumblr

Post to Tumblr using Emacs.
Emacs Lisp
6
star
43

computer-science

Data structures, algorithms, and other good stuff.
Objective-C
6
star
44

vim-auto-mkdir

vim: automatically mkdir when writing file in nonexistent directory
Vim Script
6
star
45

fang

Remove boilerplate using spf13/cobra with spf13/viper.
Go
6
star
46

vim-script-installer

rakefile to install vim scripts downloaded from github and other git repos
5
star
47

nyancat

Animated nyan cat in the terminal written in Go
Go
5
star
48

vim-colors

good vim color themes i've hacked together
Vim Script
5
star
49

vim-gotosymbol

Go to Symbol for Vim with Tagslist.
Vim Script
4
star
50

vimperator-config

see name
JavaScript
4
star
51

will_paginate_twitter_bootstrap

Make will_paginate and Bootstrap, from Twitter work together.
Ruby
4
star
52

hal

Campfire bot.
CoffeeScript
4
star
53

emacs-nav

Nav is a lightweight solution for Emacs users who want something like TextMate's file browser, or the Eclipse project view. Unlike these two, Nav only shows the contents of a single directory at a time, but it allows recursive searching for filenames using the 'f' key-binding, and recursive grepping of file contents with the 'g' key-binding. Nav can be run painlessly in terminals, where Speedbar either fails on its attempt to make a new frame or is hidden. Nav's terminal-friendliness comes from running in the frame where it was started, keeping window management simple. The Nav key bindings are simple, as well -- each key command is a single keystroke long.
Emacs Lisp
4
star
54

retry

Go
3
star
55

go-book

Go
3
star
56

iwantmyname-ddns

Shell
3
star
57

how-to-functional-options

Go
3
star
58

vim-testee

super duper testing plugin/extension for Vim
Vim Script
3
star
59

iterm-open.app

Utility for opening iTerm, the best terminal for the Mac, from the Finder.
3
star
60

clojure-fractal

makin' fractals with clojure
Clojure
3
star
61

go-ec2-metadata

Go pkg to get metadata about running ec2 instances.
Go
3
star
62

PackBot

NodeJS/Faye chatbot with ICanHaz/Moustache Templating, etc.
JavaScript
3
star
63

OpenInMacVim.app

Ported OpenInTextMate over for MacVim users, thus OpenInMacVim.
2
star
64

edi-generator

Ruby
2
star
65

mongo-to-elasticsearch

Load Mongo collections into Elasticsearch.
Go
2
star
66

testerical

Efficiently run tests for Ruby, including Ruby on Rails projects within Vim.
Vim Script
2
star
67

project-euler

Project Euler Solutions
Python
2
star
68

hub.el

hub.el helps you win at git with emacs
Emacs Lisp
2
star
69

om-nom-moize

tasty memoization
Ruby
2
star
70

haskell-repl

simple repl made in/for haskell
Haskell
2
star
71

errors.js

JavaScript
2
star
72

Reddit

a reddit iOS app without using any 3rd party libs such as afnetworking or mantle
Objective-C
2
star
73

grpcerr-metadata

Go
2
star
74

TRVSKeyboardState

Objective-C
2
star
75

judy

rainbow animate your links
JavaScript
2
star
76

twitter_bootstrap

Gem for using Bootstrap, from Twitter on Rails.
Ruby
2
star
77

skydome

Chrome extension to replace 'Rogers Centre' with the true name SkyDome.
JavaScript
2
star
78

urbandict.me

Unofficial API for Urban Dictionary
JavaScript
2
star
79

thr-client

The Heavy Rotation - show your Rdio heavy rotation on your site
JavaScript
2
star
80

stash.cool

http://stash.cool
CSS
1
star
81

burrow-stats

Go
1
star
82

gomodlib

Go
1
star
83

vim-rails-find-partial

Vim Script
1
star
84

pdf_renderer

Ruby
1
star
85

go-tron

Go
1
star
86

ooc-lang.tmbundle

textmate bundle for ooc-lang
1
star
87

homebrew.app

homebrew with a graphical user interface
C
1
star
88

benchmarks

Ruby
1
star
89

travisjeffery.github.com

personal website and blog
HTML
1
star
90

robot-wars

good thing noone will have to clean up the pieces
Clojure
1
star
91

terraform-provider-semaphoreci

Shell
1
star
92

yodlee-client

Yodlee client for Go.
Go
1
star
93

thr-lambda

The Heavy Rotation - show your Rdio heavy rotation on your site
JavaScript
1
star
94

notebox

Note keeping sidekick.
Ruby
1
star
95

vim-help

Mappings for navigating vim help docs.
Vim Script
1
star
96

vim-jump-x2

Navigate jumps motions by buffers.
Vim Script
1
star
97

ChromeGitHubSwitcher

GitHub Header/Implementation File Switcher Extension for Chrome
JavaScript
1
star
98

jays

Web app that scrapes the web for Toronto Blue Jays streams
JavaScript
1
star
99

koa-mirror

A server for testing client libs.
JavaScript
1
star
100

emacs-rails

Makin' Emacs not fail so hard when working with Ruby on Rails.
Emacs Lisp
1
star