Kafka/distributed commit log service in Go.
- Implement Kafka in Go
- Protocol compatible with Kafka so Kafka clients and services work with Jocko
- Make operating simpler
- Distribute a single binary
- Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
- Smarter configuration settings
- Able to use percentages of disk space for retention policies rather than only bytes and time kept
- Handling size configs when you change the number of partitions or add topics
- Learn a lot and have fun
- Producing
- Fetching
- Partition consensus and distribution
- Protocol
- Produce
- Fetch
- Metadata
- Create Topics
- Delete Topics
- Consumer group [current task]
- Discovery
- API versioning [more API versions to implement]
- Replication [first draft done - testing heavily now]
Iโm writing a book for PragProg called Building Distributed Services with Go. You can sign up on this mailing list and get updated when the bookโs available. It walks you through building a distributed commit log from scratch. I hope it will help Jocko contributors and people who want to work on distributed services.
- How Jocko's built-in service discovery and consensus works
- How Jocko's (and Kafka's) storage internals work
โโโ broker broker subsystem
โโโ cmd commands
โย ย โโโ jocko command to run a Jocko broker and manage topics
โโโ commitlog low-level commit log implementation
โโโ examples examples running/using Jocko
โย ย โโโ cluster example booting up a 3-broker Jocko cluster
โย ย โโโ sarama example producing/consuming with Sarama
โโโ protocol golang implementation of Kafka's protocol
โโโ prometheus wrapper around Prometheus' client lib to handle metrics
โโโ server API subsystem
โโโ testutil test utils
โโโ mock mocks of the various subsystems
-
Clone Jocko
$ go get github.com/travisjeffery/jocko
-
Build Jocko
$ cd $GOPATH/src/github.com/travisjeffery/jocko $ make build
(If you see an error about
dep
not being found, ensure that$GOPATH/bin
is in yourPATH
)
docker build -t travisjeffery/jocko:latest .
See CONTRIBUTING for details on submitting patches and the contribution workflow.
Jocko is under the MIT license, see the LICENSE file for details.
-
GitHub @travisjeffery
-
Twitter @travisjeffery
-
Medium @travisjeffery