Awesome Open Source
Awesome Open Source


ci gitter codecov

Kafka/distributed commit log service in Go.

Goals of this project:

  • Implement Kafka in Go
  • Protocol compatible with Kafka so Kafka clients and services work with Jocko
  • Make operating simpler
  • Distribute a single binary
  • Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
  • Smarter configuration settings
    • Able to use percentages of disk space for retention policies rather than only bytes and time kept
    • Handling size configs when you change the number of partitions or add topics
  • Learn a lot and have fun


  • [x] Producing
  • [x] Fetching
  • [x] Partition consensus and distribution
  • [ ] Protocol
    • [x] Produce
    • [x] Fetch
    • [x] Metadata
    • [x] Create Topics
    • [x] Delete Topics
    • [ ] Consumer group [current task]
  • [x] Discovery
  • [ ] API versioning [more API versions to implement]
  • [ ] Replication [first draft done - testing heavily now]

Hiatus Writing Book

I’m writing a book for PragProg called Building Distributed Services with Go. You can sign up on this mailing list and get updated when the book’s available. It walks you through building a distributed commit log from scratch. I hope it will help Jocko contributors and people who want to work on distributed services.


Project Layout

├── broker        broker subsystem
├── cmd           commands
│   └── jocko     command to run a Jocko broker and manage topics
├── commitlog     low-level commit log implementation
├── examples      examples running/using Jocko
│   ├── cluster   example booting up a 3-broker Jocko cluster
│   └── sarama    example producing/consuming with Sarama
├── protocol      golang implementation of Kafka's protocol
├── prometheus    wrapper around Prometheus' client lib to handle metrics
├── server        API subsystem
└── testutil      test utils
    └── mock      mocks of the various subsystems



  1. Clone Jocko

    $ go get
  2. Build Jocko

    $ cd $GOPATH/src/
    $ make build

    (If you see an error about dep not being found, ensure that $GOPATH/bin is in your PATH)


docker build -t travisjeffery/jocko:latest .


See CONTRIBUTING for details on submitting patches and the contribution workflow.


Jocko is under the MIT license, see the LICENSE file for details.

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
go (14,468
kafka (397
distributed-systems (285
streaming (283
messaging (207
queue (167

Find Open Source By Browsing 7,000 Topics Across 59 Categories