Kubernetes Kafka

Kafka cluster as Kubernetes StatefulSet, plain manifests and config
Alternatives To Kubernetes Kafka
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Kubeshark9,080
3 days ago80apache-2.0Go
The API traffic analyzer for Kubernetes providing real-time K8s protocol-level visibility, capturing and monitoring all traffic and payloads going in, out and across containers, pods, nodes and clusters.. Think TCPDump and Wireshark re-invented for Kubernetes
Redpanda6,7861a day ago343April 25, 20211,336C++
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
Kafdrop4,531
3 days ago45apache-2.0Java
Kafka Web UI
Pipeline4,160
7 months ago85July 18, 20171apache-2.0Jsonnet
PipelineAI Kubeflow Distribution
Strimzi Kafka Operator3,8552a day ago34September 06, 2022136apache-2.0Java
Apache Kafka® running on Kubernetes
Go Zero Looklook3,192
12 days ago2mitGo
🔥基于go-zero(go zero) 微服务全技术栈开发最佳实践项目。Develop best practice projects based on the full technology stack of go zero (go zero) microservices.
Vanus2,156
5 days ago59apache-2.0Go
Vanus is a Serverless, event streaming system with processing capabilities. It easily connects SaaS, Cloud Services, and Databases to help users build next-gen Event-driven Applications.
Kubernetes Kafka1,773
5 months ago83apache-2.0Shell
Kafka cluster as Kubernetes StatefulSet, plain manifests and config
Examples1,734
5 days ago573July 07, 202295apache-2.0Shell
Apache Kafka and Confluent Platform examples and demos
Practical.cleanarchitecture1,529
3 days ago25C#
Full-stack .Net 7 Clean Architecture (Microservices + Dapr, Modular Monolith, Monolith), Blazor, Angular 15, React 18, Vue 3, Domain-Driven Design, CQRS, SOLID, Asp.Net Core Identity Custom Storage, Identity Server, Entity Framework Core, Selenium, SignalR, Hosted Services, Health Checks, Rate Limiting, Cloud (Azure, AWS) Services, ...
Alternatives To Kubernetes Kafka
Select To Compare


Alternative Project Comparisons
Readme

Kafka for Kubernetes

This community seeks to provide:

  • Production-worthy Kafka setup for persistent (domain- and ops-) data at small scale.
  • Operational knowledge, biased towards resilience over throughput, as Kubernetes manifest.
  • A platform for event-driven (streaming!) microservices design using Kubernetes.

To quote @arthurk:

thanks for creating and maintaining this Kubernetes files, they're up-to-date (unlike the kubernetes contrib files, don't require helm and work great!

Getting started

We suggest you apply -f manifests in the following order:

That'll give you client "bootstrap" bootstrap.kafka.svc.cluster.local:9092.

Fork

Our only dependency is kubectl. Not because we dislike Helm or Operators, but because we think plain manifests make it easier to collaborate. If you begin to rely on this kafka setup we recommend you fork, for example to edit broker config.

Kustomize

With the introduction of app customization in kubectl 1.14 there's an alternative to forks. We as a community can maintain a set of overlays.

See the variants folder for different overlays. For example to scale to 1 kafka broker try kubectl apply -k variants/scale-1/. Variants also include examples of how to configure volumes for GKE, AWS and AKS with different storage classes.

Quickstart

kubectl create namespace kafka && \
kubectl apply -k github.com/Yolean/kubernetes-kafka/variants/dev-small/?ref=v6.0.3

When all pods are Ready, test with for example kafkacat -b localhost:9094 -L over kubectl -n kafka port-forward kafka-0 9094.

Maintaining your own kustomization

Start your variant as a new folder in your choice of version control, with a base kustomization.yaml pointing to a tag or revision in this repository:

bases:
- github.com/Yolean/kubernetes-kafka/rbac-namespace-default/?ref=60d01b5
- github.com/Yolean/kubernetes-kafka/kafka/?ref=60d01b5
- github.com/Yolean/kubernetes-kafka/zookeeper/?ref=60d01b5

Then pick and chose from patches our example variants to tailor your Kafka setup.

Version history

tag k8s ≥ highlights
v7.0.0 1.15+ Breaking with nonroot and native bases
v6.0.x 1.13+ Kafka 2.4.0 + standard storage class
v6.0.0 1.11+ Kafka 2.2.0 + apply -k (kubectl 1.14+) + #270
v5.1.0 1.11+ Kafka 2.1.1
v5.0.3 1.11+ Zookeeper fix #227 + maxClientCnxns=1
v5.0 1.11+ Destabilize because in Docker we want Java 11 #197 #191
v4.3.1 1.9+ Critical Zookeeper persistence fix #228
v4.3 1.9+ Adds a proper shutdown hook #207
v4.2 1.9+ Kafka 1.0.2 and tools upgrade
... see releases for full history ...
v1.0 1 Stateful? In Kubernetes? In 2016? Yes.

Monitoring

Have a look at:

Outside (out-of-cluster) access

Available for:

Stream...

Popular Kubernetes Projects
Popular Kafka Projects
Popular Virtualization Categories

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Shell
Kubernetes
Kafka
Zookeeper