|Project Name||Stars||Downloads||Repos Using This||Packages Using This||Most Recent Commit||Total Releases||Latest Release||Open Issues||License||Language|
|Cookbook||11,362||2 months ago||108||apache-2.0|
|The Data Engineering Cookbook|
|Sarama||9,784||1,759||2 days ago||152||August 11, 2022||317||mit||Go|
|Sarama is a Go library for Apache Kafka.|
|God Of Bigdata||7,901||19 days ago||2|
|Confluent Kafka Go||3,858||316||20 hours ago||62||August 02, 2022||198||apache-2.0||Go|
|Confluent's Apache Kafka Golang client|
|Strimzi Kafka Operator||3,725||2||8 hours ago||34||September 06, 2022||135||apache-2.0||Java|
|Apache Kafka® running on Kubernetes|
|A modern Apache Kafka client for node.js|
|Kafka Monitor||1,946||a day ago||27||apache-2.0||Java|
|Xinfra Monitor monitors the availability of Kafka clusters by producing synthetic workloads using end-to-end pipelines to obtain derived vital statistics - E2E latency, service produce/consume availability, offsets commit availability & latency, message loss rate and more.|
|Kaf||1,809||2||6 days ago||71||July 18, 2022||60||apache-2.0||Go|
|Modern CLI for Apache Kafka, written in Go.|
|Karafka||1,657||22||5||8 hours ago||70||July 08, 2022||29||other||Ruby|
|Ruby and Rails efficient Kafka processing framework|
|Meetup||1,214||3 months ago||mit||Go|
|【❤️ 互联网最全大厂技术分享PPT 👍🏻 持续更新中！】🍻各大技术交流会、活动资料汇总 ，如 👉QCon👉全球运维技术大会 👉 GDG 👉 全球技术领导力峰会👉大前端大会👉架构师峰会👉敏捷开发DevOps👉OpenResty👉Elastic，欢迎 PR / Issues|
Apache Kafka topic and message anomaly detection with automated discovery.
Very simple via
yarn add sarkac
Basically you can create your own apps with sarkac or integrate it in your existing apps, however you can also simply just spin up an instance by providing it some simple configuration info. Please note: that sarkac requires MongoDB 🌱 to store its windows.
You can find an example here.
http://localhost:8033/ to check for sarkac's HTTP endpoints, that give infos about discovery and
sarkac connects to your Kafka cluster and runs a simple discovery protocol to detect existing Kafka topics, it will automatically subscribe to them (also to newly added Kafka topics) and analyse their schema (has to be JSON), of their message payloads. It will then identify any fields of type 'number' and track them across all messages it receives. sarkac then uses MongoDB to keep multiple (as much as you configure) rolling windows of the values of the tracked fields. It runs the 68–95–99.7 rule on every window of every tracked field continously to detect anomalies. If an anomaly is detected it produces its information to an anomaly Kafka topic.
As shown in the example (uncommented dsl lines) it is also possible to deactivate auto discovery of topics and fields
and simply run sarkac on fixed topics, by configuring the
config.dsl object, do not forget to turn off discovery via
config.discovery.enabled = false.
Additionally you can also turn off anomaly production to Kafka via
config.target.produceAnomalies = false.
Given a Kafka cluster with a certain amount of topics, keeping an eye on all of them at once can be challenging. And although we do not claim that you can cover all kinds of anomalies with sarkac, it can at least help you to tackle certain problems earlier. Just spin up a few instances and let them disover you Kafka broker's topics and produce anomalies to an output topic. Use our Kafka to Prometheus Connector to sink the anomaly topic into Prometheus and use Grafanas alert magic to get you notified based on detected anomalies.
Build with ❤️ 🍕 and ☕️ by nodefluent