Apache Kafka auto-discovery anomaly detection 🐡
Alternatives To Sarkac
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
2 months ago108apache-2.0
The Data Engineering Cookbook
Sarama9,7841,7592 days ago152August 11, 2022317mitGo
Sarama is a Go library for Apache Kafka.
God Of Bigdata7,901
19 days ago2
Confluent Kafka Go3,85831620 hours ago62August 02, 2022198apache-2.0Go
Confluent's Apache Kafka Golang client
Strimzi Kafka Operator3,72528 hours ago34September 06, 2022135apache-2.0Java
Apache Kafka® running on Kubernetes
Kafkajs3,0973932224 days ago291August 16, 2022223mitJavaScript
A modern Apache Kafka client for node.js
Kafka Monitor1,946
a day ago27apache-2.0Java
Xinfra Monitor monitors the availability of Kafka clusters by producing synthetic workloads using end-to-end pipelines to obtain derived vital statistics - E2E latency, service produce/consume availability, offsets commit availability & latency, message loss rate and more.
Kaf1,80926 days ago71July 18, 202260apache-2.0Go
Modern CLI for Apache Kafka, written in Go.
Karafka1,6572258 hours ago70July 08, 202229otherRuby
Ruby and Rails efficient Kafka processing framework
3 months agomitGo
【❤️ 互联网最全大厂技术分享PPT 👍🏻 持续更新中!】🍻各大技术交流会、活动资料汇总 ,如 👉QCon👉全球运维技术大会 👉 GDG 👉 全球技术领导力峰会👉大前端大会👉架构师峰会👉敏捷开发DevOps👉OpenResty👉Elastic,欢迎 PR / Issues
Alternatives To Sarkac
Select To Compare

Alternative Project Comparisons



Apache Kafka topic and message anomaly detection with automated discovery.

npm version


Very simple via yarn add sarkac


Basically you can create your own apps with sarkac or integrate it in your existing apps, however you can also simply just spin up an instance by providing it some simple configuration info. Please note: that sarkac requires MongoDB 🌱 to store its windows.

You can find an example here.

Visit http://localhost:8033/ to check for sarkac's HTTP endpoints, that give infos about discovery and anomaly stats.

How does it work?

sarkac connects to your Kafka cluster and runs a simple discovery protocol to detect existing Kafka topics, it will automatically subscribe to them (also to newly added Kafka topics) and analyse their schema (has to be JSON), of their message payloads. It will then identify any fields of type 'number' and track them across all messages it receives. sarkac then uses MongoDB to keep multiple (as much as you configure) rolling windows of the values of the tracked fields. It runs the 68–95–99.7 rule on every window of every tracked field continously to detect anomalies. If an anomaly is detected it produces its information to an anomaly Kafka topic.

Running without auto-discovery

As shown in the example (uncommented dsl lines) it is also possible to deactivate auto discovery of topics and fields and simply run sarkac on fixed topics, by configuring the config.dsl object, do not forget to turn off discovery via config.discovery.enabled = false.

Additionally you can also turn off anomaly production to Kafka via = false.


Given a Kafka cluster with a certain amount of topics, keeping an eye on all of them at once can be challenging. And although we do not claim that you can cover all kinds of anomalies with sarkac, it can at least help you to tackle certain problems earlier. Just spin up a few instances and let them disover you Kafka broker's topics and produce anomalies to an output topic. Use our Kafka to Prometheus Connector to sink the anomaly topic into Prometheus and use Grafanas alert magic to get you notified based on detected anomalies.


Build with ❤️ 🍕 and ☕️ by nodefluent

Popular Kafka Projects
Popular Apache Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Anomaly Detection