|Project Name||Stars||Downloads||Repos Using This||Packages Using This||Most Recent Commit||Total Releases||Latest Release||Open Issues||License||Language|
|Flink Learning||13,198||24 days ago||apache-2.0||Java|
|flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》|
|Redpanda||6,349||1||10 hours ago||343||April 25, 2021||1,240||C++|
|Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!|
|Ksql||5,447||11 hours ago||1,261||other||Java|
|The database purpose-built for stream processing applications.|
|Materialize||4,919||10 hours ago||2||August 12, 2022||1,850||other||Rust|
|Materialize is a fast, distributed SQL database built on streaming internals.|
|Jocko||4,516||a year ago||61||mit||Go|
|Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)|
|Liftbridge||2,413||2||a month ago||67||September 09, 2022||44||apache-2.0||Go|
|Lightweight, fault-tolerant message streams.|
|Examples||1,693||10 days ago||573||July 07, 2022||96||apache-2.0||Shell|
|Apache Kafka and Confluent Platform examples and demos|
|Flink Streaming Platform Web||1,550||4 days ago||27||mit||Java|
|Killrweather||1,174||6 years ago||23||apache-2.0||Scala|
|KillrWeather is a reference application (work in progress) showing how to easily integrate streaming and batch data processing with Apache Spark Streaming, Apache Cassandra, Apache Kafka and Akka for fast, streaming computations on time series data in asynchronous event-driven environments.|
|Stream Reactor||922||1||3 days ago||1||December 27, 2018||77||apache-2.0||Scala|
|Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.|
The goal of this project is to make it easy to experiment with Spark Streaming based on Kafka, by creating examples that run against an embedded Kafka server and an embedded Spark instance. Of course, in making everything easy to work with we also make it perform poorly. It would be a really bad idea to try to learn anything about performance from this project: it's all about functionality, although we sometimes get insight into performance issues by understanding the way the code interacts with RDD partitioning in Spark and topic partitioning in Kafka.
This project is derived from the LearningSpark project which explores the full range of Spark APIs from the viewpoint of Scala developers. There is a corresponding, but much less comprehensive Java version at learning-spark-with-java.
The spark-data-sources project is focused on the new experimental APIs introduced in Spark 2.3.0 for developing adapters for external data sources of various kinds. This API is essentially a Java API (developed in Java) to avoid forcing developers to adopt Scala for their data source adapters. Consequently, the example data sources in this project are written in Java, but both Java and Scala usage examples are provided.
The project was created with IntelliJ Idea 14 Community Edition. It is known to work with JDK 1.8, Scala 2.11.12, and Spark 2.3.0 with its Kafka 0.10 shim library on Ubuntu Linux.
It uses the Direct DStream package spark-streaming-kafka-0-10 for Spark Streaming integration with Kafka 0.10.0.1. The details behind this are explained in the Spark 2.3.0 documentation.
Note that, with the release of Spark 2.3.0, the formerly stable Receiver DStream APIs are now deprecated, and the formerly experimental Direct DStream APIs are now stable.
I've kept around the examples for the older, stable Kafka integration on the kafka0.8 branch
There's a separate set of examples for Kafka integration with the new Structured Streaming features (mainstream as of Spark 2.2).
|util/DirectServerDemo.scala||Run this first as a Spark-free sanity check for embedded server and clients.|
|util/EmbeddedKafkaServer.scala||Starting and stopping an embedded Kafka server, and creating and modifying topics.|
|util/EmbeddedZookeeper.scala||Starting and stopping an embedded Zookeeper.|
|util/PartitionMapAnalyzer.scala||Support for understanding how subscribed Kafka topics and their Kafka partitions map to partitions in the RDD that is emitted by the Spark stream.|
|util/SimpleKafkaClient.scala||Directly connect to Kafka without using Spark.|
|util/SparkKafkaSink.scala||Support for publishing to Kafka topic in parallel from Spark.|
|util/TemporaryDirectories.scala||Support for creating and cleaning up temporary directories needed for Kafka broker, Zookeeper and Spark streaming.|
|SimpleStreaming.scala||Simple way to set up streaming from a Kafka topic. While this program also publishes to the topic, the publishing does not involve Spark|
|ExceptionPropagation.scala||Show how call to awaitTermination() throws propagated exceptions.|
Partitioning is an important factor in determining the scalability oif Kafka-based streaming applications. In this set of examples you can see the relationship between a number of facets of partitioning.
When running these examples, look for:
Another way these examples differ from the basic examples above is that Spark is used to publish to the topic. Perhaps surprisingly, this is not completely straightforward, and relies on util/SparkKafkaSink.scala. An alternative approach to this can be found here.
|SimpleStreamingFromRDD.scala||Data is published by Spark from an RDD, but is repartitioned even through the publishing RDD and the topic have the same number of partitions.|
|SendWithDifferentPartitioning.scala||Send to a topic with different number of partitions.|
|ControlledPartitioning.scala||When publishing to the topic, explicitly assign each record to a partition.|
Partitions can be added to a Kafka topic dynamically. This example shows that an existing stream will not see the data published to the new partitions, and only when the existing streaming context is terminated and a new stream is started from a new context will that data be delivered.
The topic is created with three partitions, and so each RDD the stream produces has three partitions as well, even after two more partitions are added to the topic. This is what's received after the first 500 records are published to the topic while it has only three partitions:
 *** got an RDD, size = 500  *** 3 partitions  *** partition size = 155  *** partition size = 173  *** partition size = 172
When two partitions are added and another 500 messages are published, this is what's received (note both the number of partitions and the number of messages):
 *** got an RDD, size = 288  *** 3 partitions  *** partition size = 98  *** partition size = 89  *** partition size = 101
When a new stream is subsequently created, the RDDs produced have five partitions, but only two of them contain data, as all the data has been drained from the initial three partitions of the topic, by the first stream. Now all 500 messages (288 + 212) from the second set have been delivered.
 *** got an RDD, size = 212  *** 5 partitions  *** partition size = 0  *** partition size = 0  *** partition size = 0  *** partition size = 112  *** partition size = 100
|MultipleConsumerGroups.scala||Two streams subscribing to the same topic via two consumer groups see all the same data.|
|MultipleStreams.scala||Two streams subscribing to the same topic via a single consumer group divide up the data. There's an interesting partitioning interaction here as the streams each get data from two fo the four topic partitions, and each produce RDDs with two partitions each.|
|MultipleTopics.scala||A single stream subscribing to the two topics receives data from both of them.
The partitioning behavior here is quite interesting.
*** got an RDD, size = 200 *** 9 partitions *** partition 1 has 27 records *** rdd partition = 1, topic = foo, topic partition = 0, record count = 27. *** partition 2 has 15 records *** rdd partition = 2, topic = bar, topic partition = 1, record count = 15. *** partition 3 has 17 records *** rdd partition = 3, topic = bar, topic partition = 0, record count = 17. *** partition 4 has 39 records *** rdd partition = 4, topic = foo, topic partition = 1, record count = 39. *** partition 5 has 34 records *** rdd partition = 5, topic = foo, topic partition = 2, record count = 34. *** partition 6 has 11 records *** rdd partition = 6, topic = bar, topic partition = 3, record count = 11. *** partition 7 has 18 records *** rdd partition = 7, topic = bar, topic partition = 4, record count = 18. *** partition 8 has 20 records *** rdd partition = 8, topic = bar, topic partition = 2, record count = 20.
Record timestamps were introduced into Kafka 0.10 as described in KIP-32 and KIP-33.
This example sets up two different topics that handle timestamps differently -- topic A has the timestamp set by the broker when it receives the record, while topic B passes through the timestamp provided in the record (either programmatically when the record was created, as shown here, or otherwise automatically by the producer.)
Since the record carries information about where its timestamp originates, its easy to subscribe to the two topics to create a single stream, and then examine the timestamp of every received record and its type.
NOTE: The use of timestamps to filter topics in the broker, as introduced in Kafka 0.10.1, is blocked on SPARK-18057.