Spark Streaming With Kafka

Self-contained examples of Apache Spark streaming integrated with Apache Kafka.
Alternatives To Spark Streaming With Kafka
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Flink Learning13,198
24 days agoapache-2.0Java
flink learning blog. 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例,还有 Flink 落地应用的大型项目案例(PVUV、日志存储、百亿数据实时去重、监控告警)分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Redpanda6,349110 hours ago343April 25, 20211,240C++
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
11 hours ago1,261otherJava
The database purpose-built for stream processing applications.
10 hours ago2August 12, 20221,850otherRust
Materialize is a fast, distributed SQL database built on streaming internals.
a year ago61mitGo
Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)
Liftbridge2,4132a month ago67September 09, 202244apache-2.0Go
Lightweight, fault-tolerant message streams.
10 days ago573July 07, 202296apache-2.0Shell
Apache Kafka and Confluent Platform examples and demos
Flink Streaming Platform Web1,550
4 days ago27mitJava
6 years ago23apache-2.0Scala
KillrWeather is a reference application (work in progress) showing how to easily integrate streaming and batch data processing with Apache Spark Streaming, Apache Cassandra, Apache Kafka and Akka for fast, streaming computations on time series data in asynchronous event-driven environments.
Stream Reactor92213 days ago1December 27, 201877apache-2.0Scala
Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.
Alternatives To Spark Streaming With Kafka
Select To Compare

Alternative Project Comparisons

Self-contained examples of Spark streaming integrated with Kafka

Build Status

The goal of this project is to make it easy to experiment with Spark Streaming based on Kafka, by creating examples that run against an embedded Kafka server and an embedded Spark instance. Of course, in making everything easy to work with we also make it perform poorly. It would be a really bad idea to try to learn anything about performance from this project: it's all about functionality, although we sometimes get insight into performance issues by understanding the way the code interacts with RDD partitioning in Spark and topic partitioning in Kafka.

Related projects

This project is derived from the LearningSpark project which explores the full range of Spark APIs from the viewpoint of Scala developers. There is a corresponding, but much less comprehensive Java version at learning-spark-with-java.

The spark-data-sources project is focused on the new experimental APIs introduced in Spark 2.3.0 for developing adapters for external data sources of various kinds. This API is essentially a Java API (developed in Java) to avoid forcing developers to adopt Scala for their data source adapters. Consequently, the example data sources in this project are written in Java, but both Java and Scala usage examples are provided.


The project was created with IntelliJ Idea 14 Community Edition. It is known to work with JDK 1.8, Scala 2.11.12, and Spark 2.3.0 with its Kafka 0.10 shim library on Ubuntu Linux.

It uses the Direct DStream package spark-streaming-kafka-0-10 for Spark Streaming integration with Kafka The details behind this are explained in the Spark 2.3.0 documentation.

Note that, with the release of Spark 2.3.0, the formerly stable Receiver DStream APIs are now deprecated, and the formerly experimental Direct DStream APIs are now stable.

Using the deprecated (Receiver DStream) Kafka 0.8.0 APIs

I've kept around the examples for the older, stable Kafka integration on the kafka0.8 branch

Structured Streaming

There's a separate set of examples for Kafka integration with the new Structured Streaming features (mainstream as of Spark 2.2).


File Purpose
util/DirectServerDemo.scala Run this first as a Spark-free sanity check for embedded server and clients.
util/EmbeddedKafkaServer.scala Starting and stopping an embedded Kafka server, and creating and modifying topics.
util/EmbeddedZookeeper.scala Starting and stopping an embedded Zookeeper.
util/PartitionMapAnalyzer.scala Support for understanding how subscribed Kafka topics and their Kafka partitions map to partitions in the RDD that is emitted by the Spark stream.
util/SimpleKafkaClient.scala Directly connect to Kafka without using Spark.
util/SparkKafkaSink.scala Support for publishing to Kafka topic in parallel from Spark.
util/TemporaryDirectories.scala Support for creating and cleaning up temporary directories needed for Kafka broker, Zookeeper and Spark streaming.

Basic Examples

File What's Illustrated
SimpleStreaming.scala Simple way to set up streaming from a Kafka topic. While this program also publishes to the topic, the publishing does not involve Spark
ExceptionPropagation.scala Show how call to awaitTermination() throws propagated exceptions.

Partitioning Examples

Partitioning is an important factor in determining the scalability oif Kafka-based streaming applications. In this set of examples you can see the relationship between a number of facets of partitioning.

  • The number of partitions in the RDD that is being published to a topic -- if indeed this involves an RDD, as the data is often published from a non-Spark application
  • The number of partitions of the topic itself (usually specified at topic creation)
  • THe number of partitions in the RDDs created by the Kafka stream
  • Whether and how messages move between partitions when they are transferred

When running these examples, look for:

  • The topic partition number that is printed with each ConsumerRecord
  • After all the records are printed, the number of partitions in the resulting RDD and size of each partition. For example: *** 4 partitions *** partition size = 253 *** partition size = 252 *** partition size = 258 *** partition size = 237

Another way these examples differ from the basic examples above is that Spark is used to publish to the topic. Perhaps surprisingly, this is not completely straightforward, and relies on util/SparkKafkaSink.scala. An alternative approach to this can be found here.

File What's Illustrated
SimpleStreamingFromRDD.scala Data is published by Spark from an RDD, but is repartitioned even through the publishing RDD and the topic have the same number of partitions.
SendWithDifferentPartitioning.scala Send to a topic with different number of partitions.
ControlledPartitioning.scala When publishing to the topic, explicitly assign each record to a partition.

Partitions can be added to a Kafka topic dynamically. This example shows that an existing stream will not see the data published to the new partitions, and only when the existing streaming context is terminated and a new stream is started from a new context will that data be delivered.

The topic is created with three partitions, and so each RDD the stream produces has three partitions as well, even after two more partitions are added to the topic. This is what's received after the first 500 records are published to the topic while it has only three partitions:

[1] *** got an RDD, size = 500
[1] *** 3 partitions
[1] *** partition size = 155
[1] *** partition size = 173
[1] *** partition size = 172

When two partitions are added and another 500 messages are published, this is what's received (note both the number of partitions and the number of messages):

[1] *** got an RDD, size = 288
[1] *** 3 partitions
[1] *** partition size = 98
[1] *** partition size = 89
[1] *** partition size = 101

When a new stream is subsequently created, the RDDs produced have five partitions, but only two of them contain data, as all the data has been drained from the initial three partitions of the topic, by the first stream. Now all 500 messages (288 + 212) from the second set have been delivered.

[2] *** got an RDD, size = 212
[2] *** 5 partitions
[2] *** partition size = 0
[2] *** partition size = 0
[2] *** partition size = 0
[2] *** partition size = 112
[2] *** partition size = 100

Other Examples

File What's Illustrated
MultipleConsumerGroups.scala Two streams subscribing to the same topic via two consumer groups see all the same data.
MultipleStreams.scala Two streams subscribing to the same topic via a single consumer group divide up the data. There's an interesting partitioning interaction here as the streams each get data from two fo the four topic partitions, and each produce RDDs with two partitions each.
MultipleTopics.scala A single stream subscribing to the two topics receives data from both of them. The partitioning behavior here is quite interesting.
  • The topics have three and six partitions respectively.
  • Each RDD has nine partitions.
  • Each RDD partition receives data from exactly one partition of one topic.
Hence the output of the PartitionMapAnalyzer:
*** got an RDD, size = 200
*** 9 partitions
*** partition 1 has 27 records
*** rdd partition = 1, topic = foo, topic partition = 0, record count = 27.
*** partition 2 has 15 records
*** rdd partition = 2, topic = bar, topic partition = 1, record count = 15.
*** partition 3 has 17 records
*** rdd partition = 3, topic = bar, topic partition = 0, record count = 17.
*** partition 4 has 39 records
*** rdd partition = 4, topic = foo, topic partition = 1, record count = 39.
*** partition 5 has 34 records
*** rdd partition = 5, topic = foo, topic partition = 2, record count = 34.
*** partition 6 has 11 records
*** rdd partition = 6, topic = bar, topic partition = 3, record count = 11.
*** partition 7 has 18 records
*** rdd partition = 7, topic = bar, topic partition = 4, record count = 18.
*** partition 8 has 20 records
*** rdd partition = 8, topic = bar, topic partition = 2, record count = 20.

Record timestamps were introduced into Kafka 0.10 as described in KIP-32 and KIP-33.

This example sets up two different topics that handle timestamps differently -- topic A has the timestamp set by the broker when it receives the record, while topic B passes through the timestamp provided in the record (either programmatically when the record was created, as shown here, or otherwise automatically by the producer.)

Since the record carries information about where its timestamp originates, its easy to subscribe to the two topics to create a single stream, and then examine the timestamp of every received record and its type.

NOTE: The use of timestamps to filter topics in the broker, as introduced in Kafka 0.10.1, is blocked on SPARK-18057.

Popular Kafka Projects
Popular Streaming Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Spark Streaming