Awesome Open Source
Awesome Open Source

Actions Status

Join us on slack Alt text

Lenses for Apache Kafka

Lenses offers SQL (for data browsing and Kafka Streams), Kafka Connect connector management, cluster monitoring and more.

You can find more on lenses.io

Stream Reactor

Alt text FOSSA Status

A collection of components to build a real time ingestion pipeline.

Kafka Compatibility

From version 4.0.0 separate artifacts are built targeting Kafka 3.1 and Kafka 2.8.

  • Kafka 3.1 (Confluent 7.1) - Stream Reactor 4.0.0 (Kafka 3.1 Build)
  • Kafka 2.8 (Confluent 6.2) - Stream Reactor 4.0.0 (Kafka 2.8 Build)
  • Kafka 2.5 (Confluent 5.5) - Stream reactor 2.0.0+
  • Kafka 2.0 -> 2.4 (Confluent 5.4) - Stream reactor 1.2.7

DEPRECATION NOTICE

The following connectors have been deprecated and are no longer included in any release from 3.0.

  • Blockchain
  • Bloomberg
  • Rethink
  • VoltDB

The following connectors have been deprecated and are no longer included in any release from 4.0.

  • Coap
  • Hive 1.1

Connectors

Please take a moment and read the documentation and make sure the software prerequisites are met!!

Connector Type Description Docs
AWS S3 Sink Copy data from Kafka to AWS S3. Docs
AzureDocumentDb Sink Copy data from Kafka and Azure Document Db. Docs
Cassandra Source Copy data from Cassandra to Kafka. Docs
*Cassandra Sink Certified DSE Cassandra, copy data from Kafka to Cassandra. Docs
Elastic 6 Sink Copy data from Kafka to Elastic Search 6.x w. tcp or http Docs
FTP/HTTP Source Copy data from FTP/HTTP to Kafka. Docs
Hazelcast Sink Copy data from Kafka to Hazelcast. Docs
HBase Sink Copy data from Kafka to HBase. Docs
Hive Source Copy data from Hive/HDFS to Kafka. Docs
Hive Sink Copy data from Kafka to Hive/HDFS Docs
InfluxDb Sink Copy data from Kafka to InfluxDb. Docs
Kudu Sink Copy data from Kafka to Kudu. Docs
JMS Source Copy data from JMS topics/queues to Kafka. Docs
JMS Sink Copy data from Kafka to JMS. Docs
MongoDB Sink Copy data from Kafka to MongoDB. Docs
MQTT Source Copy data from MQTT to Kafka. Docs
MQTT Sink Copy data from Kafka to MQTT. Docs
Pulsar Source Copy data from Pulsar to Kafka. Docs
Pulsar Sink Copy data from Kafka to Pulsar. Docs
Redis Sink Copy data from Kafka to Redis. Docs

Release Notes

Please see the Stream Reactor Release Notes at Lenses Documentation.

Building

To build:

sbt clean compile

To test:

sbt test

To create assemblies:

sbt assembly

To build a particular project for kafka 2.8:

sbt "project cassandra-kafka-2-8" compile

To build a particular project for kafka 3.1:

sbt "project cassandra-kafka-3-1" compile

To test a particular project:

sbt "project cassandra-kafka-2-8" test

To create a jar of a particular project:

sbt "project cassandra-kafka-2-8" assembly

Running E2E tests

If not already built, you must first build the connector archives:

sbt "project cassandra-kafka-2-8" assembly
sbt "project elastic6-kafka-2-8" assembly 
sbt "project mongodb-kafka-2-8" assembly
sbt "project redis-kafka-2-8" assembly

sbt "project cassandra-kafka-3-1" assembly 
sbt "project elastic6-kafka-3-1" assembly 
sbt "project mongodb-kafka-3-1" assembly 
sbt "project redis-kafka-3-1" assembly

To run the tests:

sbt e2e:test

Contributing

We'd love to accept your contributions! Please use GitHub pull requests: fork the repo, develop and test your code, semantically commit and submit a pull request. Thanks!

License

FOSSA Status

Alternatives To Stream Reactor
Select To Compare


Alternative Project Comparisons
Related Awesome Lists
Top Programming Languages

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Mongodb (32,022
Scala (29,378
Kubernetes (25,954
Redis (24,260
Elasticsearch (11,423
Kafka (9,775
Streaming (9,743
Mqtt (8,115
Ftp (3,682
Cassandra (3,298
Influxdb (2,861
Hive (2,688
Hbase (1,596
Rethinkdb (990
Pulsar (527
Hazelcast (497
Coap (477
Kafka Connect (386
Cosmosdb (268
Jms (138
Dataops (100
Kudu (27