Cloudflow enables users to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes.
Alternatives To Cloudflow
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Redpanda6,860116 hours ago343April 25, 20211,348C++
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
2 days ago573July 07, 202295apache-2.0Shell
Apache Kafka and Confluent Platform examples and demos
Spring Cloud Dataflow991165a day ago71April 05, 2022204apache-2.0Java
A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
Stream Reactor93512 days ago1December 27, 201886apache-2.0Scala
Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.
a day ago69October 26, 202248apache-2.0Go
Tigris is an Open Source Serverless NoSQL Database and Search Platform.
a month ago2otherC#
Kubernetes 101474
a month ago13mitHTML
Kubernetes 101 - by Jeff Geerling
15 days ago18apache-2.0Shell
NATS on Kubernetes with Helm Charts
Airy3302a month ago303April 23, 2021135apache-2.0Java
💬 Open Source App Framework to build streaming apps with real-time data - 💎 Build real-time data pipelines and make real-time data universally accessible - 🤖 Join historical and real-time data in the stream to create smarter ML and AI applications. - ⚡ Standardize complex data ingestion and stream data to apps with pre-built connectors
Cloudflow3237a month ago604October 03, 2022125apache-2.0Scala
Cloudflow enables users to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes.
Alternatives To Cloudflow
Select To Compare

Alternative Project Comparisons

Deprecated - The Cloudflow project is deprecated

If you are an existing customer of Lightbend and we have not contacted, please reach out to support.
All existing contracts will be honored and assistance with migration to new tools is available.

Join the chat at   Build status

Cloudflow enables users to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes. Cloudflow allows you to easily break down your streaming application to smaller composable components and wire them together with schema-based contracts. It also comes with a powerful CLI tool to easily manage, scale and configure your streaming applications at runtime. With its powerful abstractions, Cloudflow allows to define, build and deploy the most complex streaming applications.
  • Develop: Focus only on business logic, leave the boilerplate to us.
  • Build: We provide all the tooling for going from business logic to a deployable Docker image.
  • Deploy: We provide Kubernetes tooling to deploy your distributed system with a single command, and manage durable connections between processing stages.

As data pipelines become first-class citizens in microservices architectures, Cloudflow gives developers data-optimized programming abstractions and run-time tooling for Kubernetes. In a nutshell, Cloudflow is an application development toolkit comprising:

  • An API definition for Streamlet, the core abstraction in Cloudflow.
  • An extensible set of runtime implementations for Streamlet(s).
  • A Streamlet composition model driven by a blueprint definition.
  • A sandbox execution mode that accelerates the development and testing of your applications.
  • A set of sbt plugins that are able to package your application into a deployable container.
  • The Cloudflow operator, a Kubernetes operator that manages the application lifecycle on Kubernetes.
  • A CLI, in the form of a kubectl plugin, that facilitates manual and scripted management of the application.

The different parts of Cloudflow work in unison to dramatically accelerate your application development efforts, reducing the time required to create, package, and deploy an application from weeks to hours.

Basic Concepts

Basic components of a Cloudflow Application

As we discussed above, Cloudflow allows developers to quickly build and deploy distributed stream processing applications by breaking such applications into smaller stream processing units called Streamlets. Each Streamlet represents an independent stream processing component that implements a self-contained stage of the application logic. Streamlets let you break down your application into logical pieces that communicate with each other in a streaming fashion to accomplish an end to end goal.

Streamlets can be composed into larger systems using blueprints, which specify how Streamlets can be connected together to form a topology.

Streamlets expose one or more inlets and outlets that represent the data consumed and produced by the Streamlet, respectively. Inlets and outlets are schema-driven, ensuring that data flows are always consistent and that connections between streamlets are compatible. In the diagram above Streamlet 1 has one outlet which feeds data to Streamlet 2 inlet. Streamlet 1 is a component that generates data or could get its data from an external system eg. via an http request. Then Streamlet 2 outlet feeds its data output to Streamlet 3 inlet. Streamlet 2 in this application does the actual data processing. Streamlet 3 then may store its data to some external system. The example described here is a minimal Cloudflow application. The data sent between Streamlets is safely persisted in the underlying pub-sub system, allowing for independent lifecycle management of the different components.

Streamlets can be scaled up and down to meet the load requirements of the application. The underlying data streams are partitioned to allow for parallelism in a distributed application execution.

The Streamlet logic can be written using an extensible choice of streaming runtimes, such as Akka Streams and Spark. The lightweight API exposes the raw power of the underlying runtime and its libraries while providing a higher-level abstraction for composing streamlets and expressing data schemas. Your code is written in your familiar API.

Applications are deployed as a whole. Cloudflow takes care of deploying the individual streamlets and making sure connections get translated into data flowing between them at runtime.

Learn more about the Cloudflow building blocks in our Cloudflow Core Concepts.

The Drivers Behind Cloudflow

Technologies like mobile, the Internet of Things (IoT), Big Data analytics, machine learning, and others are driving enterprises to modernize how they process large volumes of data. A rapidly growing percentage of that data is now arriving in the form of data streams. To extract value from that data as soon as it arrives, those streams require near-realtime processing. We use the term "Fast Data" to describe applications and systems that deal with such requirements.

The Fast Data landscape has been rapidly evolving, with tools like Spark, Flink, and Kafka Streams emerging from the world of large-scale data processing while projects like Reactive Streams and Akka Streams have emerged from the world of application development and high-performance networking.

The demand for availability, scalability, and resilience is forcing fast data architectures to become more like microservice architectures. Conversely, successful organizations building microservices find their data needs grow with their organization while their data sources are becoming more stream-like and more real-time. Hence, there is a unification happening between streaming data and microservice architectures.

It can be quite hard to develop, deploy, and operate large-scale microservices-based systems that can take advantage of streaming data and seamlessly integrate with systems for analytics processing and machine learning. The individual technologies are well-documented, but combining them into fully integrated, unified systems is no easy task.

Cloudflow aims to make this easier by integrating the most popular streaming frameworks into a single platform for creating and running distributed Fast Data applications on Kubernetes.

Where to Go Next?

Popular Kubernetes Projects
Popular Streaming Projects
Popular Virtualization Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Streaming Data