Check out the real time data ingestion using Data Ingestion Platform (DiP) which harness the powers of Apache Apex, Apache Flink, Apache Spark and Apache Storm to give real time data ingestion and visualization.
DiP comes along with a UI which allows to switch between multiple data streaming engines and combines them under one single platform.
The DiP architecture has four blocks in the middle layer one for each streaming engine namely Apex Streaming, Flink Streaming, Spark Streaming and Storm Streaming respectively.
Apache Apex is an enterprise grade native YARN big data-in-motion platform that unifies stream processing as well as batch processing. It processes big data in-motion in a highly scalable, highly performant, fault tolerant, stateful, secure, distributed, and an easily operable way.
Blog link - https://techblog.xavient.com/real-time-data-ingestion-dip-apache-apex-co-dev-opportunity/ GitHub link - https://github.com/XavientInformationSystems/Data-Ingestion-Platform/tree/master/dataingest-apex
Apache Flink is an open source platform for distributed stream and batch data processing. Flink's core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.
Blog link- https://techblog.xavient.com/data-ingestion-platformdip-real-time-data-analysis-flink-streaming/ GitHub link - https://github.com/XavientInformationSystems/Data-Ingestion-Platform/tree/master/dataingest-flink
Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams.
Blog link - https://techblog.xavient.com/real-time-data-ingestion-dip-spark-streaming-co-dev-opportunity/ GitHub link - https://github.com/XavientInformationSystems/Data-Ingestion-Platform/tree/master/dataingest-spark
Apache Storm is a free and open source distributed real time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real time processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!
Blog link - https://techblog.xavient.com/real-time-data-ingestion-easy-and-simple-co-dev-opportunity/ GitHub link - https://github.com/XavientInformationSystems/Data-Ingestion-Platform/tree/master/dataingest-storm
Technical team Neeraj Sabharwal Mohiuddin Khan Inamdar Gautam Marya Puneet Singh Sumit Chauhan