Awesome Open Source

Programming Languages

Search results for kafka spark

331 search results found

Data Engineering Zoomcamp ⭐ 19,461

Free Data Engineering course!

Bigdata Notes ⭐ 14,872

大数据入门指南 ⭐

Flink Learning ⭐ 13,801

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《 Flink 实战与性能优化》

Technology Talk ⭐ 13,579

【大厂面试专栏】一份Java程序员需要的技术指南，这里有面试题、系统架构、职场锦囊、主流中间件等，让

Cookbook ⭐ 12,557

The Data Engineering Cookbook

God Of Bigdata ⭐ 8,483

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive.

Pipeline ⭐ 4,158

Bigdataguide ⭐ 2,355

大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料

Szt Bigdata ⭐ 2,055

深圳地铁大数据客流分析系统🚇🚄🌟

Jvm Profiler ⭐ 1,717

JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter

Movie_recommend ⭐ 1,441

基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

Seldon Server ⭐ 1,420

Machine Learning Platform and Recommendation Engine built on Kubernetes

Bigdata Interview ⭐ 1,397

🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop

Bigdata Growth ⭐ 1,256

大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

Kafka Storm Starter ⭐ 729

[PROJECT IS NO LONGER MAINTAINED] Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

Pkpmspark ⭐ 697

awesome 三维数据挖掘数据分析 & 推荐

Kafka Spark Consumer ⭐ 616

High Performance Kafka Connector for Spark Streaming.Supports Multi Topic Fetch, Kafka Security. Reliable offset management in Zookeeper. No Data-loss. No dependency on HDFS and WAL. In-built PID rate controller. Support Message Handler . Offset Lag checker.

Freestyle ⭐ 615

A cohesive & pragmatic framework of FP centric Scala libraries

Real Time Analytics and Data Pipelines based on Spark Streaming

Zeek Analysis Tools (ZAT): Processing and analysis of Zeek network data with Pandas, scikit-learn, Kafka and Spark

Zdh_web ⭐ 379

大数据采集,抽取平台,zdh_web是zdh系列服务的可视化管理平台，包含数据采集,调度,权限,审批

Ecommercerecommendsystem ⭐ 350

商品大数据实时推荐系统。前端：Vue + TypeScript + ElementUI，后端 Spring + Spark

Wirbelsturm ⭐ 333

[PROJECT IS NO LONGER MAINTAINED] Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.

Every Single Day I Tldr ⭐ 311

A daily digest of the articles or videos I've found interesting, that I want to share with you.

Data Accelerator ⭐ 300

Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.

Demo_11.11_storm Spark Hadoop ⭐ 257

hadoop_storm_spark结合实验的例子，模拟淘宝双11节，根据订单详细信息，汇总出总销售 --------大概流程------- 第一阶段（storm实时报表）第二阶段（离线报表）第三阶段（大规模订单即席查询,和多维度查询）第四阶段（数据挖掘和图计算）

Sparkstreaming ⭐ 253

Spark Streaming+Flume+Kafka+HBase+Hadoop+Zookeeper实现实时日志

Kafka Exactly Once ⭐ 245

Big Data Processing Framework - Unified Data API or SQL on Any Storage

Azure Event Hubs Spark ⭐ 225

Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

Avro SerDe for Apache Spark structured APIs.

Video Stream Analytics ⭐ 191

Big Data ⭐ 190

一个开源、成体系的大数据学习教程。spark学习 hadoop hive hbase flink教程 linux 从入门到精通

Bigdata Hub ⭐ 187

数据建设与大数据技术知识体系，包含hadoop、hive、spark、flink主流框架和系列框架，

Sparkstreaming ⭐ 183

💥 🚀 封装sparkstreaming动态调节batch time(有数据就执行计算)；🚀 支持运行过程中增删topic；🚀 封装sparkstreaming 1.6 - kafka 010 用以支持 SSL。

Spark Kafka Writer ⭐ 177

Write your Spark data to Kafka seamlessly

Kafka Book ⭐ 167

《Kafka技术内幕》代码

Dcos Commons ⭐ 162

DC/OS SDK is a collection of tools, libraries, and documentation for easy integration of technologies such as Kafka, Cassandra, HDFS, Spark, and TensorFlow with DC/OS.

Spark Streaming With Kafka ⭐ 161

Self-contained examples of Apache Spark streaming integrated with Apache Kafka.

Aliyun Emapreduce Datasources ⭐ 157

Extended datasource support for Spark/Hadoop on Aliyun E-MapReduce.

Bigdata In Practice ⭐ 154

大数据实践项目 Hadoop、Spark、Kafka、Hbase、Flink.....

Lambda Arch ⭐ 151

A full big data pipeline (Lambda Architecture) with Spark, Kafka, HDFS and Cassandra.

Huaweicloud Mrs Example ⭐ 150

Examples for HUAWEI CLOUD MRS.

Bigdata Learning ⭐ 136

大数据学习记录

Logvision ⭐ 136

分布式实时日志分析与入侵检测系统

Spark Kafka ⭐ 136

Low level integration of Spark and Kafka

Iot Traffic Monitor ⭐ 123

Example Spark Kafka ⭐ 118

Apache Spark and Apache Kafka integration example

Xichuan_note ⭐ 114

xichuan的学习总结笔记,覆盖了java、spring、java其他常用框架,以及大数据相关组件

Spark Atlas Connector ⭐ 112

A Spark Atlas connector to track data lineage in Apache Atlas

Dcos Iot Demo ⭐ 106

This project demonstrates how to configure a full stack geo-enabled Internet of Things (IoT) solution using Mesosphere's open sourced Data Center Operating System (DC/OS) using Docker containerization and frameworks for Mesos including Marathon, Kafka, Spark, and Elasticsearch.

Logisland ⭐ 106

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.

Streamify ⭐ 97

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Medium Articles ⭐ 97

Repo for all my code on the articles I post on medium

Kafka Spark Streaming Example ⭐ 96

Simple examle for Spark Streaming over Kafka topic

Learning Spark ⭐ 94

Practical examples of using Apache Spark in several different use cases

Blog Spark Streaming Log Aggregation ⭐ 91

Example of use of Spark Streaming with Kafka

Qstreaming ⭐ 89

A simplified, lightweight ETL pipeline framework for build stream/batch processing applications on top of Apache Spark

基于JavaScript Engine的轻量级规则引擎系统，重构于开源IOT项目thingboard

Kafka Sparkstreaming Cassandra ⭐ 86

Docker container for Kafka - Spark Streaming - Cassandra

Twitter Sentiment Analysis Using Spark Streaming And Kafka ⭐ 78

Twitter Sentiment Analysis using Spark and Kafka

Generator Mitosis ⭐ 75

A micro-service infrastructure generator based on Yeoman/Chatbot, Kubernetes/Docker Swarm, Traefik, Ansible, Jenkins, Spark, Hadoop, Kafka, etc.

本项目已废弃，笔记收藏整理参考：

Kafka_spark_hbase_demo ⭐ 72

kafka spark hbase 日志统计

Google Finance Stock Data Analysis ⭐ 69

Developed a high performance data processing platform using Apache Kafka, Apache Cassandra, and Apache Spark to analyze stock price and related stock tweets sentiment.

Spark_als ⭐ 68

基于spark-ml,spark-mllib,spark-streaming的推荐算法实现

Delta Architecture ⭐ 66

Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline

Spark Structured Streaming ⭐ 66

Spark structured streaming with Kafka data source and writing to Cassandra

Docker hello world templates

Spark Kafka Cassandra Applying Lambda Architecture ⭐ 64

Lambda Arch Spark ⭐ 63

Busfloatingdata ⭐ 59

Showcase for IoT Platform Blog

Data Processing Pipeline ⭐ 59

Real-Time Data Processing Pipeline & Visualization with Docker, Spark, Kafka and Cassandra

Platys Modern Data Platform ⭐ 58

Support for generating modern platforms dynamically with services such as Kafka, Spark, Streamsets, HDFS, ....

Titandataoperationsystem ⭐ 57

最好的大数据项目。《Titan数据运营系统》，本项目是一个全栈闭环系统，我们有用作数据可视化的web Echart等；

Recommendmoteur ⭐ 57

电影推荐系统、电影推荐引擎、使用Spark完成的电影推荐引擎

Datapipeline ⭐ 57

Real time stock data pipeline --play with Kafka, Cassandra, Spark, Redis, Node.js, Zookeeper

Kafka Streaming Click Analysis ⭐ 56

Use Kafka and Apache Spark streaming to perform click stream analytics

Pybigdata ⭐ 56

使用 python 操作大数据的各种组件

Bigdataparty ⭐ 54

大数据组件 All-in-One 的 Dockerfile

Data Stream Development With Apache Spark Kafka And Spring Boot ⭐ 54

Data Stream Development with Apache Spark, Kafka and Spring Boot by Packt Publishing

Cloud Bigdata Book ⭐ 53

Model Serving Tutorial ⭐ 53

Code and presentation for Strata Model Serving tutorial

Awesome Pulsar ⭐ 53

A curated list of Pulsar tools, integrations and resources.

Spark Cep ⭐ 51

Spark CEP is an extension of Spark Streaming to support SQL-based query processing

Movie Recommender Demo ⭐ 50

This project walks through how you can create recommendations using Apache Spark machine learning. There are a number of jupyter notebooks that you can run on IBM Data Science Experience, and there a live demo of a movie recommendation web application you can interact with. The demo also uses IBM Message Hub (kafka) to push application events to topic where they are consumed by a spark streaming job running on IBM BigInsights (hadoop).

Realtime Dashboard ⭐ 49

Real-time report dashboard with Apache Kafka, Apache Spark Streaming and Node.js

Machine learning around business processes

Sparkonkudu ⭐ 48

Based off the design of SparkOnHBase. This Repo will support Spark, Spark Streaming, and Spark SQL integration with Kudu.

Columbiaimagesearch ⭐ 44

Columbia Image and Face Search tool for MEMEX

Awesome Recommendation Engine ⭐ 43

The purpose of this tiny project is to put things together with the know how that i learned from the course big data expert from formacionhadoop.com The idea is to show how to play with apache spark streaming, kafka,mongo, spark machine learning algorithms.

Datashark ⭐ 41

dataShark is a Security & Network Event Analytics Framework built on Apache Spark

Hyperdrive ⭐ 41

Extensible streaming ingestion pipeline on top of Apache Spark

Etl Light ⭐ 38

A light Kafka to HDFS/S3 ETL library based on Apache Spark

Awesome Druid ⭐ 38

Xxhadoop ⭐ 37

Data Analysis Using Hadoop/Spark/Storm/ElasticSearch/MachineLearning etc. This is My Daily Notes/Code/Demo. Don't fork, Just star !

Real Time Stream Processing Engine ⭐ 37

This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.

Related Searches

Scala Spark (3,279)

Java Kafka (3,237)

Python Spark (2,053)

Java Spark (1,587)

Kafka Zookeeper (1,229)

Apache Spark (1,207)

Spark Hadoop (1,188)

Jupyter Notebook Spark (1,151)

Python Kafka (1,133)

Docker Kafka (1,106)

1-100 of 331 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2025 Awesome Open Source. All rights reserved.