Awesome Open Source

Programming Languages

Search results for spark kafka

373 search results found

Data Engineering Zoomcamp ⭐ 19,461

Free Data Engineering course!

Bigdata Notes ⭐ 14,872

大数据入门指南 ⭐

Flink Learning ⭐ 13,801

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《 Flink 实战与性能优化》

Technology Talk ⭐ 13,579

【大厂面试专栏】一份Java程序员需要的技术指南，这里有面试题、系统架构、职场锦囊、主流中间件等，让

Cookbook ⭐ 12,557

The Data Engineering Cookbook

God Of Bigdata ⭐ 8,483

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive.

Risingwave ⭐ 5,799

The distributed streaming database. Engineered to offer the simplest and most cost-efficient way for stream processing and management.

Pipeline ⭐ 4,158

Bigdataguide ⭐ 2,355

大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料

Szt Bigdata ⭐ 2,055

深圳地铁大数据客流分析系统🚇🚄🌟

Jvm Profiler ⭐ 1,717

JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter

Movie_recommend ⭐ 1,441

基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

Seldon Server ⭐ 1,420

Machine Learning Platform and Recommendation Engine built on Kubernetes

Bigdata Interview ⭐ 1,397

🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop

Bigdata Growth ⭐ 1,256

大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

Killrweather ⭐ 1,174

KillrWeather is a reference application (work in progress) showing how to easily integrate streaming and batch data processing with Apache Spark Streaming, Apache Cassandra, Apache Kafka and Akka for fast, streaming computations on time series data in asynchronous event-driven environments.

Dockerfiles ⭐ 1,171

50+ DockerHub public images for Docker & Kubernetes - DevOps, CI/CD, GitHub Actions, CircleCI, Jenkins, TeamCity, Alpine, CentOS, Debian, Fedora, Ubuntu, Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak

Kafka Storm Starter ⭐ 729

[PROJECT IS NO LONGER MAINTAINED] Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

Pkpmspark ⭐ 697

awesome 三维数据挖掘数据分析 & 推荐

Kafka Spark Consumer ⭐ 616

High Performance Kafka Connector for Spark Streaming.Supports Multi Topic Fetch, Kafka Security. Reliable offset management in Zookeeper. No Data-loss. No dependency on HDFS and WAL. In-built PID rate controller. Support Message Handler . Offset Lag checker.

Freestyle ⭐ 615

A cohesive & pragmatic framework of FP centric Scala libraries

Data Engineering Interview Questions ⭐ 554

More than 2000+ Data engineer interview questions.

Real Time Analytics and Data Pipelines based on Spark Streaming

Agile_data_code_2 ⭐ 435

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

Zeek Analysis Tools (ZAT): Processing and analysis of Zeek network data with Pandas, scikit-learn, Kafka and Spark

Zdh_web ⭐ 379

大数据采集,抽取平台,zdh_web是zdh系列服务的可视化管理平台，包含数据采集,调度,权限,审批

Ecommercerecommendsystem ⭐ 350

商品大数据实时推荐系统。前端：Vue + TypeScript + ElementUI，后端 Spring + Spark

Wirbelsturm ⭐ 333

[PROJECT IS NO LONGER MAINTAINED] Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.

Every Single Day I Tldr ⭐ 311

A daily digest of the articles or videos I've found interesting, that I want to share with you.

Data Accelerator ⭐ 293

Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.

Demo_11.11_storm Spark Hadoop ⭐ 257

hadoop_storm_spark结合实验的例子，模拟淘宝双11节，根据订单详细信息，汇总出总销售 --------大概流程------- 第一阶段（storm实时报表）第二阶段（离线报表）第三阶段（大规模订单即席查询,和多维度查询）第四阶段（数据挖掘和图计算）

Sparkstreaming ⭐ 253

Spark Streaming+Flume+Kafka+HBase+Hadoop+Zookeeper实现实时日志

Kafka Exactly Once ⭐ 245

Big Data Processing Framework - Unified Data API or SQL on Any Storage

Azure Event Hubs Spark ⭐ 225

Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

Avro SerDe for Apache Spark structured APIs.

Video Stream Analytics ⭐ 191

Big Data ⭐ 190

一个开源、成体系的大数据学习教程。spark学习 hadoop hive hbase flink教程 linux 从入门到精通

Wifiprobeanalysis ⭐ 189

基于WIFI探针的商业大数据分析技术

Bigdata Hub ⭐ 187

数据建设与大数据技术知识体系，包含hadoop、hive、spark、flink主流框架和系列框架，

Sparkstreaming ⭐ 183

💥 🚀 封装sparkstreaming动态调节batch time(有数据就执行计算)；🚀 支持运行过程中增删topic；🚀 封装sparkstreaming 1.6 - kafka 010 用以支持 SSL。

Spark Kafka Writer ⭐ 177

Write your Spark data to Kafka seamlessly

A distributed log store based on raft

Kafka Book ⭐ 167

《Kafka技术内幕》代码

Dcos Commons ⭐ 162

DC/OS SDK is a collection of tools, libraries, and documentation for easy integration of technologies such as Kafka, Cassandra, HDFS, Spark, and TensorFlow with DC/OS.

Spark Streaming With Kafka ⭐ 161

Self-contained examples of Apache Spark streaming integrated with Apache Kafka.

Aliyun Emapreduce Datasources ⭐ 157

Extended datasource support for Spark/Hadoop on Aliyun E-MapReduce.

Bigdata In Practice ⭐ 154

大数据实践项目 Hadoop、Spark、Kafka、Hbase、Flink.....

Spark Structured Streaming Examples ⭐ 153

Spark Structured Streaming / Kafka / Cassandra / Elastic

Lambda Arch ⭐ 151

A full big data pipeline (Lambda Architecture) with Spark, Kafka, HDFS and Cassandra.

Huaweicloud Mrs Example ⭐ 150

Examples for HUAWEI CLOUD MRS.

Spark Kafka ⭐ 136

Low level integration of Spark and Kafka

Logvision ⭐ 136

分布式实时日志分析与入侵检测系统

Bigdata Learning ⭐ 136

大数据学习记录

Iot Traffic Monitor ⭐ 123

Example Spark Kafka ⭐ 118

Apache Spark and Apache Kafka integration example

Xichuan_note ⭐ 114

xichuan的学习总结笔记,覆盖了java、spring、java其他常用框架,以及大数据相关组件

Spark Atlas Connector ⭐ 112

A Spark Atlas connector to track data lineage in Apache Atlas

Yelper_recommendation_system ⭐ 108

Yelper recommendation system

De Zoomcamp Ui ⭐ 107

🎨 UI for the Free Data Engineering Zoomcamp 2023 Course provided by DataTalksClub

Logisland ⭐ 106

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.

Dcos Iot Demo ⭐ 106

This project demonstrates how to configure a full stack geo-enabled Internet of Things (IoT) solution using Mesosphere's open sourced Data Center Operating System (DC/OS) using Docker containerization and frameworks for Mesos including Marathon, Kafka, Spark, and Elasticsearch.

Medium Articles ⭐ 97

Repo for all my code on the articles I post on medium

Streamify ⭐ 97

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Kafka Spark Streaming Example ⭐ 96

Simple examle for Spark Streaming over Kafka topic

Learning Spark ⭐ 94

Practical examples of using Apache Spark in several different use cases

Blog Spark Streaming Log Aggregation ⭐ 91

Example of use of Spark Streaming with Kafka

Qstreaming ⭐ 89

A simplified, lightweight ETL pipeline framework for build stream/batch processing applications on top of Apache Spark

基于JavaScript Engine的轻量级规则引擎系统，重构于开源IOT项目thingboard

Kafka Sparkstreaming Cassandra ⭐ 86

Docker container for Kafka - Spark Streaming - Cassandra

Twitter Sentiment Analysis Using Spark Streaming And Kafka ⭐ 78

Twitter Sentiment Analysis using Spark and Kafka

Generator Mitosis ⭐ 75

A micro-service infrastructure generator based on Yeoman/Chatbot, Kubernetes/Docker Swarm, Traefik, Ansible, Jenkins, Spark, Hadoop, Kafka, etc.

本项目已废弃，笔记收藏整理参考：

Kafka_spark_hbase_demo ⭐ 72

kafka spark hbase 日志统计

Google Finance Stock Data Analysis ⭐ 69

Developed a high performance data processing platform using Apache Kafka, Apache Cassandra, and Apache Spark to analyze stock price and related stock tweets sentiment.

Spark_als ⭐ 68

基于spark-ml,spark-mllib,spark-streaming的推荐算法实现

Kafka Spark Streaming Zeppelin Docker ⭐ 68

One click deploy docker-compose with Kafka, Spark Streaming, Zeppelin UI and Monitoring (Grafana + Kafka Manager)

Delta Architecture ⭐ 66

Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline

User Guide Smack ⭐ 66

[Cloudframeworks]SMACK Big Data Architecture - user guide ／ [云框架]SMACK大数据架构-用户指南

Spark Structured Streaming ⭐ 66

Spark structured streaming with Kafka data source and writing to Cassandra

Spark Kafka Cassandra Applying Lambda Architecture ⭐ 64

Docker hello world templates

Lambda Arch Spark ⭐ 63

Data Processing Pipeline ⭐ 59

Real-Time Data Processing Pipeline & Visualization with Docker, Spark, Kafka and Cassandra

Busfloatingdata ⭐ 59

Showcase for IoT Platform Blog

Platys Modern Data Platform ⭐ 58

Support for generating modern platforms dynamically with services such as Kafka, Spark, Streamsets, HDFS, ....

Datapipeline ⭐ 57

Real time stock data pipeline --play with Kafka, Cassandra, Spark, Redis, Node.js, Zookeeper

Titandataoperationsystem ⭐ 57

最好的大数据项目。《Titan数据运营系统》，本项目是一个全栈闭环系统，我们有用作数据可视化的web Echart等；

Recommendmoteur ⭐ 57

电影推荐系统、电影推荐引擎、使用Spark完成的电影推荐引擎

Pybigdata ⭐ 56

使用 python 操作大数据的各种组件

Kafka Streaming Click Analysis ⭐ 56

Use Kafka and Apache Spark streaming to perform click stream analytics

Data Stream Development With Apache Spark Kafka And Spring Boot ⭐ 54

Data Stream Development with Apache Spark, Kafka and Spring Boot by Packt Publishing

Bigdataparty ⭐ 54

大数据组件 All-in-One 的 Dockerfile

Model Serving Tutorial ⭐ 53

Code and presentation for Strata Model Serving tutorial

Awesome Pulsar ⭐ 53

A curated list of Pulsar tools, integrations and resources.

Cloud Bigdata Book ⭐ 53

Fb_scraper ⭐ 52

FBLYZE is a Facebook scraping system and analysis system.

Spark Cep ⭐ 51

Spark CEP is an extension of Spark Streaming to support SQL-based query processing

Related Searches

Scala Spark (3,279)

Java Kafka (3,237)

Python Spark (2,053)

Java Spark (1,587)

Kafka Zookeeper (1,229)

Apache Spark (1,207)

Spark Hadoop (1,188)

Jupyter Notebook Spark (1,151)

Python Kafka (1,133)

Docker Kafka (1,106)

1-100 of 373 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.