Awesome Open Source

Programming Languages

Search results for spark kafka

631 search results found

Data Engineering Zoomcamp ⭐ 13,734

Free Data Engineering course!

Bigdata Notes ⭐ 13,291

大数据入门指南 ⭐️

Flink Learning ⭐ 13,198

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《 Flink 实战与性能优化》

Technology Talk ⭐ 13,004

汇总java生态圈常用技术框架、开源中间件，系统架构、数据库、大公司架构案例、常用三方类库、项目管理

Cookbook ⭐ 11,769

The Data Engineering Cookbook

God Of Bigdata ⭐ 7,992

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive.

Pipeline ⭐ 4,160

PipelineAI Kubeflow Distribution

Bigdataguide ⭐ 1,994

大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料

Szt Bigdata ⭐ 1,702

深圳地铁大数据客流分析系统🚇🚄🌟

Jvm Profiler ⭐ 1,661

JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter

Movie_recommend ⭐ 1,441

基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

Seldon Server ⭐ 1,420

Machine Learning Platform and Recommendation Engine built on Kubernetes

Bigdata Interview ⭐ 1,397

🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop

Killrweather ⭐ 1,174

KillrWeather is a reference application (work in progress) showing how to easily integrate streaming and batch data processing with Apache Spark Streaming, Apache Cassandra, Apache Kafka and Akka for fast, streaming computations on time series data in asynchronous event-driven environments.

Dockerfiles ⭐ 1,142

50+ DockerHub public images for Docker & Kubernetes - DevOps, CI/CD, GitHub Actions, CircleCI, Jenkins, TeamCity, Alpine, CentOS, Debian, Fedora, Ubuntu, Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak

Bigdata Growth ⭐ 907

大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

Kafka Storm Starter ⭐ 729

[PROJECT IS NO LONGER MAINTAINED] Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

Pkpmspark ⭐ 697

awesome 三维数据挖掘数据分析 & 推荐

Kafka Spark Consumer ⭐ 616

High Performance Kafka Connector for Spark Streaming.Supports Multi Topic Fetch, Kafka Security. Reliable offset management in Zookeeper. No Data-loss. No dependency on HDFS and WAL. In-built PID rate controller. Support Message Handler . Offset Lag checker.

Freestyle ⭐ 615

A cohesive & pragmatic framework of FP centric Scala libraries

Real Time Analytics and Data Pipelines based on Spark Streaming

Data Engineering Interview Questions ⭐ 449

More than 2000+ Data engineer interview questions.

Agile_data_code_2 ⭐ 435

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

Zeek Analysis Tools (ZAT): Processing and analysis of Zeek network data with Pandas, scikit-learn, Kafka and Spark

Ecommercerecommendsystem ⭐ 350

商品大数据实时推荐系统。前端：Vue + TypeScript + ElementUI，后端 Spring + Spark

Wirbelsturm ⭐ 333

[PROJECT IS NO LONGER MAINTAINED] Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.

Every Single Day I Tldr ⭐ 304

A daily digest of the articles or videos I've found interesting, that I want to share with you.

Data Accelerator ⭐ 286

Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.

Zdh_web ⭐ 282

大数据采集,抽取平台

Demo_11.11_storm Spark Hadoop ⭐ 257

hadoop_storm_spark结合实验的例子，模拟淘宝双11节，根据订单详细信息，汇总出总销售 --------大概流程------- 第一阶段（storm实时报表）第二阶段（离线报表）第三阶段（大规模订单即席查询,和多维度查询）第四阶段（数据挖掘和图计算）

Sparkstreaming ⭐ 253

Spark Streaming+Flume+Kafka+HBase+Hadoop+Zookeeper实现实时日志

Kafka Exactly Once ⭐ 245

Big Data Processing Framework - Unified Data API or SQL on Any Storage

Azure Event Hubs Spark ⭐ 225

Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

Avro SerDe for Apache Spark structured APIs.

Video Stream Analytics ⭐ 191

Big Data ⭐ 190

一个开源、成体系的大数据学习教程。spark学习 hadoop hive hbase flink教程 linux 从入门到精通

Wifiprobeanalysis ⭐ 189

基于WIFI探针的商业大数据分析技术

Sparkstreaming ⭐ 179

💥 🚀 封装sparkstreaming动态调节batch time(有数据就执行计算)；🚀 支持运行过程中增删topic；🚀 封装sparkstreaming 1.6 - kafka 010 用以支持 SSL。

Spark Kafka Writer ⭐ 177

Write your Spark data to Kafka seamlessly

A distributed log store based on raft

Kafka Book ⭐ 167

《Kafka技术内幕》代码

Dcos Commons ⭐ 162

DC/OS SDK is a collection of tools, libraries, and documentation for easy integration of technologies such as Kafka, Cassandra, HDFS, Spark, and TensorFlow with DC/OS.

Spark Streaming With Kafka ⭐ 161

Self-contained examples of Apache Spark streaming integrated with Apache Kafka.

Aliyun Emapreduce Datasources ⭐ 157

Extended datasource support for Spark/Hadoop on Aliyun E-MapReduce.

Bigdata In Practice ⭐ 154

大数据实践项目 Hadoop、Spark、Kafka、Hbase、Flink.....

Spark Structured Streaming Examples ⭐ 153

Spark Structured Streaming / Kafka / Cassandra / Elastic

Lambda Arch ⭐ 151

A full big data pipeline (Lambda Architecture) with Spark, Kafka, HDFS and Cassandra.

Bigdata Learning ⭐ 136

大数据学习记录

Spark Kafka ⭐ 136

Low level integration of Spark and Kafka

Logvision ⭐ 136

分布式实时日志分析与入侵检测系统

Huaweicloud Mrs Example ⭐ 128

Examples for HUAWEI CLOUD MRS.

Iot Traffic Monitor ⭐ 123

Example Spark Kafka ⭐ 118

Apache Spark and Apache Kafka integration example

Xichuan_note ⭐ 114

xichuan的学习总结笔记,覆盖了java、spring、java其他常用框架,以及大数据相关组件

Spark Atlas Connector ⭐ 112

A Spark Atlas connector to track data lineage in Apache Atlas

Yelper_recommendation_system ⭐ 108

Yelper recommendation system

Dcos Iot Demo ⭐ 106

This project demonstrates how to configure a full stack geo-enabled Internet of Things (IoT) solution using Mesosphere's open sourced Data Center Operating System (DC/OS) using Docker containerization and frameworks for Mesos including Marathon, Kafka, Spark, and Elasticsearch.

Logisland ⭐ 106

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.

Streamify ⭐ 97

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Medium Articles ⭐ 97

Repo for all my code on the articles I post on medium

Kafka Spark Streaming Example ⭐ 96

Simple examle for Spark Streaming over Kafka topic

Learning Spark ⭐ 94

Practical examples of using Apache Spark in several different use cases

Blog Spark Streaming Log Aggregation ⭐ 91

Example of use of Spark Streaming with Kafka

Qstreaming ⭐ 89

A simplified, lightweight ETL pipeline framework for build stream/batch processing applications on top of Apache Spark

基于JavaScript Engine的轻量级规则引擎系统，重构于开源IOT项目thingboard

Kafka Sparkstreaming Cassandra ⭐ 86

Docker container for Kafka - Spark Streaming - Cassandra

Twitter Sentiment Analysis Using Spark Streaming And Kafka ⭐ 78

Twitter Sentiment Analysis using Spark and Kafka

Generator Mitosis ⭐ 75

A micro-service infrastructure generator based on Yeoman/Chatbot, Kubernetes/Docker Swarm, Traefik, Ansible, Jenkins, Spark, Hadoop, Kafka, etc.

Kafka_spark_hbase_demo ⭐ 72

kafka spark hbase 日志统计

本项目已废弃，笔记收藏整理参考：

Google Finance Stock Data Analysis ⭐ 69

Developed a high performance data processing platform using Apache Kafka, Apache Cassandra, and Apache Spark to analyze stock price and related stock tweets sentiment.

Kafka Spark Streaming Zeppelin Docker ⭐ 68

One click deploy docker-compose with Kafka, Spark Streaming, Zeppelin UI and Monitoring (Grafana + Kafka Manager)

Spark_als ⭐ 68

基于spark-ml,spark-mllib,spark-streaming的推荐算法实现

Spark Structured Streaming ⭐ 66

Spark structured streaming with Kafka data source and writing to Cassandra

User Guide Smack ⭐ 66

[Cloudframeworks]SMACK Big Data Architecture - user guide ／ [云框架]SMACK大数据架构-用户指南

Delta Architecture ⭐ 66

Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline

Docker hello world templates

Spark Kafka Cassandra Applying Lambda Architecture ⭐ 64

Lambda Arch Spark ⭐ 63

Data Processing Pipeline ⭐ 59

Real-Time Data Processing Pipeline & Visualization with Docker, Spark, Kafka and Cassandra

Busfloatingdata ⭐ 59

Showcase for IoT Platform Blog

Bigdata Hub ⭐ 58

数据建设与大数据技术知识体系，包含hadoop、hive、spark、flink主流框架和系列框架，

Recommendmoteur ⭐ 57

电影推荐系统、电影推荐引擎、使用Spark完成的电影推荐引擎

Titandataoperationsystem ⭐ 57

最好的大数据项目。《Titan数据运营系统》，本项目是一个全栈闭环系统，我们有用作数据可视化的web Echart等；

Datapipeline ⭐ 57

Real time stock data pipeline --play with Kafka, Cassandra, Spark, Redis, Node.js, Zookeeper

Kafka Streaming Click Analysis ⭐ 56

Use Kafka and Apache Spark streaming to perform click stream analytics

Pybigdata ⭐ 56

使用 python 操作大数据的各种组件

Bigdataparty ⭐ 54

大数据组件 All-in-One 的 Dockerfile

Data Stream Development With Apache Spark Kafka And Spring Boot ⭐ 54

Data Stream Development with Apache Spark, Kafka and Spring Boot by Packt Publishing

Awesome Pulsar ⭐ 53

A curated list of Pulsar tools, integrations and resources.

Cloud Bigdata Book ⭐ 53

Model Serving Tutorial ⭐ 53

Code and presentation for Strata Model Serving tutorial

Fb_scraper ⭐ 52

FBLYZE is a Facebook scraping system and analysis system.

Platys Modern Data Platform ⭐ 51

Support for generating modern platforms dynamically with services such as Kafka, Spark, Streamsets, HDFS, ....

Spark Cep ⭐ 51

Spark CEP is an extension of Spark Streaming to support SQL-based query processing

Movie Recommender Demo ⭐ 50

This project walks through how you can create recommendations using Apache Spark machine learning. There are a number of jupyter notebooks that you can run on IBM Data Science Experience, and there a live demo of a movie recommendation web application you can interact with. The demo also uses IBM Message Hub (kafka) to push application events to topic where they are consumed by a spark streaming job running on IBM BigInsights (hadoop).

Machine learning around business processes

Related Searches

Scala Spark (3,299)

Java Kafka (3,237)

Python Spark (2,035)

Java Spark (1,596)

Kafka Zookeeper (1,229)

Spark Hadoop (1,199)

Jupyter Notebook Spark (1,151)

Docker Kafka (1,106)

Python Kafka (1,097)

Scala Kafka (969)

1-100 of 631 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2023 Awesome Open Source. All rights reserved.