Awesome Open Source
Awesome Open Source
Combined Topics
spark x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 2,845 Spark Open Source Projects on Github

Spark ⭐ 31,919
Apache Spark - A unified analytics engine for large-scale data processing
Data Science Ipython Notebooks ⭐ 21,791
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
Redash ⭐ 20,361
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
Docker_practice ⭐ 19,840
Learn and understand Docker technologies, with real DevOps practice!
Deeplearning4j ⭐ 12,333
Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learning using automatic differentiation.
Cube.js ⭐ 12,235
📊 Cube — Open-Source Analytics API for Building Data Apps
Pdf ⭐ 12,224
编程电子书，电子书，编程书籍，包括C，C#，Docker，Elasticsearch，Git，Hadoop，HeadFirst，Java，Javascript，jvm，Kafka，Linux，Maven，MongoDB，MyBatis，MySQL，Netty，Nginx，Python，RabbitMQ，Redis，Scala，Solr，Spark，Spring，SpringBoot，SpringCloud，TCPIP，Tomcat，Zookeeper，人工智能，大数据类，并发编程，数据库类，数据挖掘，新面试题，架构设计，算法系列，计算机类，设计模式，软件测试，重构优化，等更多分类
Horovod ⭐ 12,080
Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.
Technology Talk ⭐ 12,011
汇总java生态圈常用技术框架、开源中间件，系统架构、数据库、大公司架构案例、常用三方类库、项目管理、线上问题排查、个人成长、思考等知识
Flink Learning ⭐ 11,466
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Bigdata Notes ⭐ 9,567
大数据入门指南 ⭐️
Ds Cheatsheets ⭐ 7,129
List of Data Science Cheatsheets to rule the world
Angel ⭐ 6,412
A Flexible and Powerful Parameter Server for large-scale machine learning
God Of Bigdata ⭐ 5,853
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...
H2o 3 ⭐ 5,699
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Zeppelin ⭐ 5,557
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Alluxio ⭐ 5,436
Alluxio, data orchestration for analytics and machine learning in the cloud
Dev Setup ⭐ 5,254
macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.
Pipeline ⭐ 4,113
PipelineAI Kubeflow Distribution
Delta ⭐ 3,976
An open-source storage layer that brings scalable, ACID transactions to Apache Spark™ and big data workloads.
Bigdl ⭐ 3,827
Building Large-Scale AI Applications for Distributed Big Data
Tensorflowonspark ⭐ 3,748
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
Coolplayspark ⭐ 3,277
酷玩 Spark: Spark 源代码解析、Spark 类库等
Synapseml ⭐ 3,094
Simple and Distributed Machine Learning
Spark Notebook ⭐ 3,031
Interactive and Reactive Data Science using Scala and Spark.
Koalas ⭐ 2,999
Koalas: pandas API on Apache Spark
Helk ⭐ 2,861
The Hunting ELK
Spark Jobserver ⭐ 2,735
REST job server for Apache Spark
Dpark ⭐ 2,637
Python clone of Spark, a MapReduce alike framework in Python
Spark Nlp ⭐ 2,587
State of the Art Natural Language Processing
Roaringbitmap ⭐ 2,477
A better compressed bitset in Java
Incubator Linkis ⭐ 2,397
Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.
Ballista ⭐ 2,244
Distributed compute platform implemented in Rust, and powered by Apache Arrow.
Transmogrifai ⭐ 2,061
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
Deequ ⭐ 2,020
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Zio Quill ⭐ 2,010
Compile-time Language Integrated Queries for Scala
Dataspherestudio ⭐ 1,854
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Quicksql ⭐ 1,840
A Flexible, Fast, Federated(3F) SQL Analysis Middleware for Multiple Data Sources
Spring Boot Quick ⭐ 1,839
🌿 基于springboot的快速学习示例,整合自己遇到的开源框架,如：rabbitmq(延迟队列)、Kafka、jpa、redies、oauth2、swagger、jsp、docker、spring-batch、异常处理、日志输出、多模块开发、多环境打包、缓存cache、爬虫、jwt、GraphQL、dubbo、zookeeper和Async等等📌
Spark Cassandra Connector ⭐ 1,823
DataStax Spark Cassandra Connector
Spark On K8s Operator ⭐ 1,817
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Benchm Ml ⭐ 1,766
A minimal benchmark for scalability, speed and accuracy of commonly used open source implementations (R packages, Python scikit-learn, H2O, xgboost, Spark MLlib etc.) of the top machine learning algorithms for binary classification (random forests, gradient boosted trees, deep neural networks etc.).
Spark ⭐ 1,728
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Spark Ml Source Analysis ⭐ 1,710
spark ml 算法原理剖析以及具体的源码实现分析
Ibis ⭐ 1,656
A pandas-like deferred expression system, with first-class SQL support
Gaffer ⭐ 1,648
A large-scale entity and relation database supporting aggregation of properties
Elassandra ⭐ 1,610
Elassandra = Elasticsearch + Apache Cassandra
Elephas ⭐ 1,494
Distributed Deep learning with Keras & Spark
Seldon Server ⭐ 1,420
Machine Learning Platform and Recommendation Engine built on Kubernetes
Almond ⭐ 1,411
A Scala kernel for Jupyter
Mleap ⭐ 1,349
MLeap: Deploy ML Pipelines to Production
Bigdataguide ⭐ 1,349
大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料
Home ⭐ 1,339
ApacheCN 开源组织：公告、介绍、成员、活动、交流方式
Carbondata ⭐ 1,295
High performance data store solution
Spark Py Notebooks ⭐ 1,227
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
Optimus ⭐ 1,172
🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Gatk ⭐ 1,162
Official code repository for GATK versions 4 and up
Apache Spark Internals ⭐ 1,146
The Internals of Apache Spark
Spark Doc Zh ⭐ 1,101
Apache Spark 官方文档中文版
Sparkmagic ⭐ 1,063
Jupyter magics and kernels for working with remote Spark clusters
Bigdata Interview ⭐ 1,041
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结
Utils4s ⭐ 1,033
scala、spark使用过程中，各种测试用例以及相关资料整理
Pixiedust ⭐ 1,030
Python Helper library for Jupyter Notebooks
Snappydata ⭐ 1,010
Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster
Kylo ⭐ 977
Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.
Data Algorithms Book ⭐ 973
MapReduce, Spark, Java, and Scala for Data Algorithms Book
Dockerfiles ⭐ 966
50+ DockerHub public images for Docker & Kubernetes - DevOps, GitHub Actions, CircleCI, Jenkins, TeamCity, Alpine, CentOS, Debian, Fedora, Ubuntu, Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak
Mobius ⭐ 930
C# and F# language binding and extensions to Apache Spark
Spark Scala Tutorial ⭐ 922
A free tutorial for Apache Spark.
Sparkling Water ⭐ 916
Sparkling Water provides H2O functionality inside Spark cluster
Adam ⭐ 914
ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.
Incubator Kyuubi ⭐ 903
Apache Kyuubi is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark
Szt Bigdata ⭐ 892
深圳地铁大数据客流分析系统🚇🚄🌟
Coding Now ⭐ 885
学习记录的一些笔记，以及所看得一些电子书eBooks、视频资源和平常收纳的一些自己认为比较好的博客、网站、工具。涉及大数据几大组件、Python机器学习和数据分析、Linux、操作系统、算法、网络等
Spark Redis ⭐ 830
A connector for Spark that allows reading and writing to/from Redis cluster
Sparklyr ⭐ 819
R interface for Apache Spark
Around Dataengineering ⭐ 815
A Data Engineering & Machine Learning Knowledge Hub
Flint ⭐ 796
A Time Series Library for Apache Spark
Streamx ⭐ 786
Flink & Spark development scaffold, Make stream processing easier!!! The original intention of StreamX is to make the development of Flink easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.
Frameless ⭐ 786
Expressive types for Spark.
Hail ⭐ 775
Scalable genomic data analysis.
Sparkctr ⭐ 769
CTR prediction model based on spark(LR, GBDT, DNN)
Spark Movie Lens ⭐ 757
An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset
Pyspark Example Project ⭐ 727
Example project implementing best practices for PySpark ETL jobs and applications.
Kafka Storm Starter ⭐ 725
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Tutorial ⭐ 718
Java全栈知识架构体系总结
Scriptis ⭐ 694
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
Bdp Platform ⭐ 653
大数据生态解决方案数据平台：基于大数据、数据平台、微服务、机器学习、商城、自动化运维、DevOps、容器部署平台、数据平台采集、数据平台存储、数据平台计算、数据平台开发、数据平台应用搭建的大数据解决方案。
Mongo Spark ⭐ 633
The MongoDB Spark Connector
Freestyle ⭐ 615
A cohesive & pragmatic framework of FP centric Scala libraries
Spark Daria ⭐ 606
Essential Spark extensions and helper methods ✨😲
Elasticsearch Spark Recommender ⭐ 603
Use Jupyter Notebooks to demonstrate how to build a Recommender with Apache Spark & Elasticsearch
Cdap ⭐ 601
An open source framework for building data analytic applications.
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Datafusion ⭐ 584
DataFusion has now been donated to the Apache Arrow project
Useractionanalyzeplatform ⭐ 581
电商用户行为分析大数据平台
Sparklearning ⭐ 573
Learning Apache spark,including code and data .Most part can run local.
Learningsparkv2 ⭐ 570
This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
Dji Firmware Tools ⭐ 560
Tools for handling firmwares of DJI products, with focus on quadcopters.
Justenoughscalaforspark ⭐ 545
A tutorial on the most important features and idioms of Scala that you need to use Spark's Scala APIs.
1-100 of 2,845 projects

Related Projects

Scala Spark Projects (1,044)
Python Spark Projects (713)
Java Spark Projects (527)
Spark Big Data Projects (424)
Spark Hadoop Projects (417)
Jupyter Notebook Spark Projects (401)
Machine Learning Spark Projects (369)
Kafka Spark Projects (338)
Spark Pyspark Projects (300)
Docker Spark Projects (230)
Python Spark Pyspark Projects (192)
Spark Hive Projects (177)
Data Science Spark Projects (167)
Scala Spark Big Data Projects (154)
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
Privacy policy
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2022