Awesome Open Source
Awesome Open Source
Combined Topics
spark x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 369 Spark Open Source Projects

Spark ⭐ 30,441
Apache Spark - A unified analytics engine for large-scale data processing
Data Science Ipython Notebooks ⭐ 21,372
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
Redash ⭐ 19,199
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
Docker_practice ⭐ 19,057
Learn and understand Docker technologies, with real DevOps practice!
Deeplearning4j ⭐ 12,130
Model import deployment framework for retraining models (pytorch, tensorflow,keras) deploying in JVM Micro service environments, mobile devices, iot, and Apache Spark
Technology Talk ⭐ 11,760
汇总java生态圈常用技术框架、开源中间件，系统架构、数据库、大公司架构案例、常用三方类库、项目管理、线上问题排查、个人成长、思考等知识
Horovod ⭐ 11,484
Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.
Cube.js ⭐ 10,823
📊 Cube.js — Open-Source Analytical API Platform
Flink Learning ⭐ 10,388
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Bigdata Notes ⭐ 10,024
大数据入门指南 ⭐️
Ds Cheatsheets ⭐ 9,007
List of Data Science Cheatsheets to rule the world
Thingsboard ⭐ 8,959
Open-source IoT Platform - Device management, data collection, processing and visualization.
Pdf ⭐ 8,718
编程电子书，电子书，编程书籍，包括C，C#，Docker，Elasticsearch，Git，Hadoop，HeadFirst，Java，Javascript，jvm，Kafka，Linux，Maven，MongoDB，MyBatis，MySQL，Netty，Nginx，Python，RabbitMQ，Redis，Scala，Solr，Spark，Spring，SpringBoot，SpringCloud，TCPIP，Tomcat，Zookeeper，人工智能，大数据类，并发编程，数据库类，数据挖掘，新面试题，架构设计，算法系列，计算机类，设计模式，软件测试，重构优化，等更多分类
Angel ⭐ 6,322
A Flexible and Powerful Parameter Server for large-scale machine learning
Dev Setup ⭐ 5,476
macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.
H2o 3 ⭐ 5,459
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Zeppelin ⭐ 5,344
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Alluxio ⭐ 5,170
Alluxio, data orchestration for analytics and machine learning in the cloud
God Of Bigdata ⭐ 4,937
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...
Pipeline ⭐ 4,113
PipelineAI Kubeflow Distribution
Bigdl ⭐ 3,752
BigDL: Distributed Deep Learning Framework for Apache Spark
Tensorflowonspark ⭐ 3,694
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
Delta ⭐ 3,491
An open-source storage layer that brings scalable, ACID transactions to Apache Spark™ and big data workloads.
Coolplayspark ⭐ 3,271
酷玩 Spark: Spark 源代码解析、Spark 类库等
Spark Notebook ⭐ 3,036
Interactive and Reactive Data Science using Scala and Spark.
Helk ⭐ 2,934
The Hunting ELK
Koalas ⭐ 2,880
Koalas: pandas API on Apache Spark
Spark Jobserver ⭐ 2,717
REST job server for Apache Spark
Dpark ⭐ 2,662
Python clone of Spark, a MapReduce alike framework in Python
Mmlspark ⭐ 2,381
Microsoft Machine Learning for Apache Spark
Roaringbitmap ⭐ 2,322
A better compressed bitset in Java
Ballista ⭐ 2,251
Distributed compute platform implemented in Rust, and powered by Apache Arrow.
Spark Nlp ⭐ 2,246
State of the Art Natural Language Processing
Feast ⭐ 2,057
Feature Store for Machine Learning
Linkis ⭐ 2,057
Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.
Transmogrifai ⭐ 2,028
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
Quill ⭐ 1,892
Compile-time Language Integrated Queries for Scala
Spark Ml Source Analysis ⭐ 1,846
spark ml 算法原理剖析以及具体的源码实现分析
Benchm Ml ⭐ 1,825
A minimal benchmark for scalability, speed and accuracy of commonly used open source implementations (R packages, Python scikit-learn, H2O, xgboost, Spark MLlib etc.) of the top machine learning algorithms for binary classification (random forests, gradient boosted trees, deep neural networks etc.).
Deequ ⭐ 1,811
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Spark Cassandra Connector ⭐ 1,800
DataStax Spark Cassandra Connector
Nd4j ⭐ 1,783
Fast, Scientific and Numerical Computing for the JVM (NDArrays)
Quicksql ⭐ 1,755
A Flexible, Fast, Federated(3F) SQL Analysis Middleware for Multiple Data Sources
Spring Boot Quick ⭐ 1,730
🌿 基于springboot的快速学习示例,整合自己遇到的开源框架,如：rabbitmq(延迟队列)、Kafka、jpa、redies、oauth2、swagger、jsp、docker、spring-batch、异常处理、日志输出、多模块开发、多环境打包、缓存cache、爬虫、jwt、GraphQL、dubbo、zookeeper和Async等等📌
Spark ⭐ 1,668
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Gaffer ⭐ 1,625
A large-scale entity and relation database supporting aggregation of properties
Waterdrop ⭐ 1,623
生产环境的海量数据计算产品，文档地址：
Elassandra ⭐ 1,575
Elassandra = Elasticsearch + Apache Cassandra
Spark On K8s Operator ⭐ 1,569
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Elephas ⭐ 1,490
Distributed Deep learning with Keras & Spark
Dataspherestudio ⭐ 1,444
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Seldon Server ⭐ 1,430
Machine Learning Platform and Recommendation Engine built on Kubernetes
Spark Py Notebooks ⭐ 1,383
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
Almond ⭐ 1,382
A Scala kernel for Jupyter
Mleap ⭐ 1,273
MLeap: Deploy ML Pipelines to Production
Home ⭐ 1,248
ApacheCN 开源组织：公告、介绍、成员、活动、交流方式
Carbondata ⭐ 1,226
High performance data store solution
Spark Doc Zh ⭐ 1,152
Apache Spark 官方文档中文版
Apache Spark Internals ⭐ 1,097
The Internals of Apache Spark
Gatk ⭐ 1,074
Official code repository for GATK versions 4 and up
Utils4s ⭐ 1,074
scala、spark使用过程中，各种测试用例以及相关资料整理
Bigdataguide ⭐ 1,056
大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料
Optimus ⭐ 1,038
🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF and PySpark
Pixiedust ⭐ 1,019
Python Helper library for Jupyter Notebooks
Sparkmagic ⭐ 1,015
Jupyter magics and kernels for working with remote Spark clusters
Szt Bigdata ⭐ 1,013
深圳地铁大数据客流分析系统🚇🚄🌟
Bigdata Interview ⭐ 1,008
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结
Snappydata ⭐ 1,007
Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster
Data Algorithms Book ⭐ 959
MapReduce, Spark, Java, and Scala for Data Algorithms Book
Kylo ⭐ 951
Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.
Mobius ⭐ 931
C# and F# language binding and extensions to Apache Spark
Spark Scala Tutorial ⭐ 924
A free tutorial for Apache Spark.
Sparkling Water ⭐ 900
Sparkling Water provides H2O functionality inside Spark cluster
Flint ⭐ 898
A Time Series Library for Apache Spark
Dockerfiles ⭐ 892
50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak, TeamCity and DevOps tools built on the major Linux distros: Alpine, CentOS, Debian, Fedora, Ubuntu
Goodreads_etl_pipeline ⭐ 843
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Coding Now ⭐ 833
学习记录的一些笔记，以及所看得一些电子书eBooks、视频资源和平常收纳的一些自己认为比较好的博客、网站、工具。涉及大数据几大组件、Python机器学习和数据分析、Linux、操作系统、算法、网络等
Spark Redis ⭐ 803
A connector for Spark that allows reading and writing to/from Redis cluster
Sparkctr ⭐ 800
CTR prediction model based on spark(LR, GBDT, DNN)
Sparklyr ⭐ 798
R interface for Apache Spark
Cdhproject ⭐ 765
hadoop各组件使用，持续更新
Spark Movie Lens ⭐ 757
An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset
Frameless ⭐ 741
Expressive types for Spark.
Hail ⭐ 737
Scalable genomic data analysis.
Kafka Storm Starter ⭐ 729
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Elasticsearch Spark Recommender ⭐ 729
Use Jupyter Notebooks to demonstrate how to build a Recommender with Apache Spark & Elasticsearch
Scriptis ⭐ 724
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
Pyspark Example Project ⭐ 721
Example project implementing best practices for PySpark ETL jobs and applications.
Useractionanalyzeplatform ⭐ 683
电商用户行为分析大数据平台
Datafusion ⭐ 622
DataFusion has now been donated to the Apache Arrow project
Freestyle ⭐ 620
A cohesive & pragmatic framework of FP centric Scala libraries
Incubator Kyuubi ⭐ 607
Apache Kyuubi is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark
Mongo Spark ⭐ 606
The MongoDB Spark Connector
Spark Daria ⭐ 600
Essential Spark extensions and helper methods ✨😲
Tutorial ⭐ 583
Java全栈知识架构体系总结
Bdp Platform ⭐ 576
大数据生态解决方案数据平台：基于大数据、数据平台、微服务、机器学习、商城、自动化运维、DevOps、容器部署平台、数据平台采集、数据平台存储、数据平台计算、数据平台开发、数据平台应用搭建的大数据解决方案。
Justenoughscalaforspark ⭐ 575
A tutorial on the most important features and idioms of Scala that you need to use Spark's Scala APIs.
Sparklearning ⭐ 566
Learning Apache spark,including code and data .Most part can run local.
Cdap ⭐ 552
An open source framework for building data analytic applications.
Lopq ⭐ 532
Training of Locally Optimized Product Quantization (LOPQ) models for approximate nearest neighbor search of high dimensional data in Python and Spark.
1-100 of 369 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210