Awesome Open Source
Awesome Open Source
Combined Topics
spark x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 351 Spark Open Source Projects

Spark ⭐ 29,357
Apache Spark - A unified analytics engine for large-scale data processing
Data Science Ipython Notebooks ⭐ 20,952
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
Redash ⭐ 18,556
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
Docker_practice ⭐ 18,520
Learn and understand Docker technologies, with real DevOps practice!
Deeplearning4j ⭐ 12,017
Model import deployment framework for retraining models (pytorch, tensorflow,keras) deploying in JVM Micro service environments, mobile devices, iot, and Apache Spark
Technology Talk ⭐ 11,459
汇总java生态圈常用技术框架、开源中间件，系统架构、数据库、大公司架构案例、常用三方类库、项目管理、线上问题排查、个人成长、思考等知识
Horovod ⭐ 11,096
Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.
Cube.js ⭐ 10,154
📊 Cube.js — Open-Source Analytical API Platform
Flink Learning ⭐ 9,752
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Bigdata Notes ⭐ 9,389
大数据入门指南 ⭐️
Ds Cheatsheets ⭐ 8,573
List of Data Science Cheatsheets to rule the world
Thingsboard ⭐ 8,421
Open-source IoT Platform - Device management, data collection, processing and visualization.
Angel ⭐ 6,183
A Flexible and Powerful Parameter Server for large-scale machine learning
Pdf ⭐ 5,464
编程电子书，电子书，编程书籍，包括C，C#，Docker，Elasticsearch，Git，Hadoop，HeadFirst，Java，Javascript，jvm，Kafka，Linux，Maven，MongoDB，MyBatis，MySQL，Netty，Nginx，Python，RabbitMQ，Redis，Scala，Solr，Spark，Spring，SpringBoot，SpringCloud，TCPIP，Tomcat，Zookeeper，人工智能，大数据类，并发编程，数据库类，数据挖掘，新面试题，架构设计，算法系列，计算机类，设计模式，软件测试，重构优化，等更多分类
Dev Setup ⭐ 5,379
macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.
H2o 3 ⭐ 5,299
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Zeppelin ⭐ 5,208
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Alluxio ⭐ 5,044
Alluxio, data orchestration for analytics and machine learning in the cloud
God Of Bigdata ⭐ 4,364
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...
Pipeline ⭐ 4,106
PipelineAI Kubeflow Distribution
Bigdl ⭐ 3,714
BigDL: Distributed Deep Learning Framework for Apache Spark
Tensorflowonspark ⭐ 3,656
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
Coolplayspark ⭐ 3,220
酷玩 Spark: Spark 源代码解析、Spark 类库等
Delta ⭐ 3,213
An open-source storage layer that brings scalable, ACID transactions to Apache Spark™ and big data workloads.
Spark Notebook ⭐ 3,026
Interactive and Reactive Data Science using Scala and Spark.
Helk ⭐ 2,826
The Hunting ELK
Koalas ⭐ 2,745
Koalas: pandas API on Apache Spark
Spark Jobserver ⭐ 2,688
REST job server for Apache Spark
Dpark ⭐ 2,658
Python clone of Spark, a MapReduce alike framework in Python
Mmlspark ⭐ 2,303
Microsoft Machine Learning for Apache Spark
Ballista ⭐ 2,249
Distributed compute platform implemented in Rust, and powered by Apache Arrow.
Roaringbitmap ⭐ 2,180
A better compressed bitset in Java
Spark Nlp ⭐ 2,055
State of the Art Natural Language Processing
Transmogrifai ⭐ 2,002
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
Linkis ⭐ 1,935
Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.
Quill ⭐ 1,838
Compile-time Language Integrated Queries for Scala
Spark Ml Source Analysis ⭐ 1,818
spark ml 算法原理剖析以及具体的源码实现分析
Benchm Ml ⭐ 1,812
A minimal benchmark for scalability, speed and accuracy of commonly used open source implementations (R packages, Python scikit-learn, H2O, xgboost, Spark MLlib etc.) of the top machine learning algorithms for binary classification (random forests, gradient boosted trees, deep neural networks etc.).
Spark Cassandra Connector ⭐ 1,781
DataStax Spark Cassandra Connector
Nd4j ⭐ 1,756
Fast, Scientific and Numerical Computing for the JVM (NDArrays)
Quicksql ⭐ 1,694
A Flexible, Fast, Federated(3F) SQL Analysis Middleware for Multiple Data Sources
Feast ⭐ 1,692
Feature Store for Machine Learning
Spark ⭐ 1,626
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Spring Boot Quick ⭐ 1,620
🌿 基于springboot的快速学习示例,整合自己遇到的开源框架,如：rabbitmq(延迟队列)、Kafka、jpa、redies、oauth2、swagger、jsp、docker、spring-batch、异常处理、日志输出、多模块开发、多环境打包、缓存cache、爬虫、jwt、GraphQL、dubbo、zookeeper和Async等等📌
Gaffer ⭐ 1,613
A large-scale entity and relation database supporting aggregation of properties
Deequ ⭐ 1,579
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Elassandra ⭐ 1,541
Elassandra = Elasticsearch + Apache Cassandra
Waterdrop ⭐ 1,471
生产环境的海量数据计算产品，文档地址：
Elephas ⭐ 1,462
Distributed Deep learning with Keras & Spark
Spark On K8s Operator ⭐ 1,440
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Seldon Server ⭐ 1,427
Machine Learning Platform and Recommendation Engine built on Kubernetes
Almond ⭐ 1,362
A Scala kernel for Jupyter
Spark Py Notebooks ⭐ 1,348
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
Dataspherestudio ⭐ 1,247
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Mleap ⭐ 1,241
MLeap: Deploy ML Pipelines to Production
Home ⭐ 1,206
ApacheCN 开源组织：公告、介绍、成员、活动、交流方式
Carbondata ⭐ 1,167
High performance data store solution
Spark Doc Zh ⭐ 1,135
Apache Spark 官方文档中文版
Utils4s ⭐ 1,071
scala、spark使用过程中，各种测试用例以及相关资料整理
Apache Spark Internals ⭐ 1,046
The Internals of Apache Spark
Gatk ⭐ 1,020
Official code repository for GATK versions 4 and up
Pixiedust ⭐ 1,003
Python Helper library for Jupyter Notebooks
Snappydata ⭐ 999
Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster
Optimus ⭐ 996
🚚 Agile Data Preparation Workflows made easy with pandas, dask, cudf, dask_cudf and pyspark
Sparkmagic ⭐ 957
Jupyter magics and kernels for working with remote Spark clusters
Data Algorithms Book ⭐ 950
MapReduce, Spark, Java, and Scala for Data Algorithms Book
Mobius ⭐ 929
C# and F# language binding and extensions to Apache Spark
Kylo ⭐ 923
Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.
Spark Scala Tutorial ⭐ 910
A free tutorial for Apache Spark.
Bigdata Interview ⭐ 907
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结
Sparkling Water ⭐ 888
Sparkling Water provides H2O functionality inside Spark cluster
Flint ⭐ 885
A Time Series Library for Apache Spark
Szt Bigdata ⭐ 867
深圳地铁大数据客流分析系统🚇🚄🌟
Dockerfiles ⭐ 855
50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak, TeamCity and DevOps tools built on the major Linux distros: Alpine, CentOS, Debian, Fedora, Ubuntu
Bigdataguide ⭐ 846
大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料
Goodreads_etl_pipeline ⭐ 798
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Spark Redis ⭐ 783
A connector for Spark that allows reading and writing to/from Redis cluster
Sparklyr ⭐ 779
R interface for Apache Spark
Coding Now ⭐ 771
学习记录的一些笔记，以及所看得一些电子书eBooks、视频资源和平常收纳的一些自己认为比较好的博客、网站、工具。涉及大数据几大组件、Python机器学习和数据分析、Linux、操作系统、算法、网络等
Sparkctr ⭐ 752
CTR prediction model based on spark(LR, GBDT, DNN)
Spark Movie Lens ⭐ 748
An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset
Cdhproject ⭐ 743
hadoop各组件使用，持续更新
Kafka Storm Starter ⭐ 728
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Frameless ⭐ 722
Expressive types for Spark.
Elasticsearch Spark Recommender ⭐ 713
Use Jupyter Notebooks to demonstrate how to build a Recommender with Apache Spark & Elasticsearch
Hail ⭐ 712
Scalable genomic data analysis.
Scriptis ⭐ 702
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
Useractionanalyzeplatform ⭐ 656
电商用户行为分析大数据平台
Pyspark Example Project ⭐ 648
Example project implementing best practices for PySpark ETL jobs and applications.
Freestyle ⭐ 625
A cohesive & pragmatic framework of FP centric Scala libraries
Datafusion ⭐ 614
DataFusion has now been donated to the Apache Arrow project
Mongo Spark ⭐ 591
The MongoDB Spark Connector
Spark Daria ⭐ 564
Essential Spark extensions and helper methods ✨😲
Sparklearning ⭐ 560
Learning Apache spark,including code and data .Most part can run local.
Justenoughscalaforspark ⭐ 549
A tutorial on the most important features and idioms of Scala that you need to use Spark's Scala APIs.
Lopq ⭐ 530
Training of Locally Optimized Product Quantization (LOPQ) models for approximate nearest neighbor search of high dimensional data in Python and Spark.
Cdap ⭐ 518
An open source framework for building data analytic applications.
Sparta ⭐ 516
Real Time Analytics and Data Pipelines based on Spark Streaming
Magellan ⭐ 508
Geo Spatial Data Analytics on Spark
Bdp Dataplatform ⭐ 489
大数据生态解决方案数据平台：基于大数据、数据平台、微服务、机器学习、商城、自动化运维、DevOps、容器部署平台、数据平台采集、数据平台存储、数据平台计算、数据平台开发、数据平台应用搭建的大数据解决方案。
1-100 of 351 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210