Awesome Open Source
Awesome Open Source
Combined Topics
spark x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 334 Spark Open Source Projects

Spark ⭐ 28,592
Apache Spark - A unified analytics engine for large-scale data processing
Data Science Ipython Notebooks ⭐ 20,090
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
Docker_practice ⭐ 18,116
Learn and understand Docker technologies, with real DevOps practice!
Redash ⭐ 18,044
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
Deeplearning4j ⭐ 11,922
Eclipse Deeplearning4j, ND4J, DataVec and more - deep learning & linear algebra for Java/Scala with GPUs + Spark
Technology Talk ⭐ 11,227
汇总java生态圈常用技术框架、开源中间件，系统架构、数据库、大公司架构案例、常用三方类库、项目管理、线上问题排查、个人成长、思考等知识
Horovod ⭐ 10,724
Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.
Cube.js ⭐ 9,566
📊 Cube.js — Open-Source Analytical API Platform
Flink Learning ⭐ 9,256
flink learning blog. http://www.flink-learning.com 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Bigdata Notes ⭐ 8,801
大数据入门指南 ⭐️
Ds Cheatsheets ⭐ 8,343
List of Data Science Cheatsheets to rule the world
Thingsboard ⭐ 7,985
Open-source IoT Platform - Device management, data collection, processing and visualization.
Angel ⭐ 6,122
A Flexible and Powerful Parameter Server for large-scale machine learning
Dev Setup ⭐ 5,291
macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.
H2o 3 ⭐ 5,168
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Zeppelin ⭐ 5,112
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Alluxio ⭐ 4,915
Alluxio, data orchestration for analytics and machine learning in the cloud
Pipeline ⭐ 4,096
PipelineAI Kubeflow Distribution
Bigdl ⭐ 3,695
BigDL: Distributed Deep Learning Framework for Apache Spark
Tensorflowonspark ⭐ 3,618
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
God Of Bigdata ⭐ 3,434
大数据面试题，大数据成神之路开启...Flink/Spark/Hadoop/Hbase/Hive...
Coolplayspark ⭐ 3,175
酷玩 Spark: Spark 源代码解析、Spark 类库等
Delta ⭐ 3,024
An open-source storage layer that brings scalable, ACID transactions to Apache Spark™ and big data workloads.
Spark Notebook ⭐ 3,001
Interactive and Reactive Data Science using Scala and Spark.
Helk ⭐ 2,688
The Hunting ELK
Spark Jobserver ⭐ 2,652
REST job server for Apache Spark
Dpark ⭐ 2,646
Python clone of Spark, a MapReduce alike framework in Python
Koalas ⭐ 2,591
Koalas: pandas API on Apache Spark
Mmlspark ⭐ 2,231
Microsoft Machine Learning for Apache Spark
Roaringbitmap ⭐ 2,061
A better compressed bitset in Java
Transmogrifai ⭐ 1,982
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
Spark Nlp ⭐ 1,843
State of the Art Natural Language Processing
Linkis ⭐ 1,823
Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.
Benchm Ml ⭐ 1,805
A minimal benchmark for scalability, speed and accuracy of commonly used open source implementations (R packages, Python scikit-learn, H2O, xgboost, Spark MLlib etc.) of the top machine learning algorithms for binary classification (random forests, gradient boosted trees, deep neural networks etc.).
Spark Ml Source Analysis ⭐ 1,797
spark ml 算法原理剖析以及具体的源码实现分析
Quill ⭐ 1,795
Compile-time Language Integrated Queries for Scala
Spark Cassandra Connector ⭐ 1,762
DataStax Spark Cassandra Connector
Nd4j ⭐ 1,725
Fast, Scientific and Numerical Computing for the JVM (NDArrays)
Ballista ⭐ 1,633
Distributed compute platform implemented in Rust, using Apache Arrow memory model.
Quicksql ⭐ 1,630
A Flexible, Fast, Federated(3F) SQL Analysis Middleware for Multiple Data Sources
Gaffer ⭐ 1,600
A large-scale entity and relation database supporting aggregation of properties
Spark ⭐ 1,594
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Spring Boot Quick ⭐ 1,540
🌿 基于springboot的快速学习示例,整合自己遇到的开源框架,如：rabbitmq(延迟队列)、Kafka、jpa、redies、oauth2、swagger、jsp、docker、spring-batch、异常处理、日志输出、多模块开发、多环境打包、缓存cache、爬虫、jwt、GraphQL、dubbo、zookeeper和Async等等📌
Elassandra ⭐ 1,514
Elassandra = Elasticsearch + Apache Cassandra
Elephas ⭐ 1,438
Distributed Deep learning with Keras & Spark
Seldon Server ⭐ 1,423
Machine Learning Platform and Recommendation Engine built on Kubernetes
Deequ ⭐ 1,414
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Spark On K8s Operator ⭐ 1,344
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Waterdrop ⭐ 1,339
生产环境的海量数据计算产品，文档地址：
Almond ⭐ 1,336
A Scala kernel for Jupyter
Spark Py Notebooks ⭐ 1,320
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
Home ⭐ 1,196
ApacheCN 开源组织：公告、介绍、成员、活动、交流方式
Mleap ⭐ 1,194
MLeap: Deploy Spark Pipelines to Production
Spark Doc Zh ⭐ 1,112
Apache Spark 官方文档中文版
Dataspherestudio ⭐ 1,088
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Utils4s ⭐ 1,066
scala、spark使用过程中，各种测试用例以及相关资料整理
Apache Spark Internals ⭐ 1,029
The Internals of Apache Spark
Snappydata ⭐ 992
Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster
Pixiedust ⭐ 989
Python Helper library for Jupyter Notebooks
Optimus ⭐ 968
🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark
Gatk ⭐ 966
Official code repository for GATK versions 4 and up
Data Algorithms Book ⭐ 942
MapReduce, Spark, Java, and Scala for Data Algorithms Book
Sparkmagic ⭐ 928
Jupyter magics and kernels for working with remote Spark clusters
Mobius ⭐ 927
C# and F# language binding and extensions to Apache Spark
Spark Scala Tutorial ⭐ 894
A free tutorial for Apache Spark.
Kylo ⭐ 894
Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.
Sparkling Water ⭐ 887
Sparkling Water provides H2O functionality inside Spark cluster
Flint ⭐ 870
A Time Series Library for Apache Spark
Dockerfiles ⭐ 828
50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak, TeamCity and DevOps tools built on the major Linux distros: Alpine, CentOS, Debian, Fedora, Ubuntu
Bigdata Interview ⭐ 807
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结
Sparklyr ⭐ 768
R interface for Apache Spark
Goodreads_etl_pipeline ⭐ 760
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Spark Redis ⭐ 756
A connector for Spark that allows reading and writing to/from Redis cluster
Spark Movie Lens ⭐ 740
An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset
Bigdataguide ⭐ 737
大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料
Kafka Storm Starter ⭐ 728
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Coding Now ⭐ 725
学习记录的一些笔记，以及所看得一些电子书eBooks、视频资源和平常收纳的一些自己认为比较好的博客、网站、工具。涉及大数据几大组件、Python机器学习和数据分析、Linux、操作系统、算法、网络等
Sparkctr ⭐ 721
CTR prediction model based on spark(LR, GBDT, DNN)
Szt Bigdata ⭐ 721
深圳地铁大数据客流分析系统🚇🚄🌟
Cdhproject ⭐ 710
hadoop各组件使用，持续更新
Frameless ⭐ 702
Expressive types for Spark.
Hail ⭐ 693
Scalable genomic data analysis.
Scriptis ⭐ 688
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
Elasticsearch Spark Recommender ⭐ 687
Use Jupyter Notebooks to demonstrate how to build a Recommender with Apache Spark & Elasticsearch
Freestyle ⭐ 628
A cohesive & pragmatic framework of FP centric Scala libraries
Useractionanalyzeplatform ⭐ 623
电商用户行为分析大数据平台
Datafusion ⭐ 598
DataFusion has now been donated to the Apache Arrow project
Pyspark Example Project ⭐ 598
Example project implementing best practices for PySpark ETL jobs and applications.
Mongo Spark ⭐ 586
The MongoDB Spark Connector
Sparklearning ⭐ 550
Learning Apache spark,including code and data .Most part can run local.
Spark Daria ⭐ 529
Essential Spark extensions and helper methods ✨😲
Justenoughscalaforspark ⭐ 528
A tutorial on the most important features and idioms of Scala that you need to use Spark's Scala APIs.
Lopq ⭐ 527
Training of Locally Optimized Product Quantization (LOPQ) models for approximate nearest neighbor search of high dimensional data in Python and Spark.
Sparta ⭐ 514
Real Time Analytics and Data Pipelines based on Spark Streaming
Magellan ⭐ 505
Geo Spatial Data Analytics on Spark
Cdap ⭐ 492
An open source framework for building data analytic applications.
Spark ⭐ 468
Cross-platform real-time collaboration client optimized for business and organizations.
Pointblank ⭐ 458
Data validation and organization of metadata for data frames and database tables
Bigdataie ⭐ 436
大数据博客、笔试题、教程、项目、面经的整理
High Performance Spark Examples ⭐ 433
Examples for High Performance Spark
1-100 of 334 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210