Awesome Open Source
Awesome Open Source
Combined Topics
apache-spark
x
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
The Top 528 Apache Spark Open Source Projects on Github
Categories
>
Data Processing
>
Apache Spark
Mlflow
⭐
10,258
Open source platform for the machine learning lifecycle
Coolplayspark
⭐
3,277
酷玩 Spark: Spark 源代码解析、Spark 类库等
Spark Notebook
⭐
3,031
Interactive and Reactive Data Science using Scala and Spark.
Analytics Zoo
⭐
2,390
Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
Hudi
⭐
2,286
Upserts, Deletes And Incremental Processing on Big Data.
Oryx
⭐
1,759
Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
Spark
⭐
1,677
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Lakefs
⭐
1,649
Git-like capabilities for your object storage
Spark On K8s Operator
⭐
1,639
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Docker Spark
⭐
1,591
Apache Spark docker image
Awesome Spark
⭐
1,169
A curated list of awesome Apache Spark packages and resources.
Apache Spark Internals
⭐
1,112
The Internals of Apache Spark
Sparkit Learn
⭐
1,054
PySpark + Scikit-learn = Sparkit-learn
Spark Sklearn
⭐
1,039
(Deprecated) Scikit-learn integration package for Apache Spark
Mobius
⭐
930
C# and F# language binding and extensions to Apache Spark
Sparklyr
⭐
808
R interface for Apache Spark
Kafka Storm Starter
⭐
725
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Dist Keras
⭐
611
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
Spark
⭐
597
Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
Goodreads_etl_pipeline
⭐
593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Flintrock
⭐
577
A command-line tool for launching Apache Spark clusters.
Openscoring
⭐
533
REST web service for the true real-time scoring (<1 ms) of Scikit-Learn, R and Apache Spark models
Streaming Readings
⭐
489
Streaming System 相关的论文读物
Awesome Kafka
⭐
436
A list about Apache Kafka
Sparkle
⭐
426
Haskell on Apache Spark.
Agile_data_code_2
⭐
418
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
Sparkmeasure
⭐
384
This is the development repository of SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task metrics data.
Spark Structured Streaming Book
⭐
372
The Internals of Spark Structured Streaming
Learningsparkv2
⭐
346
This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
Wirbelsturm
⭐
331
Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.
Morpheus
⭐
307
Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.
Mist
⭐
303
Serverless proxy for Spark cluster
Sparkflow
⭐
290
Easy to use library to bring Tensorflow on Apache Spark
Mastering Spark Sql Book
⭐
285
The Internals of Spark SQL
Spark Gotchas
⭐
276
Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
Pyspark Boilerplate
⭐
273
A boilerplate for writing PySpark Jobs
Quinn
⭐
269
pyspark methods to enhance developer productivity 📣 👯 🎉
Spark Jupyter Aws
⭐
255
A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support
Data Accelerator
⭐
253
Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
Cuelake
⭐
240
Use SQL to build ELT pipelines on a data lakehouse.
Pysparkling
⭐
234
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
Parquet Dotnet
⭐
226
🏐 Apache Parquet for modern .NET
Spark Workshop
⭐
212
Apache Spark™ and Scala Workshops
Parquetviewer
⭐
202
Simple windows desktop application for viewing & querying Apache Parquet files
Sparkrdma
⭐
191
RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark
Whylogs Java
⭐
179
Profile and monitor your ML data pipeline end-to-end
Azure Cosmosdb Spark
⭐
176
Apache Spark Connector for Azure Cosmos DB
Awesome Ai Infrastructures
⭐
171
Infrastructures™ for Machine Learning Training/Inference in Production.
Delight
⭐
166
A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.
Learning Apache Spark
⭐
163
Notes on Apache Spark (pyspark)
Azure Event Hubs Spark
⭐
160
Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs
Cheatsheets.pdf
⭐
159
📚 Various cheatsheets in PDF
Bigdata Playground
⭐
154
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Albedo
⭐
142
A recommender system for discovering GitHub repos, built with Apache Spark
Spark On Lambda
⭐
139
Apache Spark on AWS Lambda
Hydrograph
⭐
138
A visual ETL development and debugging tool for big data
Sparktorch
⭐
138
Train and run Pytorch models on Apache Spark.
Griffon Vm
⭐
123
Griffon Data Science Virtual Machine
Parquet Dotnet
⭐
122
🐬 Apache Parquet for modern .Net
Spark Atlas Connector
⭐
112
A Spark Atlas connector to track data lineage in Apache Atlas
Aut
⭐
108
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Scala Spark Tutorial
⭐
106
Project for James' Apache Spark with Scala course
Spark Tpc Ds Performance Test
⭐
104
Use the TPC-DS benchmark to test Spark SQL performance
Pyspark Stubs
⭐
104
Apache (Py)Spark type annotations (stub files).
Spark Operator
⭐
103
Operator for managing the Spark clusters on Kubernetes and OpenShift.
Sansa Stack
⭐
99
Big Data RDF Processing and Analytics Stack built on Apache Spark and Apache Jena
Spark With Python
⭐
98
Fundamentals of Spark with Python (using PySpark), code examples
Splash
⭐
86
Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange
Cuesheet
⭐
84
A framework for writing Spark 2.x applications in a pretty way
Sparkprojecttemplate.g8
⭐
77
Template for Spark Projects
Learning Hadoop And Spark
⭐
75
Companion to Learning Hadoop and Learning Spark courses on Linked In Learning
Euphoria
⭐
74
Euphoria is an open source Java API for creating unified big-data processing flows. It provides an engine independent programming model which can express both batch and stream transformations.
Learn By Examples
⭐
72
Real-world Spark pipelines examples
Cleanframes
⭐
70
type-class based data cleansing library for Apache Spark SQL
Pulsar Spark
⭐
69
When Apache Pulsar meets Apache Spark
Hal 9000
⭐
69
Automatically setup a productive development environment with Ansible on macOS
Sparksql For Hbase
⭐
66
Learn how to use Spark SQL and HSpark connector package to create / query data tables that reside in HBase region servers
Spark States
⭐
64
Custom state store providers for Apache Spark
Lambda Arch Spark
⭐
63
Net.jgp.books.spark.ch01
⭐
61
Spark in Action, 2nd edition - chapter 1 - Introduction
Osm Parquetizer
⭐
58
A converter for the OSM PBFs to Parquet files
Spark Records
⭐
58
Bulletproof Apache Spark jobs with fast root cause analysis of failures.
Mmtf Pyspark
⭐
58
Methods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.
Kafka Streaming Click Analysis
⭐
56
Use Kafka and Apache Spark streaming to perform click stream analytics
Jupyterlab Sparkmonitor
⭐
56
JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook
Awesome Pulsar
⭐
53
A curated list of Pulsar tools, integrations and resources.
Connected Component
⭐
52
Map Reduce Implementation of Connected Component on Apache Spark
Mmtf Workshop 2018
⭐
48
Structural Bioinformatics Training Workshop & Hackathon 2018
Aardpfark
⭐
47
A library for exporting Spark ML models and pipelines to PFA
Spark Nkp
⭐
47
Natural Korean Processor for Apache Spark
Spark Tda
⭐
46
SparkTDA is a package for Apache Spark providing Topological Data Analysis Functionalities.
Liquidsvm
⭐
45
Support vector machines (SVMs) and related kernel-based learning algorithms are a well-known class of machine learning algorithms, for non-parametric classification and regression. liquidSVM is an implementation of SVMs whose key features are: fully integrated hyper-parameter selection, extreme speed on both small and large data sets, full flexibility for experts, and inclusion of a variety of different learning scenarios: multi-class classification, ROC, and Neyman-Pearson learning, and least-squares, quantile, and expectile regression.
Geospark
⭐
44
bring sf to spark in production
Spark Scala Maven Example
⭐
43
Example Maven configuration for a Spark, Scala project
Sparker
⭐
42
SparkER: an Entity Resolution framework for Apache Spark
Stark
⭐
41
A framework for Spatio-Temporal Data Analytics on Spark
Ansible Spark Cluster
⭐
41
Ansible roles to install an Spark Standalone cluster (HDFS/Spark/Jupyter Notebook) or Ambari based Spark cluster
Sparkxgb
⭐
41
R interface for XGBoost on Spark
Spark Examples
⭐
40
Spark examples
Spark With Python My Learning Notes
⭐
39
ETL pipeline using pyspark (Spark - Python)
1-100 of 528 projects
Next >
Related Projects
Scala Apache Spark Projects (176)
Python Apache Spark Projects (116)
Java Apache Spark Projects (110)
Big Data Apache Spark Projects (90)
Machine Learning Apache Spark Projects (72)
Jupyter Notebook Apache Spark Projects (64)
Pyspark Apache Spark Projects (57)
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.
