Awesome Open Source
Awesome Open Source
Combined Topics
apache-spark x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 76 Apache Spark Open Source Projects

Categories > Data Processing > Apache Spark
Mlflow ⭐ 8,876
Open source platform for the machine learning lifecycle
Coolplayspark ⭐ 3,220
酷玩 Spark: Spark 源代码解析、Spark 类库等
Spark Notebook ⭐ 3,026
Interactive and Reactive Data Science using Scala and Spark.
Analytics Zoo ⭐ 2,260
Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
Oryx ⭐ 1,754
Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
Spark ⭐ 1,626
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Spark On K8s Operator ⭐ 1,440
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Docker Spark ⭐ 1,439
Apache Spark docker image
Awesome Spark ⭐ 1,083
A curated list of awesome Apache Spark packages and resources.
Sparkit Learn ⭐ 1,077
PySpark + Scikit-learn = Sparkit-learn
Spark Sklearn ⭐ 1,060
(Deprecated) Scikit-learn integration package for Apache Spark
Apache Spark Internals ⭐ 1,046
The Internals of Apache Spark
Mobius ⭐ 929
C# and F# language binding and extensions to Apache Spark
Goodreads_etl_pipeline ⭐ 798
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Sparklyr ⭐ 779
R interface for Apache Spark
Kafka Storm Starter ⭐ 728
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Dist Keras ⭐ 616
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
Flintrock ⭐ 573
A command-line tool for launching Apache Spark clusters.
Streaming Readings ⭐ 557
Streaming System 相关的论文读物
Openscoring ⭐ 539
REST web service for the true real-time scoring (<1 ms) of Scikit-Learn, R and Apache Spark models
Sparkle ⭐ 420
Haskell on Apache Spark.
Agile_data_code_2 ⭐ 411
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
Awesome Kafka ⭐ 409
A list about Apache Kafka
Spark Structured Streaming Book ⭐ 372
The Internals of Spark Structured Streaming
Sparkmeasure ⭐ 371
This is the development repository of SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task metrics data.
Learningsparkv2 ⭐ 340
This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
Wirbelsturm ⭐ 333
Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.
Pyspark Boilerplate ⭐ 320
A boilerplate for writing PySpark Jobs
Mist ⭐ 309
Serverless proxy for Spark cluster
Spark Gotchas ⭐ 309
Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
Morpheus ⭐ 303
Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.
Sparkflow ⭐ 285
Easy to use library to bring Tensorflow on Apache Spark
Parquet Dotnet ⭐ 280
🏐 Apache Parquet for modern .NET
Spark Jupyter Aws ⭐ 259
A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support
Data Accelerator ⭐ 248
Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
Mastering Spark Sql Book ⭐ 241
The Internals of Spark SQL
Pysparkling ⭐ 231
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
Awesome Ai Infrastructures ⭐ 229
Infrastructures™ for Machine Learning Training/Inference in Production.
Quinn ⭐ 225
pyspark methods to enhance developer productivity 📣 👯 🎉
Spark Workshop ⭐ 224
Apache Spark™ and Scala Workshops
Sparkrdma ⭐ 214
RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark
Learning Apache Spark ⭐ 213
Notes on Apache Spark (pyspark)
Sparktorch ⭐ 197
Train and run Pytorch models on Apache Spark.
Bigdata Playground ⭐ 180
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Azure Cosmosdb Spark ⭐ 170
Apache Spark Connector for Azure Cosmos DB
Spark Atlas Connector ⭐ 166
A Spark Atlas connector to track data lineage in Apache Atlas
Whylogs Java ⭐ 164
Profile and monitor your ML data pipeline end-to-end
Cheatsheets.pdf ⭐ 161
📚 Various cheatsheets in PDF
Spark With Python ⭐ 160
Fundamentals of Spark with Python (using PySpark), code examples
Parquetviewer ⭐ 154
Simple windows desktop application for viewing & querying Apache Parquet files
Albedo ⭐ 150
A recommender system for discovering GitHub repos, built with Apache Spark
Hydrograph ⭐ 144
A visual ETL development and debugging tool for big data
Scalable Data Science ⭐ 143
Scalable Data Science, course sets in big data Using Apache Spark over databricks and their mathematical, statistical and computational foundations using SageMath.
Azure Event Hubs Spark ⭐ 142
Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs
Spark On Lambda ⭐ 137
Apache Spark on AWS Lambda
Spark Tpc Ds Performance Test ⭐ 135
Use the TPC-DS benchmark to test Spark SQL performance
Griffon Vm ⭐ 128
Griffon Data Science Virtual Machine
Scala Spark Tutorial ⭐ 121
Project for James' Apache Spark with Scala course
Splash ⭐ 105
Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange
Pyspark Stubs ⭐ 100
Apache (Py)Spark type annotations (stub files).
Cuesheet ⭐ 86
A framework for writing Spark 2.x applications in a pretty way
Spark States ⭐ 84
Custom state store providers for Apache Spark
Awesome Pulsar ⭐ 59
A curated list of Pulsar tools, integrations and resources.
Pulsar Spark ⭐ 56
When Apache Pulsar meets Apache Spark
Spark Nkp ⭐ 51
Natural Korean Processor for Apache Spark
Spark As Service Using Embedded Server ⭐ 47
This application comes as Spark2.1-as-Service-Provider using an embedded, Reactive-Streams-based, fully asynchronous HTTP server
Spark Scala Maven Example ⭐ 46
Example Maven configuration for a Spark, Scala project
Spark Tda ⭐ 45
SparkTDA is a package for Apache Spark providing Topological Data Analysis Functionalities.
Spark Examples ⭐ 41
Spark examples
Real Time Stream Processing Engine ⭐ 38
This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.
Dblink ⭐ 38
Distributed Bayesian Entity Resolution in Apache Spark
Datahacksummit_2017 ⭐ 31
Apache Zeppelin notebooks for Recommendation Engines using Keras and Machine Learning on Apache Spark
Cloud Based Sql Engine Using Spark ⭐ 30
Cloud-based SQL engine using SPARK where data is accessible as JDBC/ODBC data source via Spark ThriftServer.
Spark Flamegraph ⭐ 30
Easy CPU Profiling for Apache Spark applications
Spark Streaming Monitoring With Lightning ⭐ 15
Plot live-stats as graph from ApacheSpark application using Lightning-viz
Live_log_analyzer_spark ⭐ 14
Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.
1-76 of 76 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210