Awesome Open Source
Awesome Open Source
Combined Topics
apache-spark x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 75 Apache Spark Open Source Projects

Categories > Data Processing > Apache Spark
Mlflow ⭐ 8,230
Open source platform for the machine learning lifecycle
Coolplayspark ⭐ 3,175
酷玩 Spark: Spark 源代码解析、Spark 类库等
Spark Notebook ⭐ 3,001
Interactive and Reactive Data Science using Scala and Spark.
Analytics Zoo ⭐ 2,182
Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
Oryx ⭐ 1,743
Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
Spark ⭐ 1,594
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Spark On K8s Operator ⭐ 1,344
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Docker Spark ⭐ 1,343
Apache Spark docker image
Sparkit Learn ⭐ 1,060
PySpark + Scikit-learn = Sparkit-learn
Spark Sklearn ⭐ 1,054
(Deprecated) Scikit-learn integration package for Apache Spark
Apache Spark Internals ⭐ 1,029
The Internals of Apache Spark
Awesome Spark ⭐ 1,024
A curated list of awesome Apache Spark packages and resources.
Mobius ⭐ 927
C# and F# language binding and extensions to Apache Spark
Sparklyr ⭐ 768
R interface for Apache Spark
Goodreads_etl_pipeline ⭐ 760
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Kafka Storm Starter ⭐ 728
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Dist Keras ⭐ 616
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
Flintrock ⭐ 555
A command-line tool for launching Apache Spark clusters.
Streaming Readings ⭐ 547
Streaming System 相关的论文读物
Openscoring ⭐ 533
REST web service for the true real-time scoring (<1 ms) of R, Scikit-Learn and Apache Spark models
Sparkle ⭐ 414
Haskell on Apache Spark.
Agile_data_code_2 ⭐ 411
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
Awesome Kafka ⭐ 384
A list about Apache Kafka
Spark Structured Streaming Book ⭐ 366
The Internals of Spark Structured Streaming
Sparkmeasure ⭐ 348
This is the development repository of SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task metrics data.
Wirbelsturm ⭐ 331
Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.
Pyspark Boilerplate ⭐ 313
A boilerplate for writing PySpark Jobs
Mist ⭐ 304
Serverless proxy for Spark cluster
Spark Gotchas ⭐ 302
Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
Morpheus ⭐ 301
Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.
Sparkflow ⭐ 279
Easy to use library to bring Tensorflow on Apache Spark
Parquet Dotnet ⭐ 267
🏐 Apache Parquet for modern .NET
Spark Jupyter Aws ⭐ 258
A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support
Learningsparkv2 ⭐ 250
This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
Data Accelerator ⭐ 247
Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
Pysparkling ⭐ 231
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
Mastering Spark Sql Book ⭐ 224
The Internals of Spark SQL
Spark Workshop ⭐ 221
Apache Spark™ and Scala Workshops
Sparkrdma ⭐ 212
RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark
Awesome Ai Infrastructures ⭐ 206
Infrastructures™ for Machine Learning Training/Inference in Production.
Quinn ⭐ 204
pyspark methods to enhance developer productivity 📣 👯 🎉
Learning Apache Spark ⭐ 202
Notes on Apache Spark (pyspark)
Sparktorch ⭐ 189
Train and run Pytorch models on Apache Spark.
Bigdata Playground ⭐ 173
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Whylogs Java ⭐ 163
Profile and monitor your ML data pipeline end-to-end
Azure Cosmosdb Spark ⭐ 162
Apache Spark Connector for Azure Cosmos DB
Cheatsheets.pdf ⭐ 153
📚 Various cheatsheets in PDF
Spark Atlas Connector ⭐ 149
A Spark Atlas connector to track data lineage in Apache Atlas
Albedo ⭐ 148
A recommender system for discovering GitHub repos, built with Apache Spark
Hydrograph ⭐ 143
A visual ETL development and debugging tool for big data
Scalable Data Science ⭐ 142
Scalable Data Science, course sets in big data Using Apache Spark over databricks and their mathematical, statistical and computational foundations using SageMath.
Spark With Python ⭐ 141
Fundamentals of Spark with Python (using PySpark), code examples
Azure Event Hubs Spark ⭐ 134
Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs
Parquetviewer ⭐ 134
Simple windows desktop application for viewing & querying Apache Parquet files
Spark On Lambda ⭐ 133
Apache Spark on AWS Lambda
Griffon Vm ⭐ 128
Griffon Data Science Virtual Machine
Spark Tpc Ds Performance Test ⭐ 126
Use the TPC-DS benchmark to test Spark SQL performance
Scala Spark Tutorial ⭐ 119
Project for James' Apache Spark with Scala course
Pyspark Stubs ⭐ 95
Apache (Py)Spark type annotations (stub files).
Cuesheet ⭐ 86
A framework for writing Spark 2.x applications in a pretty way
Spark States ⭐ 74
Custom state store providers for Apache Spark
Awesome Pulsar ⭐ 54
A curated list of Pulsar tools, integrations and resources.
Pulsar Spark ⭐ 52
When Apache Pulsar meets Apache Spark
Spark Nkp ⭐ 49
Natural Korean Processor for Apache Spark
Spark Scala Maven Example ⭐ 45
Example Maven configuration for a Spark, Scala project
Spark Tda ⭐ 45
SparkTDA is a package for Apache Spark providing Topological Data Analysis Functionalities.
Spark As Service Using Embedded Server ⭐ 44
This application comes as Spark2.1-as-Service-Provider using an embedded, Reactive-Streams-based, fully asynchronous HTTP server
Spark Examples ⭐ 41
Spark examples
Real Time Stream Processing Engine ⭐ 37
This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.
Dblink ⭐ 37
Distributed Bayesian Entity Resolution in Apache Spark
Spark Flamegraph ⭐ 30
Easy CPU Profiling for Apache Spark applications
Datahacksummit_2017 ⭐ 30
Apache Zeppelin notebooks for Recommendation Engines using Keras and Machine Learning on Apache Spark
Cloud Based Sql Engine Using Spark ⭐ 29
Cloud-based SQL engine using SPARK where data is accessible as JDBC/ODBC data source via Spark ThriftServer.
Spark Streaming Monitoring With Lightning ⭐ 15
Plot live-stats as graph from ApacheSpark application using Lightning-viz
Live_log_analyzer_spark ⭐ 14
Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.
1-75 of 75 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210