Awesome Open Source
Awesome Open Source
Combined Topics
apache-spark x

The Top 1,191 Apache Spark Open Source Projects

Categories > Data Processing > Apache Spark
Spark ⭐ 32,943
Apache Spark - A unified analytics engine for large-scale data processing
dependent packages 882total releases 46most recent commit 4 hours ago
Mlflow ⭐ 11,890
Open source platform for the machine learning lifecycle
dependent packages 235total releases 57most recent commit 5 hours agopypi mlflow} Downloads
Sparkinternals ⭐ 4,665
Notes talking about the design and implementation of Apache Spark
most recent commit 7 months ago
Bigdl ⭐ 3,931
Building Large-Scale AI Applications for Distributed Big Data
dependent packages 10total releases 16most recent commit 5 hours ago
Coolplayspark ⭐ 3,333
酷玩 Spark: Spark 源代码解析、Spark 类库等
most recent commit 7 days ago
Synapseml ⭐ 3,301
Simple and Distributed Machine Learning
dependent packages 1total releases 3most recent commit 17 hours agopypi synapseml} Downloads
Hudi ⭐ 3,152
Upserts, Deletes And Incremental Processing on Big Data.
dependent packages 25total releases 11most recent commit 3 hours ago
Spark Notebook ⭐ 3,031
Interactive and Reactive Data Science using Scala and Spark.
most recent commit a year ago
Koalas ⭐ 2,999
Koalas: pandas API on Apache Spark
dependent packages 12total releases 47most recent commit 7 months agopypi koalas} Downloads
Spark Nlp ⭐ 2,734
State of the Art Natural Language Processing
dependent packages 2total releases 90most recent commit 3 days ago
Lakefs ⭐ 2,490
Git-like capabilities for your object storage
dependent packages 1total releases 45most recent commit 3 hours agopypi lakefs-client} Downloads
Analytics Zoo ⭐ 2,487
Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
dependent packages 3total releases 183most recent commit 2 months agopypi bigdl-orca} Downloads
Ballista ⭐ 2,244
Distributed compute platform implemented in Rust, and powered by Apache Arrow.
dependent packages 13total releases 4most recent commit a year ago
Deequ ⭐ 2,113
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
dependent packages 4total releases 31most recent commit 3 months ago
Transmogrifai ⭐ 2,099
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
dependent packages 3total releases 9most recent commit 3 months ago
Spark On K8s Operator ⭐ 1,949
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
dependent packages 16total releases 18most recent commit 7 days ago
Vega ⭐ 1,904
A new arguably faster implementation of Apache Spark from scratch in Rust
most recent commit a year ago
Spark ⭐ 1,778
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
dependent packages 15total releases 19most recent commit 13 hours agonuget Microsoft.Spark} Downloads
Oryx ⭐ 1,759
Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
dependent packages 2total releases 14most recent commit 10 months ago
Docker Spark ⭐ 1,658
Apache Spark docker image
most recent commit 5 months ago
Awesome Spark ⭐ 1,285
A curated list of awesome Apache Spark packages and resources.
most recent commit 2 months ago
Dr Elephant ⭐ 1,230
Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark
most recent commit 6 months ago
Spark Timeseries ⭐ 1,147
A library for time series analysis on Apache Spark
total releases 2most recent commit 2 years ago
Spark Doc Zh ⭐ 1,101
Apache Spark 官方文档中文版
most recent commit a year ago
Sparkit Learn ⭐ 1,054
PySpark + Scikit-learn = Sparkit-learn
total releases 13most recent commit a year ago
Spark Sklearn ⭐ 1,039
(Deprecated) Scikit-learn integration package for Apache Spark
total releases 8most recent commit 2 years ago
Mobius ⭐ 930
C# and F# language binding and extensions to Apache Spark
total releases 22most recent commit a year ago
Livy ⭐ 911
Livy is an open source REST interface for interacting with Apache Spark from anywhere
total releases 2most recent commit 2 years ago
Sparklyr ⭐ 860
R interface for Apache Spark
dependent packages 26total releases 46most recent commit a day ago
Tispark ⭐ 814
TiSpark is built for running Apache Spark on top of TiDB/TiKV
dependent packages 3total releases 51most recent commit 4 hours ago
Tensorframes ⭐ 766
[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark
total releases 2most recent commit 2 years ago
Extraction Framework ⭐ 741
The software used to extract structured data from Wikipedia
total releases 3most recent commit 20 days ago
Kafka Storm Starter ⭐ 729
[PROJECT IS NO LONGER MAINTAINED] Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
most recent commit 2 months ago
Incubator Livy ⭐ 682
Mirror of Apache livy (Incubating)
dependent packages 4total releases 5most recent commit 2 months ago
Streaming Readings ⭐ 640
Streaming System 相关的论文读物
most recent commit 3 months ago
Docker Spark ⭐ 626
Docker build for Apache Spark
most recent commit 2 years ago
Dist Keras ⭐ 611
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
total releases 2most recent commit 4 years ago
Flintrock ⭐ 601
A command-line tool for launching Apache Spark clusters.
total releases 13most recent commit a month ago
Spark ⭐ 597
Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
most recent commit 2 years ago
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 2 years ago
Sparklearning ⭐ 573
Learning Apache spark,including code and data .Most part can run local.
most recent commit 7 months ago
Learningsparkv2 ⭐ 570
This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
most recent commit 5 months ago
Openscoring ⭐ 562
REST web service for the true real-time scoring (<1 ms) of Scikit-Learn, R and Apache Spark models
dependent packages 1total releases 34most recent commit 3 months ago
Awesome Kafka ⭐ 462
A list about Apache Kafka
most recent commit 5 months ago
Sparkmeasure ⭐ 454
This is the development repository for sparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task and stage metrics data.
dependent packages 2total releases 8most recent commit a month ago
Sparkle ⭐ 426
Haskell on Apache Spark.
total releases 16most recent commit 9 months ago
Sparkling ⭐ 423
A Clojure library for Apache Spark: fast, fully-features, and developer friendly
total releases 23most recent commit 3 months ago
Spark Rapids ⭐ 423
Spark RAPIDS plugin - accelerate Apache Spark with GPUs
total releases 10most recent commit 6 hours ago
Agile_data_code_2 ⭐ 423
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
most recent commit 7 months ago
Spark Corenlp ⭐ 409
Stanford CoreNLP wrapper for Apache Spark
most recent commit 4 years ago
Machinelearning ⭐ 406
Machine Learning
most recent commit 5 years ago
Spline ⭐ 401
Data Lineage Tracking And Visualization Solution
dependent packages 4total releases 14most recent commit a month ago
Spark Perf ⭐ 346
Performance tests for Apache Spark
most recent commit 4 years ago
Eclairjs Node ⭐ 340
Node.js API for Apache Spark with Remote Client
most recent commit 6 years ago
Wirbelsturm ⭐ 333
[PROJECT IS NO LONGER MAINTAINED] Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.
most recent commit 3 months ago
Parquet Dotnet ⭐ 319
🏐 Apache Parquet for modern .NET
total releases 4most recent commit 3 months ago
Morpheus ⭐ 307
Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.
dependent packages 7total releases 33most recent commit 2 years ago
Mist ⭐ 303
Serverless proxy for Spark cluster
dependent packages 4total releases 3most recent commit 2 years ago
Kotlin Spark Api ⭐ 297
This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
dependent packages 1total releases 1most recent commit 4 hours ago
Incubator Hivemall ⭐ 297
Mirror of Apache Hivemall (incubating)
most recent commit 6 months ago
Sparkflow ⭐ 290
Easy to use library to bring Tensorflow on Apache Spark
total releases 13most recent commit a year ago
Parquetviewer ⭐ 284
Simple windows desktop application for viewing & querying Apache Parquet files
most recent commit 3 months ago
Akka Analytics ⭐ 281
Large-scale event processing with Akka Persistence and Apache Spark
most recent commit 6 years ago
Spark Gotchas ⭐ 276
Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
most recent commit 5 years ago
Quinn ⭐ 269
pyspark methods to enhance developer productivity 📣 👯 🎉
dependent packages 1total releases 12most recent commit a year agopypi quinn} Downloads
Cuelake ⭐ 266
Use SQL to build ELT pipelines on a data lakehouse.
most recent commit 2 months ago
Data Accelerator ⭐ 264
Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
dependent packages 5total releases 26most recent commit 18 days agonpm datax-common} Downloads
Spark Jupyter Aws ⭐ 255
A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support
most recent commit 5 years ago
Spark Indexedrdd ⭐ 247
An efficient updatable key-value store for Apache Spark
most recent commit 5 years ago
Succinct ⭐ 239
Enabling queries on compressed data.
most recent commit 2 years ago
Pysparkling ⭐ 234
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
dependent packages 1total releases 68most recent commit a year agopypi pysparkling} Downloads
Spark Workshop ⭐ 227
Apache Spark™ and Scala Workshops
most recent commit 4 months ago
Bitnami Docker Spark ⭐ 219
Bitnami Docker Image for Apache Spark
most recent commit 18 hours ago
Ruby Spark ⭐ 215
Ruby wrapper for Apache Spark
total releases 5most recent commit 5 years ago
Spark_dbscan ⭐ 215
DBSCAN clustering algorithm on top of Apache Spark
most recent commit 4 years ago
Delight ⭐ 212
A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.
most recent commit 4 months ago
Sql Data Analysis And Visualization Projects ⭐ 200
SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
most recent commit a month ago
Splink ⭐ 193
Implementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters
dependent packages 1total releases 56most recent commit 5 days agopypi splink} Downloads
Bigdl Tutorials ⭐ 192
Step-by-step Deep Leaning Tutorials on Apache Spark using BigDL
most recent commit a year ago
Sparkrdma ⭐ 191
RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark
most recent commit 3 years ago
Azure Event Hubs Spark ⭐ 190
Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs
total releases 68most recent commit 9 days ago
Vn.vitk ⭐ 189
A Vietnamese Text Processing Toolkit
most recent commit 2 years ago
Azure Cosmosdb Spark ⭐ 179
Apache Spark Connector for Azure Cosmos DB
dependent packages 1total releases 45most recent commit 4 months ago
Whylogs Java ⭐ 179
Profile and monitor your ML data pipeline end-to-end
dependent packages 2total releases 5most recent commit 8 months ago
Parquet Dotnet ⭐ 176
🐬 Apache Parquet for modern .Net
dependent packages 15total releases 209most recent commit 3 months agonuget Parquet.Net} Downloads
Awesome Ai Infrastructures ⭐ 171
Infrastructures™ for Machine Learning Training/Inference in Production.
most recent commit 3 years ago
Crossdata ⭐ 170
DISCONTINUED - Easy access to big things. Library for Apache Spark extending and improving its capabilities
dependent packages 11total releases 24most recent commit 3 years ago
Spark.jl ⭐ 166
Julia binding for Apache Spark
most recent commit 8 months ago
Rumble ⭐ 165
⛈️ RumbleDB 1.18.0 "Scarlet Ixora" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
total releases 4most recent commit a month ago
Spark Authorizer ⭐ 158
A Spark SQL extension which provides SQL Standard Authorization for Apache Spark
most recent commit 2 months ago
Bigdata Playground ⭐ 154
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
most recent commit 3 years ago
Spark Snowflake ⭐ 148
Snowflake Data Source for Apache Spark.
total releases 139most recent commit 2 months ago
Spark Ext ⭐ 147
Spark Extension : ML transformers, SQL aggregations, etc that are missing in Apache Spark
most recent commit 6 years ago
Dbscan On Spark ⭐ 146
An implementation of DBSCAN runing on top of Apache Spark
most recent commit 4 years ago
Albedo ⭐ 142
A recommender system for discovering GitHub repos, built with Apache Spark
most recent commit 2 years ago
Sparknotebook ⭐ 142
An example of running Apache Spark using Scala in ipython notebook
most recent commit 7 years ago
Spark On Lambda ⭐ 139
Apache Spark on AWS Lambda
most recent commit a year ago
Sparktorch ⭐ 138
Train and run Pytorch models on Apache Spark.
total releases 10most recent commit 2 years ago
Hydrograph ⭐ 138
A visual ETL development and debugging tool for big data
most recent commit 2 years ago
Databricks ⭐ 134
Repository of sample Databricks notebooks
most recent commit a year ago
1-100 of 1,191 projects
Related Awesome Lists
Scala Apache Spark Projects (495)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,048,735)
Python  (796,103)
Java  (385,413)
Php  (284,589)
Typescript  (239,722)
C Plus Plus  (236,213)
Ruby  (221,696)
C  (180,396)
Shell  (169,594)
C Sharp  (162,015)
Golang  (156,089)
Swift  (64,698)
R  (57,248)
Objective C  (57,169)
Rust  (56,117)
Dart  (51,466)
Kotlin  (45,511)
Lua  (33,293)
Matlab  (30,891)
Perl  (30,483)
Language  (30,085)
Scala  (28,601)
Haskell  (20,260)
Clojure  (19,072)
Powershell  (18,797)
Bash  (18,606)
Coffeescript  (17,331)
Elixir  (16,565)
Assembly  (14,526)
Processing  (13,298)
Julia  (12,536)
Basic  (10,765)
Flow  (10,528)
Emacs Lisp  (10,368)
Erlang  (9,058)
Scheme  (8,437)
Groovy  (8,431)
Solidity  (7,611)
Ocaml  (6,668)
Pascal  (6,378)
Hack  (6,365)
Lisp  (6,340)
Fortran  (5,483)
Elm  (5,477)
Common Lisp  (5,277)
Dlang  (4,747)
Shell Script  (4,633)
Python Library  (4,446)
Crystal  (4,374)
Scripting  (4,203)
Top Projects
Freecodecamp ⭐ 346,432
996.icu ⭐ 262,320
Free Programming Books ⭐ 233,678
Coding Interview University ⭐ 220,499
Awesome ⭐ 202,578
Vue ⭐ 196,186
Developer Roadmap ⭐ 195,398
Public Apis ⭐ 193,925
React ⭐ 188,417
System Design Primer ⭐ 180,789
Tensorflow ⭐ 165,144
Bootstrap ⭐ 157,483
You Dont Know Js ⭐ 153,649
Cs Notes ⭐ 150,780
Ohmyzsh ⭐ 145,459
Build Your Own X ⭐ 142,246
Flutter ⭐ 140,926
Javascript Algorithms ⭐ 140,055
Python ⭐ 136,303
Gitignore ⭐ 133,450
Linux ⭐ 132,199
Vscode ⭐ 132,056
Awesome Python ⭐ 127,528
Javascript ⭐ 123,347
Javaguide ⭐ 121,667
Python 100 Days ⭐ 117,585
Computer Science ⭐ 113,544
Youtube Dl ⭐ 110,239
Fucking Algorithm ⭐ 106,048
The Art Of Command Line ⭐ 105,874
React Native ⭐ 102,862
Electron ⭐ 101,813
D3 ⭐ 100,901
Go ⭐ 99,493
30 Seconds Of Code ⭐ 95,528
Create React App ⭐ 95,174
Axios ⭐ 93,562
Awesome Selfhosted ⭐ 89,230
Kubernetes ⭐ 88,689
Free Programming Books Zh_cn ⭐ 88,386
Node ⭐ 87,823
Next.js ⭐ 87,114
Terminal ⭐ 83,251
Deno ⭐ 82,597
Three.js ⭐ 82,214
Angular ⭐ 81,516
Awesome Go ⭐ 81,089
Typescript ⭐ 80,876
Ant Design ⭐ 80,416
Material Ui ⭐ 78,638
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.