Awesome Open Source
Awesome Open Source
Combined Topics
apache-spark x
scala x

The Top 384 Scala Apache Spark Open Source Projects

Categories > Data Processing > Apache Spark
Spark ⭐ 33,271
Apache Spark - A unified analytics engine for large-scale data processing
dependent packages 882total releases 46most recent commit 5 hours ago
Bigdl ⭐ 3,963
Building Large-Scale AI Applications for Distributed Big Data
dependent packages 10total releases 16most recent commit 6 hours ago
Coolplayspark ⭐ 3,333
酷玩 Spark: Spark 源代码解析、Spark 类库等
most recent commit a month ago
Synapseml ⭐ 3,328
Simple and Distributed Machine Learning
dependent packages 1total releases 3most recent commit 6 days agopypi synapseml} Downloads
Hudi ⭐ 3,278
Upserts, Deletes And Incremental Processing on Big Data.
dependent packages 25total releases 11most recent commit 5 hours ago
Spark Notebook ⭐ 3,031
Interactive and Reactive Data Science using Scala and Spark.
most recent commit a year ago
Spark Nlp ⭐ 2,796
State of the Art Natural Language Processing
dependent packages 2total releases 90most recent commit 5 hours ago
Analytics Zoo ⭐ 2,504
Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
dependent packages 3total releases 216most recent commit a month agopypi bigdl-orca} Downloads
Ballista ⭐ 2,244
Distributed compute platform implemented in Rust, and powered by Apache Arrow.
dependent packages 13total releases 4most recent commit a year ago
Deequ ⭐ 2,113
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
dependent packages 4total releases 31most recent commit 5 months ago
Transmogrifai ⭐ 2,099
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
dependent packages 3total releases 9most recent commit 4 months ago
Docker Spark ⭐ 1,658
Apache Spark docker image
most recent commit 6 months ago
Awesome Spark ⭐ 1,285
A curated list of awesome Apache Spark packages and resources.
most recent commit 4 months ago
Spark Timeseries ⭐ 1,147
A library for time series analysis on Apache Spark
total releases 2most recent commit 2 years ago
Livy ⭐ 911
Livy is an open source REST interface for interacting with Apache Spark from anywhere
total releases 2most recent commit 2 years ago
Tensorframes ⭐ 766
[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark
total releases 2most recent commit 2 years ago
Feathr ⭐ 748
Feathr – An Enterprise-Grade, High Performance Feature Store
total releases 17most recent commit 4 hours ago
Extraction Framework ⭐ 741
The software used to extract structured data from Wikipedia
total releases 3most recent commit 2 months ago
Kafka Storm Starter ⭐ 729
[PROJECT IS NO LONGER MAINTAINED] Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
most recent commit 3 months ago
Incubator Livy ⭐ 682
Mirror of Apache livy (Incubating)
dependent packages 4total releases 5most recent commit 3 months ago
Sparklearning ⭐ 573
Learning Apache spark,including code and data .Most part can run local.
most recent commit 8 months ago
Learningsparkv2 ⭐ 570
This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
most recent commit 7 months ago
Sparkmeasure ⭐ 454
This is the development repository for sparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task and stage metrics data.
dependent packages 2total releases 8most recent commit 2 months ago
Spark Rapids ⭐ 433
Spark RAPIDS plugin - accelerate Apache Spark with GPUs
total releases 10most recent commit 15 hours ago
Spline ⭐ 415
Data Lineage Tracking And Visualization Solution
dependent packages 4total releases 14most recent commit 11 days ago
Spark Corenlp ⭐ 409
Stanford CoreNLP wrapper for Apache Spark
most recent commit 4 years ago
Spark Perf ⭐ 346
Performance tests for Apache Spark
most recent commit 4 years ago
Morpheus ⭐ 307
Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.
dependent packages 7total releases 33most recent commit 2 years ago
Mist ⭐ 303
Serverless proxy for Spark cluster
dependent packages 4total releases 3most recent commit 2 years ago
Akka Analytics ⭐ 281
Large-scale event processing with Akka Persistence and Apache Spark
most recent commit 6 years ago
Spark Indexedrdd ⭐ 247
An efficient updatable key-value store for Apache Spark
most recent commit 5 years ago
Succinct ⭐ 239
Enabling queries on compressed data.
most recent commit 2 years ago
Spark Workshop ⭐ 231
Apache Spark™ and Scala Workshops
most recent commit 9 days ago
Delight ⭐ 212
A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.
most recent commit 5 months ago
Sparkrdma ⭐ 191
RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark
most recent commit 3 years ago
Azure Event Hubs Spark ⭐ 190
Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs
total releases 68most recent commit a month ago
Azure Cosmosdb Spark ⭐ 179
Apache Spark Connector for Azure Cosmos DB
dependent packages 1total releases 45most recent commit 5 months ago
Spark Authorizer ⭐ 158
A Spark SQL extension which provides SQL Standard Authorization for Apache Spark
most recent commit 3 months ago
Bigdata Playground ⭐ 154
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
most recent commit 3 years ago
Spark Snowflake ⭐ 151
Snowflake Data Source for Apache Spark.
total releases 139most recent commit 8 days ago
Spark Ext ⭐ 147
Spark Extension : ML transformers, SQL aggregations, etc that are missing in Apache Spark
most recent commit 6 years ago
Dbscan On Spark ⭐ 146
An implementation of DBSCAN runing on top of Apache Spark
most recent commit 4 years ago
Albedo ⭐ 142
A recommender system for discovering GitHub repos, built with Apache Spark
most recent commit 2 years ago
Sparknotebook ⭐ 142
An example of running Apache Spark using Scala in ipython notebook
most recent commit 7 years ago
Spark On Lambda ⭐ 139
Apache Spark on AWS Lambda
most recent commit a year ago
Spark Tsne ⭐ 134
Distributed t-SNE via Apache Spark
most recent commit 5 years ago
Sansa Stack ⭐ 125
Big Data RDF Processing and Analytics Stack built on Apache Spark and Apache Jena http://sansa-stack.github.io/SANSA-Stack/
most recent commit 23 days ago
Griffon Vm ⭐ 123
Griffon Data Science Virtual Machine
most recent commit 2 years ago
Example Spark Kafka ⭐ 118
Apache Spark and Apache Kafka integration example
most recent commit 5 years ago
Aut ⭐ 117
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
total releases 25most recent commit 14 days ago
Drizzle Spark ⭐ 113
Drizzle integration with Apache Spark
most recent commit 4 years ago
Spark Atlas Connector ⭐ 112
A Spark Atlas connector to track data lineage in Apache Atlas
most recent commit 2 years ago
Cobrix ⭐ 106
A COBOL parser and Mainframe/EBCDIC data source for Apache Spark
dependent packages 2total releases 30most recent commit 3 days ago
Learning Spark ⭐ 94
Practical examples of using Apache Spark in several different use cases
most recent commit 6 years ago
Mongo Spark ⭐ 93
Example application on how to use mongo-hadoop connector with Spark
most recent commit 8 years ago
Spark Dynamodb ⭐ 90
DynamoDB data source for Apache Spark
total releases 12most recent commit 10 months ago
Qstreaming ⭐ 89
A simplified, lightweight ETL pipeline framework for build stream/batch processing applications on top of Apache Spark
total releases 5most recent commit 9 months ago
Spark States ⭐ 88
Custom state store providers for Apache Spark
most recent commit 5 months ago
Sparkcube ⭐ 87
SparkCube is an open-source project for extremely fast OLAP data analysis. SparkCube is an extension of Apache Spark.
most recent commit a year ago
Splash ⭐ 86
Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange
most recent commit 2 years ago
Pulsar Spark ⭐ 85
When Apache Pulsar meets Apache Spark
total releases 3most recent commit 15 days ago
Cuesheet ⭐ 84
A framework for writing Spark 2.x applications in a pretty way
most recent commit 5 years ago
Sparkprojecttemplate.g8 ⭐ 77
Template for Spark Projects
most recent commit 9 months ago
Docker Spark ⭐ 77
🚢 Docker image for Apache Spark
most recent commit 3 years ago
Spark Examples ⭐ 75
Apache Spark jobs such as Principal Coordinate Analysis.
most recent commit 5 years ago
Spark Annoy ⭐ 72
Building Annoy Index on Apache Spark
total releases 4most recent commit a year ago
Learn By Examples ⭐ 72
Real-world Spark pipelines examples
most recent commit 4 years ago
Scalaapacheaccesslogparser ⭐ 71
An Apache access log parser written in Scala
most recent commit a year ago
Avocado ⭐ 71
A Variant Caller, Distributed. Apache 2 licensed.
dependent packages 1total releases 1most recent commit 3 years ago
Cleanframes ⭐ 70
type-class based data cleansing library for Apache Spark SQL
total releases 16most recent commit 3 years ago
Reforest ⭐ 69
Random Forests in Apache Spark
most recent commit 3 years ago
Spark Twitter Stream Example ⭐ 69
"Sentiment analysis" on a live Twitter feed with Apache Spark and Apache Bahir
most recent commit 5 years ago
Spark Lp ⭐ 64
Distributed Linear Programming Solver on top of Apache Spark
most recent commit a year ago
Lambda Arch Spark ⭐ 63
most recent commit 2 years ago
Spark Etl ⭐ 62
Apache Spark based ETL Engine
most recent commit 6 years ago
Net.jgp.books.spark.ch01 ⭐ 61
Spark in Action, 2nd edition - chapter 1 - Introduction
most recent commit 9 months ago
Spark Records ⭐ 58
Bulletproof Apache Spark jobs with fast root cause analysis of failures.
most recent commit a year ago
Lighthouse ⭐ 54
Lighthouse is a library for data lakes built on top of Apache Spark. It provides high-level APIs in Scala to streamline data pipelines and apply best practices.
dependent packages 3total releases 10most recent commit a year ago
Pyspark Setup Guide ⭐ 54
A guide for setting up Spark + PySpark under Ubuntu linux
most recent commit 5 years ago
Connected Component ⭐ 52
Map Reduce Implementation of Connected Component on Apache Spark
most recent commit 4 years ago
Spark Tpcds Datagen ⭐ 50
All the things about TPC-DS in Apache Spark
most recent commit a year ago
Spark Scala Maven Example ⭐ 50
Example Maven configuration for a Spark, Scala project
most recent commit 3 months ago
Spark Json Schema ⭐ 50
JSON schema parser for Apache Spark
total releases 8most recent commit 2 years ago
Prosparkstreaming ⭐ 49
Code used in "Pro Spark Streaming: The Zen of Real-time Analytics using Apache Spark" published by Apress Publishing.
most recent commit 6 years ago
Sparker ⭐ 48
SparkER: an Entity Resolution framework for Apache Spark
most recent commit a month ago
Spark Activator ⭐ 47
Spark Streaming with Scala and Akka Activator template
most recent commit 6 years ago
Spark Nkp ⭐ 47
Natural Korean Processor for Apache Spark
total releases 7most recent commit 4 years ago
Aardpfark ⭐ 47
A library for exporting Spark ML models and pipelines to PFA
most recent commit 4 years ago
Spark Hive Udf ⭐ 47
Example project showing how to use Hive UDFs in Apache Spark
most recent commit 3 years ago
Spark Tda ⭐ 46
SparkTDA is a package for Apache Spark providing Topological Data Analysis Functionalities.
most recent commit 4 years ago
Spark Es ⭐ 44
ElasticSearch integration for Apache Spark
total releases 8most recent commit 6 years ago
Vagrant Spark Zeppelin ⭐ 43
Vagrant, Apache Spark and Apache Zeppelin VM for teaching
most recent commit 5 years ago
Sparkplug ⭐ 42
A framework for creating composable and pluggable data processing pipelines using Apache Spark, and running them on a cluster.
most recent commit 6 years ago
Stark ⭐ 41
A framework for Spatio-Temporal Data Analytics on Spark
most recent commit a year ago
Dblink ⭐ 38
Distributed Bayesian Entity Resolution in Apache Spark
most recent commit a year ago
Spark Ref Architecture ⭐ 38
Reference Architectures for Apache Spark
most recent commit 5 years ago
Spylon ⭐ 38
Utilities to work with Scala/Java code with py4j
most recent commit 4 years ago
Spark Transformers ⭐ 37
Spark-Transformers: Library for exporting Apache Spark MLLIB models to use them in any Java application with no other dependencies.
dependent packages 1total releases 22most recent commit 5 years ago
Real Time Stream Processing Engine ⭐ 37
This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.
most recent commit 6 years ago
Aws Kinesis Scala ⭐ 35
Scala client for Amazon Kinesis. Also provides write to Kinesis capability for Apache Spark or Spark Streaming.
dependent packages 1total releases 9most recent commit a year ago
1-100 of 384 projects
Related Awesome Lists
Scala Sbt Projects (4,178)
Scala Spark Projects (3,279)
Scala Akka Projects (2,110)
Java Scala Projects (1,794)
Scala Play Framework Projects (1,309)
Plugin Scala Projects (1,107)
Scala Kafka Projects (969)
Scala Scalajs Projects (887)
Docker Scala Projects (728)
Scala Apache Projects (705)
Scala Jar Projects (657)
Scala Types Projects (654)
Scala Stream Projects (637)
Javascript Scala Projects (601)
Scala Streaming Projects (595)
Http Scala Projects (590)
Json Scala Projects (573)
Scala Functional Programming Projects (499)
Scala Cats Projects (480)
Scala Hadoop Projects (479)
Scala Intellij Projects (452)
Python Scala Projects (445)
Scala Server Projects (398)
Scala Language Projects (394)
Scala Sql Projects (367)
Scala Reactive Projects (330)
Database Scala Projects (319)
Scala Compiler Projects (294)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 71
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 343
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 115
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,059,720)
Python  (807,238)
Java  (388,471)
Php  (285,994)
Typescript  (246,339)
C Plus Plus  (238,807)
Ruby  (221,828)
C  (182,009)
Shell  (171,220)
C Sharp  (164,144)
Golang  (158,264)
Swift  (65,220)
R  (57,927)
Rust  (57,578)
Objective C  (57,186)
Dart  (52,616)
Kotlin  (46,457)
Lua  (33,881)
Matlab  (31,153)
Perl  (30,540)
Language  (30,061)
Scala  (28,666)
Haskell  (20,368)
Clojure  (19,137)
Powershell  (19,017)
Bash  (18,679)
Coffeescript  (17,289)
Elixir  (16,659)
Assembly  (14,681)
Processing  (13,309)
Julia  (12,746)
Basic  (10,736)
Flow  (10,547)
Emacs Lisp  (10,418)
Erlang  (9,066)
Groovy  (8,445)
Scheme  (8,437)
Solidity  (7,959)
Ocaml  (6,704)
Pascal  (6,439)
Hack  (6,398)
Lisp  (6,341)
Fortran  (5,540)
Elm  (5,484)
Common Lisp  (5,298)
Dlang  (4,745)
Shell Script  (4,654)
Python Library  (4,455)
Crystal  (4,387)
Scripting  (4,205)
Top Projects
Freecodecamp ⭐ 348,263
996.icu ⭐ 262,763
Free Programming Books ⭐ 237,545
Coding Interview University ⭐ 222,708
Awesome ⭐ 206,565
Developer Roadmap ⭐ 198,345
Public Apis ⭐ 197,271
Vue ⭐ 197,267
React ⭐ 190,675
System Design Primer ⭐ 184,236
Tensorflow ⭐ 166,079
Bootstrap ⭐ 158,247
You Dont Know Js ⭐ 153,649
Cs Notes ⭐ 151,618
Ohmyzsh ⭐ 146,966
Build Your Own X ⭐ 145,201
Javascript Algorithms ⭐ 143,983
Flutter ⭐ 142,275
Python ⭐ 139,334
Gitignore ⭐ 134,582
Linux ⭐ 133,961
Vscode ⭐ 133,493
Awesome Python ⭐ 131,954
Javascript ⭐ 124,360
Javaguide ⭐ 124,025
Python 100 Days ⭐ 120,066
Computer Science ⭐ 118,096
Youtube Dl ⭐ 111,107
Fucking Algorithm ⭐ 107,431
The Art Of Command Line ⭐ 106,741
React Native ⭐ 103,553
Electron ⭐ 102,320
D3 ⭐ 101,735
Go ⭐ 101,161
30 Seconds Of Code ⭐ 96,728
Create React App ⭐ 95,709
Axios ⭐ 94,209
Free Programming Books Zh_cn ⭐ 93,295
Awesome Selfhosted ⭐ 92,838
Kubernetes ⭐ 89,765
Next.js ⭐ 88,863
Node ⭐ 88,499
Terminal ⭐ 83,802
Deno ⭐ 83,454
Three.js ⭐ 83,205
Awesome Go ⭐ 83,128
Angular ⭐ 82,404
Typescript ⭐ 81,883
Ant Design ⭐ 80,890
Material Ui ⭐ 79,404
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.