Awesome Open Source
Awesome Open Source
Combined Topics
pyspark
x
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
The Top 560 Pyspark Open Source Projects on Github
Categories
>
Data Processing
>
Pyspark
Synapseml
⭐
2,436
Microsoft Machine Learning for Apache Spark
Spark Nlp
⭐
2,367
State of the Art Natural Language Processing
Linkis
⭐
2,195
Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.
Spark Py Notebooks
⭐
1,227
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
Petastorm
⭐
1,223
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Awesome Spark
⭐
1,169
A curated list of awesome Apache Spark packages and resources.
Optimus
⭐
1,108
🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Sparkmagic
⭐
1,038
Jupyter magics and kernels for working with remote Spark clusters
Pyspark Example Project
⭐
727
Example project implementing best practices for PySpark ETL jobs and applications.
Scriptis
⭐
694
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
Devops Python Tools
⭐
463
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Spark Syntax
⭐
391
This is a repo documenting the best practices in PySpark.
Spark Gotchas
⭐
276
Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
Pyspark Boilerplate
⭐
273
A boilerplate for writing PySpark Jobs
Quinn
⭐
269
pyspark methods to enhance developer productivity 📣 👯 🎉
Tdigest
⭐
258
t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
Morphl Community Edition
⭐
233
MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization
Gimel
⭐
215
Big Data Processing Framework - Unified Data API or SQL on Any Storage
Eat_pyspark_in_10_days
⭐
176
pyspark🍒🥭 is delicious，just eat it!😋😋
Azure Cosmosdb Spark
⭐
176
Apache Spark Connector for Azure Cosmos DB
Butterfree
⭐
166
A tool for building feature stores.
Cc Pyspark
⭐
159
Process Common Crawl data with Python and Spark
Spark Practice
⭐
153
Apache Spark (PySpark) Practice on Real Data
Spark Iforest
⭐
147
Isolation Forest on Spark
Repo 2019
⭐
135
BERT, AWS RDS, AWS Forecast, EMR Spark Cluster, Hive, Serverless, Google Assistant + Raspberry Pi, Infrared, Google Cloud Platform Natural Language, Anomaly detection, Tensorflow, Mathematics
Pyspark Learning
⭐
133
Updated repository
Handyspark
⭐
129
HandySpark - bringing pandas-like capabilities to Spark dataframes
Hnswlib
⭐
125
Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs
Sparklearning
⭐
124
A comprehensive Spark guide collated from multiple sources that can be referred to learn more about Spark or as an interview refresher.
Spark Standalone Cluster On Docker
⭐
122
Learn Apache Spark in Scala, Python (PySpark) and R (SparkR) by building your own cluster with a JupyterLab interface on Docker. ⚡️
Pyspark Cheatsheet
⭐
121
🐍 Quick reference guide to common patterns & functions in PySpark.
Learningapachespark
⭐
119
LearningApacheSpark
Ai Deployment
⭐
116
关注AI模型上线、模型部署
Aut
⭐
108
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Big Data Mapreduce Course
⭐
107
Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University
Pyspark Stubs
⭐
104
Apache (Py)Spark type annotations (stub files).
Spark With Python
⭐
98
Fundamentals of Spark with Python (using PySpark), code examples
Chispa
⭐
97
PySpark test helper methods with beautiful error messages
Relation_extraction
⭐
93
Relation Extraction using Deep learning(CNN)
Bitcoin Value Predictor
⭐
90
[NOT MAINTAINED] Predicting Bit coin price using Time series analysis and sentiment analysis of tweets on bitcoin
Phrase At Scale
⭐
84
Detect common phrases in large amounts of text using a data-driven approach. Size of discovered phrases can be arbitrary. Can be used in languages other than English
Spark_python_ml_examples
⭐
81
Spark 2.0 Python Machine Learning examples
Pyspark Tutorial
⭐
77
PySpark Code for Hands-on Learners
Bigflow
⭐
73
A Python framework for data processing on GCP.
Learn By Examples
⭐
72
Real-world Spark pipelines examples
Pyspark Tutorial
⭐
68
Jupyter notebooks for pyspark tutorials given at the university
Jgit Spark Connector
⭐
67
jgit-spark-connector is a library for running scalable data retrieval pipelines that process any number of Git repositories for source code analysis.
W2v
⭐
62
Word2Vec models with Twitter data using Spark. Blog:
Pysparkgeoanalysis
⭐
60
🌐 Interactive Workshop on GeoAnalysis using PySpark
Spark
⭐
60
Apache Spark (Scala, PySpark, SparkR) Code, Tricks, and References
Pyspark Cassandra
⭐
59
pyspark-cassandra is a Python port of the awesome @datastax Spark Cassandra connector. Compatible w/ Spark 2.0, 2.1, 2.2, 2.3 and 2.4
Mmtf Pyspark
⭐
58
Methods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.
Jupyterlab Sparkmonitor
⭐
56
JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook
Spark Select
⭐
53
A library for Spark DataFrame using MinIO Select API
Spark Training
⭐
52
Repository used for Spark Trainings
Mmtf Workshop 2018
⭐
48
Structural Bioinformatics Training Workshop & Hackathon 2018
Sparkly
⭐
46
Helpers & syntactic sugar for PySpark.
Data_processing_course
⭐
45
Some class materials for a data processing course using PySpark
Osci
⭐
44
Open Source Contributor Index
Azure Databricks Nyc Taxi Workshop
⭐
39
An Azure Databricks workshop leveraging the New York Taxi and Limousine Commission Trip Records dataset
Spark Twitter Sentiment Analysis
⭐
33
Sentiment Analysis of a Twitter Topic with Spark Structured Streaming
Pyspark Algorithms
⭐
33
PySpark Algorithms Book: https://www.amazon.com/dp/B07X4B2218/ref=sr_1_2
Cluster Pack
⭐
28
A library on top of either pex or conda-pack to make your Python code easily available on a cluster
Kafka Compose
⭐
28
🎼 Docker compose files for various kafka stacks
Movalytics Data Warehouse
⭐
27
Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
Big_data
⭐
27
A collection of tutorials on Hadoop, MapReduce, Spark, Docker
Odsc_india_2018
⭐
26
My presentation at ODSC India 2018 about Deep Learning with Apache Spark
Courses
⭐
25
Just the stuff from the faculty (homework, projects, lectures)
Flask Spark Docker
⭐
24
Just a boilerplate for PySpark and Flask
Decorators4ds
⭐
24
Useful decorators every Data Scientist should know
Terraform Emr Pyspark
⭐
24
Quickstart PySpark with Anaconda on AWS/EMR using Terraform
Datalake Etl Pipeline
⭐
24
Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations
Mmtf Genomics
⭐
23
Methods for mapping genomic data onto 3D protein structure.
Dbldatagen
⭐
23
Generate relevant data quickly for your projects. The Databricks data generator can be used to generate large simulated / synthetic data sets for test, POCs, and other uses
Isarn Sketches Spark
⭐
23
Routines and data structures for using isarn-sketches idiomatically in Apache Spark
Spark Dgraph Connector
⭐
23
A connector for Apache Spark and PySpark to Dgraph databases.
Springboard Data Science Immersive
⭐
23
Sql Data Analysis And Visualization Projects
⭐
22
SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
Python_mozetl
⭐
22
ETL jobs for Firefox Telemetry
Dataanalysiswithpythonandpyspark
⭐
22
Code repository for the "PySpark in Action" book
Gutenberg
⭐
21
A content-based recommender system for books using the Project Gutenberg text corpus
Spark And Kafka_iot Data Processing And Analytics
⭐
21
Final Project for IoT: Big Data Processing and Analytics class in UCSC Extension. Analyzing U.S nationwide temperature from IoT sensors in real-time
Pyspark Setup Demo
⭐
21
Demo of PySpark and Jupyter Notebook with the Jupyter Docker Stacks
Spark Tdd Example
⭐
20
A simple Spark TDD example
Dlsa
⭐
20
Distributed least squares approximation (dlsa) implemented with Apache Spark
Sparkclean
⭐
20
A Scalable Data Cleaning Library for PySpark.
Movie Recommender System
⭐
19
Movie recommender system with Collaborative Filtering using PySpark
Pyspark Cheatsheet
⭐
19
PySpark Cheat Sheet - learn PySpark and develop apps faster
Oshinko S2i
⭐
19
This is a place to put s2i images and utilities for spark application builders for openshift
Basin
⭐
19
Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser
Spark3d
⭐
19
Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, …
Azure Databricks
⭐
19
Azure Databricks - Advent of 2020 Blogposts
Spark And Mllib Projects
⭐
18
This repository contains Spark, MLlib, PySpark and Dataframes projects
Pyspark Asyncactions
⭐
18
Asynchronous actions for PySpark
Bigdata Workshop Es
⭐
17
Workshop Big Data en Español
Spark Extension
⭐
17
A library that provides useful extensions to Apache Spark and PySpark.
Nyc Taxi Analysis
⭐
17
Analyzing 200 GB of NYC taxi dataset.
Sparklanes
⭐
16
A lightweight data processing framework for Apache Spark
Kafka Twitter Spark Streaming
⭐
16
Counting Tweets Per User in Real-Time
Pyspark For Data Processing
⭐
16
Code for my presentation: Using PySpark to Process Boat Loads of Data
1-100 of 560 projects
Next >
Related Projects
Python Pyspark Projects (334)
Spark Pyspark Projects (281)
Jupyter Notebook Pyspark Projects (230)
Python Spark Pyspark Projects (181)
Jupyter Notebook Spark Pyspark Projects (103)
Machine Learning Pyspark Projects (90)
Python Jupyter Notebook Pyspark Projects (83)
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.
