Awesome Open Source
Awesome Open Source
Combined Topics
pyspark x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 43 Pyspark Open Source Projects

Categories > Data Processing > Pyspark
Mmlspark ⭐ 2,305
Microsoft Machine Learning for Apache Spark
Spark Nlp ⭐ 2,061
State of the Art Natural Language Processing
Linkis ⭐ 1,938
Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.
Spark Py Notebooks ⭐ 1,350
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
Petastorm ⭐ 1,127
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Awesome Spark ⭐ 1,084
A curated list of awesome Apache Spark packages and resources.
Optimus ⭐ 997
🚚 Agile Data Preparation Workflows made easy with pandas, dask, cudf, dask_cudf and pyspark
Sparkmagic ⭐ 958
Jupyter magics and kernels for working with remote Spark clusters
Scriptis ⭐ 703
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
Pyspark Example Project ⭐ 649
Example project implementing best practices for PySpark ETL jobs and applications.
Devops Python Tools ⭐ 430
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Spark Syntax ⭐ 415
This is a repo documenting the best practices in PySpark.
Pyspark Boilerplate ⭐ 320
A boilerplate for writing PySpark Jobs
Spark Gotchas ⭐ 309
Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
Tdigest ⭐ 280
t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
Morphl Community Edition ⭐ 254
MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization
Quinn ⭐ 226
pyspark methods to enhance developer productivity 📣 👯 🎉
Gimel ⭐ 216
Big Data Processing Framework - Unified Data API or SQL on Any Storage
Spark Practice ⭐ 201
Apache Spark (PySpark) Practice on Real Data
Azure Cosmosdb Spark ⭐ 170
Apache Spark Connector for Azure Cosmos DB
Spark Iforest ⭐ 170
Isolation Forest on Spark
Spark With Python ⭐ 160
Fundamentals of Spark with Python (using PySpark), code examples
Eat_pyspark_in_10_days ⭐ 160
pyspark🍒🥭 is delicious，just eat it!😋😋
Handyspark ⭐ 158
HandySpark - bringing pandas-like capabilities to Spark dataframes
Learningapachespark ⭐ 157
LearningApacheSpark
Cc Pyspark ⭐ 156
Process Common Crawl data with Python and Spark
Pyspark Learning ⭐ 149
Updated repository
Butterfree ⭐ 136
A tool for building feature stores.
Repo 2019 ⭐ 134
BERT, AWS RDS, AWS Forecast, EMR Spark Cluster, Hive, Serverless, Google Assistant + Raspberry Pi, Infrared, Google Cloud Platform Natural Language, Anomaly detection, Tensorflow, Mathematics
Pyspark Cheatsheet ⭐ 112
🐍 Quick reference guide to common patterns & functions in PySpark.
Hnswlib ⭐ 110
Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs
Pyspark Stubs ⭐ 100
Apache (Py)Spark type annotations (stub files).
Relation_extraction ⭐ 96
Relation Extraction using Deep learning(CNN)
Pyspark Tutorial ⭐ 91
PySpark Code for Hands-on Learners
Bitcoin Value Predictor ⭐ 90
[NOT MAINTAINED] Predicting Bit coin price using Time series analysis and sentiment analysis of tweets on bitcoin
Spark_python_ml_examples ⭐ 87
Spark 2.0 Python Machine Learning examples
W2v ⭐ 65
Word2Vec models with Twitter data using Spark. Blog:
Pysparkgeoanalysis ⭐ 63
🌐 Interactive Workshop on GeoAnalysis using PySpark
Pyspark Setup Demo ⭐ 27
Demo of PySpark and Jupyter Notebook with the Jupyter Docker Stacks
Cluster Pack ⭐ 25
A library on top of either pex or conda-pack to make your Python code easily available on a cluster
Spark Tdd Example ⭐ 23
A simple Spark TDD example
Live_log_analyzer_spark ⭐ 14
Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.
Sparkling Titanic ⭐ 12
Training models with Apache Spark, PySpark for Titanic Kaggle competition
1-43 of 43 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210