Awesome Open Source
Awesome Open Source
Combined Topics
pyspark x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 43 Pyspark Open Source Projects

Categories > Data Processing > Pyspark
Mmlspark ⭐ 2,230
Microsoft Machine Learning for Apache Spark
Spark Nlp ⭐ 1,838
State of the Art Natural Language Processing
Linkis ⭐ 1,815
Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.
Spark Py Notebooks ⭐ 1,319
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
Petastorm ⭐ 1,050
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Awesome Spark ⭐ 1,023
A curated list of awesome Apache Spark packages and resources.
Optimus ⭐ 968
🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark
Sparkmagic ⭐ 928
Jupyter magics and kernels for working with remote Spark clusters
Scriptis ⭐ 688
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
Pyspark Example Project ⭐ 593
Example project implementing best practices for PySpark ETL jobs and applications.
Spark Syntax ⭐ 406
This is a repo documenting the best practices in PySpark.
Devops Python Tools ⭐ 377
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Pyspark Boilerplate ⭐ 313
A boilerplate for writing PySpark Jobs
Spark Gotchas ⭐ 302
Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
Tdigest ⭐ 263
t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
Morphl Community Edition ⭐ 253
MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization
Gimel ⭐ 214
Big Data Processing Framework - Unified Data API or SQL on Any Storage
Quinn ⭐ 203
pyspark methods to enhance developer productivity 📣 👯 🎉
Spark Practice ⭐ 190
Apache Spark (PySpark) Practice on Real Data
Azure Cosmosdb Spark ⭐ 162
Apache Spark Connector for Azure Cosmos DB
Spark Iforest ⭐ 162
Isolation Forest on Spark
Handyspark ⭐ 152
HandySpark - bringing pandas-like capabilities to Spark dataframes
Learningapachespark ⭐ 149
LearningApacheSpark
Pyspark Learning ⭐ 144
Updated repository
Cc Pyspark ⭐ 140
Process Common Crawl data with Python and Spark
Spark With Python ⭐ 140
Fundamentals of Spark with Python (using PySpark), code examples
Repo 2019 ⭐ 131
BERT, AWS RDS, AWS Forecast, EMR Spark Cluster, Hive, Serverless, Google Assistant + Raspberry Pi, Infrared, Google Cloud Platform Natural Language, Anomaly detection, Tensorflow, Mathematics
Butterfree ⭐ 106
A tool for building feature stores.
Pyspark Cheatsheet ⭐ 101
🐍 Quick reference guide to common patterns & functions in PySpark.
Hnswlib ⭐ 101
Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs
Relation_extraction ⭐ 95
Relation Extraction using Deep learning(CNN)
Pyspark Stubs ⭐ 94
Apache (Py)Spark type annotations (stub files).
Bitcoin Value Predictor ⭐ 91
[NOT MAINTAINED] Predicting Bit coin price using Time series analysis and sentiment analysis of tweets on bitcoin
Pyspark Tutorial ⭐ 87
PySpark Code for Hands-on Learners
Spark_python_ml_examples ⭐ 86
Spark 2.0 Python Machine Learning examples
Eat_pyspark_in_10_days ⭐ 67
pyspark🍒🥭 is delicious，just eat it!😋😋
W2v ⭐ 63
Word2Vec models with Twitter data using Spark. Blog:
Pysparkgeoanalysis ⭐ 63
🌐 Interactive Workshop on GeoAnalysis using PySpark
Spark Tdd Example ⭐ 23
A simple Spark TDD example
Pyspark Setup Demo ⭐ 23
Demo of PySpark and Jupyter Notebook with the Jupyter Docker Stacks
Cluster Pack ⭐ 20
A library on top of either pex or conda-pack to make your Python code easily available on a cluster
Live_log_analyzer_spark ⭐ 14
Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.
Sparkling Titanic ⭐ 12
Training models with Apache Spark, PySpark for Titanic Kaggle competition
1-43 of 43 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210