Awesome Open Source
Awesome Open Source
Combined Topics
pyspark x

The Top 1,350 Pyspark Open Source Projects

Categories > Data Processing > Pyspark
Cheatsheets Ai ⭐ 13,281
Essential Cheat Sheets for deep learning and machine learning researchers https://medium.com/@kailashahirwar/essential-cheat
most recent commit 3 years ago
Recommenders ⭐ 13,132
Best Practices on Recommendation Systems
dependent packages 2total releases 10most recent commit 6 days agopypi recommenders} Downloads
Synapseml ⭐ 3,297
Simple and Distributed Machine Learning
dependent packages 1total releases 3most recent commit 6 days agopypi synapseml} Downloads
Spark Nlp ⭐ 2,727
State of the Art Natural Language Processing
dependent packages 2total releases 90most recent commit 2 days ago
Incubator Linkis ⭐ 2,529
Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.
dependent packages 5total releases 5most recent commit 7 hours ago
Machine Learning ⭐ 1,997
🌎 machine learning tutorials (mainly in Python3)
most recent commit 8 days ago
Ibis ⭐ 1,814
Expressive analytics in Python at any scale.
dependent packages 15total releases 30most recent commit 5 hours agopypi ibis-framework} Downloads
Petastorm ⭐ 1,413
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
dependent packages 6total releases 77most recent commit a month agopypi petastorm} Downloads
Mleap ⭐ 1,379
MLeap: Deploy ML Pipelines to Production
dependent packages 12total releases 26most recent commit 16 days ago
Awesome Spark ⭐ 1,285
A curated list of awesome Apache Spark packages and resources.
most recent commit 2 months ago
Spark Py Notebooks ⭐ 1,227
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
most recent commit 5 years ago
Optimus ⭐ 1,213
🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
total releases 26most recent commit 15 days ago
Sparkmagic ⭐ 1,117
Jupyter magics and kernels for working with remote Spark clusters
dependent packages 5total releases 47most recent commit 17 days agopypi hdijupyterutils} Downloads
Bigflow ⭐ 1,066
Baidu Bigflow is an interface that allows for writing distributed computing programs and provides lots of simple, flexible, powerful APIs. Using Bigflow, you can easily handle data of any scale. Bigflow processes 4P+ data inside Baidu and runs about 10k jobs every day.
most recent commit 2 years ago
Sparkit Learn ⭐ 1,054
PySpark + Scikit-learn = Sparkit-learn
total releases 13most recent commit a year ago
Sparkling Water ⭐ 924
Sparkling Water provides H2O functionality inside Spark cluster
dependent packages 4total releases 157most recent commit 7 days ago
Pyspark Tutorial ⭐ 853
PySpark-Tutorial provides basic algorithms using PySpark
most recent commit 5 months ago
Pyspark Example Project ⭐ 727
Example project implementing best practices for PySpark ETL jobs and applications.
most recent commit 9 months ago
Scriptis ⭐ 694
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
most recent commit a year ago
Devops Python Tools ⭐ 550
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
most recent commit 9 days ago
Pandapy ⭐ 483
PandaPy has the speed of NumPy and the usability of Pandas 10x to 50x faster (by @firmai)
total releases 22most recent commit 7 months ago
Kuwala ⭐ 475
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demograp
most recent commit 5 hours ago
Sparklearning ⭐ 451
A comprehensive Spark guide collated from multiple sources that can be referred to learn more about Spark or as an interview refresher.
most recent commit a month ago
Findspark ⭐ 428
dependent packages 35total releases 14most recent commit 3 months agopypi findspark} Downloads
Pyspark Examples ⭐ 418
Pyspark RDD, DataFrame and Dataset Examples in Python language
most recent commit 2 months ago
Learningpyspark ⭐ 409
Code base for the Learning PySpark book (in preparation)
most recent commit 3 years ago
Spark Syntax ⭐ 391
This is a repo documenting the best practices in PySpark.
most recent commit 3 years ago
Gather Deployment ⭐ 345
Gathers scalable Tensorflow, Python infrastructure deployment and practices, 100% Docker.
most recent commit a month ago
Sparklingpandas ⭐ 338
Sparkling Pandas
total releases 7most recent commit 5 years ago
Eat_pyspark_in_10_days ⭐ 327
pyspark🍒🥭 is delicious，just eat it!😋😋
most recent commit 4 months ago
Tdigest ⭐ 307
t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
dependent packages 11total releases 14most recent commit 6 months agopypi tdigest} Downloads
Spark Gotchas ⭐ 276
Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
most recent commit 5 years ago
Quinn ⭐ 269
pyspark methods to enhance developer productivity 📣 👯 🎉
dependent packages 1total releases 12most recent commit a year agopypi quinn} Downloads
Miscellaneous ⭐ 267
Includes notes on Apache Spark, Spark for Physics, Jupyter notebook examples for Spark and Oracle.
total releases 2most recent commit 9 hours ago
Sk Dist ⭐ 267
Distributed scikit-learn meta-estimators in PySpark
total releases 12most recent commit 10 months ago
Pyspark Style Guide ⭐ 264
This is a guide to PySpark code style presenting common situations and the associated best practices based on the most frequent recurring topics across the PySpark repos we've encountered.
most recent commit a year ago
Spark Jupyter Aws ⭐ 255
A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support
most recent commit 5 years ago
Pysparkling ⭐ 234
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
dependent packages 1total releases 68most recent commit a year agopypi pysparkling} Downloads
Pyspark Tutorials ⭐ 233
Code snippets and tutorials for working with social science data in PySpark
most recent commit 5 years ago
Morphl Community Edition ⭐ 233
MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization
most recent commit 3 years ago
Sagemaker Spark ⭐ 228
A Spark library for Amazon SageMaker.
total releases 32most recent commit a year ago
Gimel ⭐ 224
Big Data Processing Framework - Unified Data API or SQL on Any Storage
most recent commit 3 months ago
Learning Pyspark ⭐ 220
Code repository for Learning PySpark by Packt
most recent commit a year ago
Zeppelin Notebooks ⭐ 206
Gallery of Apache Zeppelin notebooks
most recent commit 3 years ago
Cc Pyspark ⭐ 201
Process Common Crawl data with Python and Spark
most recent commit a month ago
Sql Data Analysis And Visualization Projects ⭐ 200
SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
most recent commit a month ago
Learningapachespark ⭐ 192
LearningApacheSpark
most recent commit 5 months ago
Joblib Spark ⭐ 183
Joblib Apache Spark Backend
dependent packages 1total releases 5most recent commit 2 months agopypi joblibspark} Downloads
Azure Cosmosdb Spark ⭐ 179
Apache Spark Connector for Azure Cosmos DB
dependent packages 1total releases 45most recent commit 4 months ago
Pyspark Cheatsheet ⭐ 176
🐍 Quick reference guide to common patterns & functions in PySpark.
most recent commit 2 months ago
Hunter ⭐ 170
A threat hunting / data analysis environment based on Python, Pandas, PySpark and Jupyter Notebook.
most recent commit 10 months ago
Drunken Data Quality ⭐ 167
Spark package for checking data quality
total releases 6most recent commit 2 years ago
Butterfree ⭐ 166
A tool for building feature stores.
dependent packages 1total releases 30most recent commit 10 months agopypi butterfree} Downloads
Chispa ⭐ 159
PySpark test helper methods with beautiful error messages
total releases 15most recent commit 3 months ago
Cloud Dataproc ⭐ 158
Cloud Dataproc: Samples and Utils
most recent commit 2 months ago
Spark Practice ⭐ 153
Apache Spark (PySpark) Practice on Real Data
most recent commit 2 years ago
Geopyspark ⭐ 151
GeoTrellis for PySpark
most recent commit 2 years ago
Pyspark Pictures ⭐ 149
Learn the pyspark API through pictures and simple examples
most recent commit a year ago
Spark Iforest ⭐ 147
Isolation Forest on Spark
most recent commit 2 years ago
Hnswlib ⭐ 146
Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs
dependent packages 2total releases 50most recent commit 4 months ago
Nyc Transport ⭐ 144
A Unified Database of NYC transport (subway, taxi/Uber, and citibike) data.
most recent commit 5 years ago
Repo 2019 ⭐ 135
BERT, AWS RDS, AWS Forecast, EMR Spark Cluster, Hive, Serverless, Google Assistant + Raspberry Pi, Infrared, Google Cloud Platform Natural Language, Anomaly detection, Tensorflow, Mathematics
most recent commit 9 months ago
Handyspark ⭐ 129
HandySpark - bringing pandas-like capabilities to Spark dataframes
total releases 7most recent commit 3 years ago
Spark Standalone Cluster On Docker ⭐ 122
Learn Apache Spark in Scala, Python (PySpark) and R (SparkR) by building your own cluster with a JupyterLab interface on Docker. ⚡️
most recent commit a year ago
Aliyun Emapreduce Demo ⭐ 116
most recent commit a year ago
Ai Deployment ⭐ 116
关注AI模型上线、模型部署
most recent commit 10 months ago
Spark Df Profiling ⭐ 115
Create HTML profiling reports from Apache Spark DataFrames
total releases 13most recent commit 2 years ago
Spark Knn Recommender ⭐ 113
Item and User-based KNN recommendation algorithms using PySpark
most recent commit 5 years ago
Aut ⭐ 112
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
total releases 23most recent commit 2 months ago
Spark R Notebooks ⭐ 109
R on Apache Spark (SparkR) tutorials for Big Data analysis and Machine Learning as IPython / Jupyter notebooks
most recent commit 5 years ago
Pyspark Cheatsheet ⭐ 109
PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
most recent commit a month ago
Big Data Mapreduce Course ⭐ 108
Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University
most recent commit a day ago
Pyspark Stubs ⭐ 107
Apache (Py)Spark type annotations (stub files).
most recent commit 4 months ago
Dampr ⭐ 102
Python Data Processing library
most recent commit 2 years ago
Spark With Python ⭐ 98
Fundamentals of Spark with Python (using PySpark), code examples
most recent commit 2 years ago
Medium Articles ⭐ 94
Repo for all my code on the articles I post on medium
most recent commit 16 days ago
Relation_extraction ⭐ 93
Relation Extraction using Deep learning(CNN)
most recent commit 5 years ago
Bitcoin Value Predictor ⭐ 90
[NOT MAINTAINED] Predicting Bit coin price using Time series analysis and sentiment analysis of tweets on bitcoin
most recent commit 3 years ago
Mastering Big Data Analytics With Pyspark ⭐ 88
Mastering Big Data Analytics with PySpark, Published by Packt
most recent commit 5 months ago
Pyspark Csv ⭐ 87
An external PySpark module that works like R's read.csv or Panda's read_csv, with automatic type inference and null value handling. Parses csv data into SchemaRDD. No installation required, simply include pyspark_csv.py via SparkContext.
most recent commit 7 years ago
Kdd Cup 99 Spark ⭐ 87
PySpark solution to the KDDCup99
most recent commit 6 years ago
Pyspark Predictive Maintenance ⭐ 85
Predictive Maintenance using Pyspark
most recent commit 3 years ago
Phrase At Scale ⭐ 84
Detect common phrases in large amounts of text using a data-driven approach. Size of discovered phrases can be arbitrary. Can be used in languages other than English
most recent commit 3 years ago
Pyspark Tutorial ⭐ 82
Jupyter notebooks for pyspark tutorials given at the university
most recent commit 2 months ago
Spark_python_ml_examples ⭐ 81
Spark 2.0 Python Machine Learning examples
most recent commit 3 years ago
Pyspark Cassandra ⭐ 81
PySpark Cassandra brings back the fun in working with Cassandra data in PySpark.
most recent commit 5 years ago
Python Spark Streaming ⭐ 73
most recent commit 4 years ago
Jupyterlab Sparkmonitor ⭐ 72
JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook
most recent commit 2 months ago
Learn By Examples ⭐ 72
Real-world Spark pipelines examples
most recent commit 4 years ago
Data_science_blogs ⭐ 71
A repository to keep track of all the code that I end up writing for my blog posts.
most recent commit 2 years ago
Jgit Spark Connector ⭐ 67
jgit-spark-connector is a library for running scalable data retrieval pipelines that process any number of Git repositories for source code analysis.
most recent commit 3 years ago
Osci ⭐ 67
Open Source Contributor Index
most recent commit 3 months ago
Machinelearning ⭐ 65
Machine learning for beginner(Data Science enthusiast)
most recent commit a year ago
Pyspark_dist_explore ⭐ 64
Data Exploration in PySpark made easy - Pyspark_dist_explore provides methods to get fast insights in your Spark DataFrames.
most recent commit 3 years ago
Nsl Kdd ⭐ 63
PySpark solution to the NSL-KDD dataset: https://www.unb.ca/cic/datasets/nsl.html
most recent commit 2 years ago
Pyspark Twitter Stream Mining ⭐ 63
Real-time Machine Learning with Apache Spark on Twitter Public Stream
most recent commit 5 years ago
W2v ⭐ 62
Word2Vec models with Twitter data using Spark. Blog:
most recent commit 3 years ago
Spark ⭐ 60
Apache Spark (Scala, PySpark, SparkR) Code, Tricks, and References
most recent commit 3 years ago
Mmtf Pyspark ⭐ 60
Methods for the parallel and distributed analysis and mining of the Protein Data Bank using MMTF and Apache Spark.
most recent commit 3 days ago
Pysparkgeoanalysis ⭐ 60
🌐 Interactive Workshop on GeoAnalysis using PySpark
most recent commit 5 years ago
1-100 of 1,350 projects
Related Awesome Lists
Spark Pyspark Projects (773)
Python Pyspark Projects (689)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,046,513)
Python  (794,038)
Java  (384,745)
Php  (284,342)
Typescript  (238,372)
C Plus Plus  (235,678)
Ruby  (221,647)
C  (180,021)
Shell  (169,271)
C Sharp  (161,668)
Golang  (155,705)
Swift  (64,594)
Objective C  (57,180)
R  (57,097)
Rust  (55,814)
Dart  (51,233)
Kotlin  (45,308)
Lua  (33,168)
Matlab  (30,832)
Perl  (30,482)
Language  (30,083)
Scala  (28,582)
Haskell  (20,223)
Clojure  (19,065)
Powershell  (18,770)
Bash  (18,586)
Coffeescript  (17,335)
Elixir  (16,549)
Assembly  (14,508)
Processing  (13,295)
Julia  (12,509)
Basic  (10,772)
Flow  (10,528)
Emacs Lisp  (10,358)
Erlang  (9,057)
Scheme  (8,442)
Groovy  (8,426)
Solidity  (7,543)
Ocaml  (6,660)
Pascal  (6,371)
Hack  (6,365)
Lisp  (6,345)
Fortran  (5,475)
Elm  (5,468)
Common Lisp  (5,285)
Dlang  (4,742)
Shell Script  (4,626)
Python Library  (4,446)
Crystal  (4,373)
Scripting  (4,199)
Top Projects
Freecodecamp ⭐ 346,085
996.icu ⭐ 262,270
Free Programming Books ⭐ 233,047
Coding Interview University ⭐ 219,725
Awesome ⭐ 201,499
Vue ⭐ 195,787
Developer Roadmap ⭐ 194,812
Public Apis ⭐ 192,026
React ⭐ 188,079
System Design Primer ⭐ 180,726
Tensorflow ⭐ 164,981
Bootstrap ⭐ 157,299
You Dont Know Js ⭐ 152,909
Cs Notes ⭐ 150,780
Ohmyzsh ⭐ 145,101
Build Your Own X ⭐ 141,447
Flutter ⭐ 140,685
Javascript Algorithms ⭐ 140,055
Python ⭐ 136,032
Gitignore ⭐ 133,164
Linux ⭐ 131,866
Vscode ⭐ 131,762
Awesome Python ⭐ 126,837
Javascript ⭐ 122,717
Javaguide ⭐ 121,487
Python 100 Days ⭐ 117,585
Computer Science ⭐ 112,893
Youtube Dl ⭐ 110,029
Fucking Algorithm ⭐ 106,048
The Art Of Command Line ⭐ 105,874
React Native ⭐ 102,754
Electron ⭐ 101,722
D3 ⭐ 100,901
Go ⭐ 99,254
30 Seconds Of Code ⭐ 95,528
Create React App ⭐ 95,021
Axios ⭐ 93,400
Awesome Selfhosted ⭐ 88,479
Kubernetes ⭐ 88,475
Free Programming Books Zh_cn ⭐ 88,386
Node ⭐ 87,657
Next.js ⭐ 86,824
Terminal ⭐ 83,149
Deno ⭐ 82,461
Three.js ⭐ 82,021
Angular ⭐ 81,395
Awesome Go ⭐ 80,742
Typescript ⭐ 80,677
Ant Design ⭐ 80,313
Material Ui ⭐ 78,463
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.