Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Python Pyspark Open Source Projects

Open source projects categorized as Python Pyspark
Recommenders ⭐ 15,291
Best Practices on Recommendation Systems
dependent packages 2total releases 11most recent commit 15 hours agopypi recommenders} Downloads
Ibis ⭐ 2,551
Expressive analytics in Python at any scale.
dependent packages 16total releases 32most recent commit 10 hours agopypi ibis-framework} Downloads
Machine Learning ⭐ 2,272
🌎 machine learning tutorials (mainly in Python3)
most recent commit 14 days ago
Petastorm ⭐ 1,593
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
dependent packages 6total releases 77most recent commit 4 days agopypi petastorm} Downloads
Mleap ⭐ 1,434
MLeap: Deploy ML Pipelines to Production
dependent packages 12total releases 26most recent commit 24 days ago
Spark Py Notebooks ⭐ 1,227
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
most recent commit 6 years ago
Sparkmagic ⭐ 1,199
Jupyter magics and kernels for working with remote Spark clusters
dependent packages 5total releases 47most recent commit a month agopypi hdijupyterutils} Downloads
Bigflow ⭐ 1,122
Baidu Bigflow is an interface that allows for writing distributed computing programs and provides lots of simple, flexible, powerful APIs. Using Bigflow, you can easily handle data of any scale. Bigflow processes 4P+ data inside Baidu and runs about 10k jobs every day.
most recent commit 5 months ago
Sparkit Learn ⭐ 1,054
PySpark + Scikit-learn = Sparkit-learn
total releases 13most recent commit 2 years ago
Pyspark Example Project ⭐ 1,034
Example project implementing best practices for PySpark ETL jobs and applications.
most recent commit 4 months ago
Hopsworks ⭐ 899
Hopsworks - Data-Intensive AI platform with a Feature Store
total releases 1most recent commit 2 days ago
Pyspark Examples ⭐ 661
Pyspark RDD, DataFrame and Dataset Examples in Python language
most recent commit 4 months ago
Devops Python Tools ⭐ 631
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
most recent commit a month ago
Kuwala ⭐ 610
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demograp
most recent commit 7 months ago
Eat_pyspark_in_10_days ⭐ 534
pyspark🍒🥭 is delicious，just eat it!😋😋
most recent commit 6 months ago
Pandapy ⭐ 483
PandaPy has the speed of NumPy and the usability of Pandas 10x to 50x faster (by @firmai)
total releases 22most recent commit a year ago
Quinn ⭐ 436
pyspark methods to enhance developer productivity 📣 👯 🎉
dependent packages 1total releases 12most recent commit 2 days agopypi quinn} Downloads
Findspark ⭐ 428
dependent packages 35total releases 14most recent commit a year agopypi findspark} Downloads
Gather Deployment ⭐ 345
Gathers scalable Tensorflow, Python infrastructure deployment and practices, 100% Docker.
most recent commit a year ago
Sparklingpandas ⭐ 338
Sparkling Pandas
total releases 7most recent commit 6 years ago
Tdigest ⭐ 332
t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
dependent packages 12total releases 14most recent commit 5 months agopypi tdigest} Downloads
Chispa ⭐ 330
PySpark test helper methods with beautiful error messages
total releases 15most recent commit 14 days ago
Spark Standalone Cluster On Docker ⭐ 311
Learn Apache Spark in Scala, Python (PySpark) and R (SparkR) by building your own cluster with a JupyterLab interface on Docker. ⚡️
most recent commit 4 months ago
The 10 Most Depended On Python Pyspark Open Source Projects
Findspark ⭐ 428
dependent packages 35total releases 14latest release February 11, 2022most recent commit a year agopypi findspark} Downloads
Ibis ⭐ 2,551
Expressive analytics in Python at any scale.
dependent packages 16total releases 32latest release April 28, 2022most recent commit 10 hours agopypi ibis-framework} Downloads
Mleap ⭐ 1,434
MLeap: Deploy ML Pipelines to Production
dependent packages 12total releases 26latest release May 07, 2021most recent commit 24 days ago
Tdigest ⭐ 332
t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
dependent packages 12total releases 14latest release May 07, 2019most recent commit 5 months agopypi tdigest} Downloads
Sparkmagic ⭐ 1,199
Jupyter magics and kernels for working with remote Spark clusters
dependent packages 5total releases 47latest release May 02, 2022most recent commit a month agopypi hdijupyterutils} Downloads
Pypmml ⭐ 64
Python PMML scoring library
dependent packages 5total releases 14latest release April 24, 2022most recent commit 3 months agopypi pypmml} Downloads
Cluster Pack ⭐ 39
A library on top of either pex or conda-pack to make your Python code easily available on a cluster
dependent packages 4total releases 29latest release March 22, 2022most recent commit 2 months agopypi cluster-pack} Downloads
Pyspark Stubs ⭐ 116
Apache (Py)Spark type annotations (stub files).
dependent packages 3total releases 38latest release January 06, 2022most recent commit 7 months agopypi pyspark-stubs} Downloads
Recommenders ⭐ 15,291
Best Practices on Recommendation Systems
dependent packages 2total releases 11latest release April 01, 2022most recent commit 15 hours agopypi recommenders} Downloads
The 10 Latest Releases In Python Pyspark Open Source Projects
Aut ⭐ 122
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
total releases 25latest release June 17, 2022most recent commit 4 months ago
Anovos ⭐ 78
Anovos - An Open Source Library for Scalable feature engineering Using Apache-Spark
total releases 5latest release May 28, 2022most recent commit 21 days ago
Soda Spark ⭐ 49
Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes
dependent packages 1total releases 11latest release May 11, 2022most recent commit 10 months agopypi soda-spark} Downloads
Sparkmagic ⭐ 1,199
Jupyter magics and kernels for working with remote Spark clusters
dependent packages 5total releases 47latest release May 02, 2022most recent commit a month agopypi hdijupyterutils} Downloads
Ibis ⭐ 2,551
Expressive analytics in Python at any scale.
dependent packages 16total releases 32latest release April 28, 2022most recent commit 10 hours agopypi ibis-framework} Downloads
Pypmml ⭐ 64
Python PMML scoring library
dependent packages 5total releases 14latest release April 24, 2022most recent commit 3 months agopypi pypmml} Downloads
Recommenders ⭐ 15,291
Best Practices on Recommendation Systems
dependent packages 2total releases 11latest release April 01, 2022most recent commit 15 hours agopypi recommenders} Downloads
Joblib Spark ⭐ 217
Joblib Apache Spark Backend
dependent packages 1total releases 6latest release March 23, 2022most recent commit 23 days agopypi joblibspark} Downloads
Cluster Pack ⭐ 39
A library on top of either pex or conda-pack to make your Python code easily available on a cluster
dependent packages 4total releases 29latest release March 22, 2022most recent commit 2 months agopypi cluster-pack} Downloads
Chispa ⭐ 330
PySpark test helper methods with beautiful error messages
total releases 15latest release February 21, 2022most recent commit 14 days ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Classification
Scraper
Transaction
Data Science
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2023 Awesome Open Source.  All rights reserved.