Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Python Spark Pyspark Open Source Projects

Open source projects categorized as Python Spark Pyspark
Ibis ⭐ 3,404
The flexibility of Python with the scale and performance of modern SQL.
dependent packages 29total releases 68most recent commit 3 months agopypi ibis-framework} Downloads
Petastorm ⭐ 1,693
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
dependent packages 8total releases 86most recent commit 5 months agopypi petastorm} Downloads
Spark Py Notebooks ⭐ 1,515
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
most recent commit a year ago
Mleap ⭐ 1,479
MLeap: Deploy ML Pipelines to Production
dependent packages 12total releases 26most recent commit 5 months ago
Sparkmagic ⭐ 1,272
Jupyter magics and kernels for working with remote Spark clusters
dependent packages 6total releases 54most recent commit 3 months agopypi autovizwidget} Downloads
Bigflow ⭐ 1,122
Baidu Bigflow is an interface that allows for writing distributed computing programs and provides lots of simple, flexible, powerful APIs. Using Bigflow, you can easily handle data of any scale. Bigflow processes 4P+ data inside Baidu and runs about 10k jobs every day.
most recent commit a year ago
Pyspark Example Project ⭐ 1,034
Example project implementing best practices for PySpark ETL jobs and applications.
most recent commit a year ago
Devops Python Tools ⭐ 709
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
most recent commit 3 months ago
Eat_pyspark_in_10_days ⭐ 534
pyspark🍒🥭 is delicious，just eat it!😋😋
most recent commit 2 years ago
Findspark ⭐ 428
dependent packages 63total releases 14most recent commit 2 years agopypi findspark} Downloads
Datacompy ⭐ 339
Pandas and Spark DataFrame comparison for humans and more!
dependent packages 10total releases 20most recent commit 3 months agopypi datacompy} Downloads
Sparklingpandas ⭐ 338
Sparkling Pandas
total releases 7most recent commit 7 years ago
Spark Standalone Cluster On Docker ⭐ 311
Learn Apache Spark in Scala, Python (PySpark) and R (SparkR) by building your own cluster with a JupyterLab interface on Docker. ⚡
most recent commit a year ago
Learning Pyspark ⭐ 294
Code repository for Learning PySpark by Packt
most recent commit a year ago
Sagemaker Spark ⭐ 285
A Spark library for Amazon SageMaker.
total releases 36most recent commit 7 months ago
Sk Dist ⭐ 283
Distributed scikit-learn meta-estimators in PySpark
total releases 12most recent commit a year ago
Cc Pyspark ⭐ 280
Process Common Crawl data with Python and Spark
most recent commit a year ago
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35most recent commit 5 months agopypi butterfree} Downloads
Pyspark Style Guide ⭐ 264
This is a guide to PySpark code style presenting common situations and the associated best practices based on the most frequent recurring topics across the PySpark repos we've encountered.
most recent commit 3 years ago
Pysparkling ⭐ 253
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
dependent packages 1total releases 69most recent commit a year agopypi pysparkling} Downloads
Dbldatagen ⭐ 234
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines
total releases 12most recent commit 4 months ago
Learningapachespark ⭐ 233
LearningApacheSpark
most recent commit 4 months ago
Data_science_blogs ⭐ 232
A repository to keep track of all the code that I end up writing for my blog posts.
most recent commit a year ago
The 10 Most Depended On Python Spark Pyspark Open Source Projects
Findspark ⭐ 428
dependent packages 63total releases 14latest release February 11, 2022most recent commit 2 years agopypi findspark} Downloads
Ibis ⭐ 3,404
The flexibility of Python with the scale and performance of modern SQL.
dependent packages 29total releases 68latest release December 10, 2023most recent commit 3 months agopypi ibis-framework} Downloads
Mleap ⭐ 1,479
MLeap: Deploy ML Pipelines to Production
dependent packages 12total releases 26latest release May 07, 2021most recent commit 5 months ago
Datacompy ⭐ 339
Pandas and Spark DataFrame comparison for humans and more!
dependent packages 10total releases 20latest release November 15, 2023most recent commit 3 months agopypi datacompy} Downloads
Pytest Spark ⭐ 38
pytest plugin to run the tests with support of pyspark
dependent packages 7total releases 16latest release February 23, 2020most recent commit 4 years agopypi pytest-spark} Downloads
Sparkmagic ⭐ 1,272
Jupyter magics and kernels for working with remote Spark clusters
dependent packages 6total releases 54latest release September 13, 2023most recent commit 3 months agopypi autovizwidget} Downloads
Pypmml ⭐ 64
Python PMML scoring library
dependent packages 6total releases 15latest release November 03, 2022most recent commit a year agopypi pypmml} Downloads
Pyspark Stubs ⭐ 116
Apache (Py)Spark type annotations (stub files).
dependent packages 4total releases 38latest release January 06, 2022most recent commit 2 years agopypi pyspark-stubs} Downloads
Joblib Spark ⭐ 226
Joblib Apache Spark Backend
dependent packages 4total releases 7latest release July 12, 2023most recent commit 9 months agopypi joblibspark} Downloads
The 10 Latest Releases In Python Spark Pyspark Open Source Projects
Ibis ⭐ 3,404
The flexibility of Python with the scale and performance of modern SQL.
dependent packages 29total releases 68latest release December 10, 2023most recent commit 3 months agopypi ibis-framework} Downloads
Datacompy ⭐ 339
Pandas and Spark DataFrame comparison for humans and more!
dependent packages 10total releases 20latest release November 15, 2023most recent commit 3 months agopypi datacompy} Downloads
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35latest release November 14, 2023most recent commit 5 months agopypi butterfree} Downloads
Spark Extension ⭐ 152
A library that provides useful extensions to Apache Spark and PySpark.
total releases 65latest release September 28, 2023most recent commit 3 months ago
Sparkql ⭐ 10
sparkql: Apache Spark SQL DataFrame schema management for sensible humans
total releases 20latest release September 18, 2023most recent commit 7 months ago
Sparkmagic ⭐ 1,272
Jupyter magics and kernels for working with remote Spark clusters
dependent packages 6total releases 54latest release September 13, 2023most recent commit 3 months agopypi autovizwidget} Downloads
Joblib Spark ⭐ 226
Joblib Apache Spark Backend
dependent packages 4total releases 7latest release July 12, 2023most recent commit 9 months agopypi joblibspark} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.