Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Pyspark Open Source Projects

Open source projects categorized as Pyspark
Categories > Data Processing > Pyspark
Cheatsheets Ai ⭐ 13,281
Essential Cheat Sheets for deep learning and machine learning researchers https://medium.com/@kailashahirwar/essential-cheat
most recent commit 4 years ago
Synapseml ⭐ 4,295
Simple and Distributed Machine Learning
dependent packages 1total releases 5most recent commit 2 days agopypi synapseml} Downloads
Spark Nlp ⭐ 3,278
State of the Art Natural Language Processing
dependent packages 2total releases 90most recent commit a day ago
Linkis ⭐ 3,079
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
most recent commit 18 hours ago
Ibis ⭐ 2,773
The flexibility of Python with the scale and performance of modern SQL.
dependent packages 16total releases 32most recent commit 16 hours agopypi ibis-framework} Downloads
Machine Learning ⭐ 2,305
🌎 machine learning tutorials (mainly in Python3)
most recent commit 21 days ago
Petastorm ⭐ 1,614
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
dependent packages 6total releases 77most recent commit a month agopypi petastorm} Downloads
Spark Py Notebooks ⭐ 1,515
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
most recent commit 2 months ago
Awesome Spark ⭐ 1,461
A curated list of awesome Apache Spark packages and resources.
most recent commit 2 months ago
Mleap ⭐ 1,454
MLeap: Deploy ML Pipelines to Production
dependent packages 12total releases 26most recent commit a month ago
Optimus ⭐ 1,382
🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
total releases 27most recent commit 5 days ago
Sparkmagic ⭐ 1,210
Jupyter magics and kernels for working with remote Spark clusters
dependent packages 5total releases 47most recent commit 2 months agopypi hdijupyterutils} Downloads
Bigflow ⭐ 1,122
Baidu Bigflow is an interface that allows for writing distributed computing programs and provides lots of simple, flexible, powerful APIs. Using Bigflow, you can easily handle data of any scale. Bigflow processes 4P+ data inside Baidu and runs about 10k jobs every day.
most recent commit 7 months ago
Sparkit Learn ⭐ 1,054
PySpark + Scikit-learn = Sparkit-learn
total releases 13most recent commit 2 years ago
Pyspark Example Project ⭐ 1,034
Example project implementing best practices for PySpark ETL jobs and applications.
most recent commit 7 months ago
Pyspark Tutorial ⭐ 959
PySpark-Tutorial provides basic algorithms using PySpark
most recent commit 5 months ago
Sparkling Water ⭐ 945
Sparkling Water provides H2O functionality inside Spark cluster
dependent packages 4total releases 163most recent commit 3 days ago
Hopsworks ⭐ 926
Hopsworks - Data-Intensive AI platform with a Feature Store
total releases 1most recent commit a day ago
Pyspark Examples ⭐ 778
Pyspark RDD, DataFrame and Dataset Examples in Python language
most recent commit 2 months ago
Scriptis ⭐ 767
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
most recent commit a year ago
Devops Python Tools ⭐ 659
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
most recent commit 7 days ago
Kuwala ⭐ 610
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demograp
most recent commit 10 months ago
Eat_pyspark_in_10_days ⭐ 534
pyspark🍒🥭 is delicious，just eat it!😋😋
most recent commit 9 months ago
The 10 Most Depended On Pyspark Open Source Projects
Findspark ⭐ 428
dependent packages 35total releases 14latest release February 11, 2022most recent commit a year agopypi findspark} Downloads
Ibis ⭐ 2,773
The flexibility of Python with the scale and performance of modern SQL.
dependent packages 16total releases 32latest release April 28, 2022most recent commit 16 hours agopypi ibis-framework} Downloads
Mleap ⭐ 1,454
MLeap: Deploy ML Pipelines to Production
dependent packages 12total releases 26latest release May 07, 2021most recent commit a month ago
Tdigest ⭐ 332
t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
dependent packages 12total releases 14latest release May 07, 2019most recent commit 8 months agopypi tdigest} Downloads
Sparkmagic ⭐ 1,210
Jupyter magics and kernels for working with remote Spark clusters
dependent packages 5total releases 47latest release May 02, 2022most recent commit 2 months agopypi hdijupyterutils} Downloads
Sparkling Water ⭐ 945
Sparkling Water provides H2O functionality inside Spark cluster
dependent packages 4total releases 163latest release July 11, 2022most recent commit 3 days ago
Pyspark Stubs ⭐ 116
Apache (Py)Spark type annotations (stub files).
dependent packages 3total releases 38latest release January 06, 2022most recent commit 10 months agopypi pyspark-stubs} Downloads
Spark Nlp ⭐ 3,278
State of the Art Natural Language Processing
dependent packages 2total releases 90latest release March 05, 2021most recent commit a day ago
Hnswlib ⭐ 178
Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs
dependent packages 2total releases 52latest release December 30, 2022most recent commit 5 months ago
The 10 Latest Releases In Pyspark Open Source Projects
Spark Extension ⭐ 104
A library that provides useful extensions to Apache Spark and PySpark.
total releases 46latest release May 05, 2023most recent commit 3 days ago
Hnswlib ⭐ 178
Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs
dependent packages 2total releases 52latest release December 30, 2022most recent commit 5 months ago
Sparkling Water ⭐ 945
Sparkling Water provides H2O functionality inside Spark cluster
dependent packages 4total releases 163latest release July 11, 2022most recent commit 3 days ago
Aut ⭐ 122
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
total releases 25latest release June 17, 2022most recent commit 7 months ago
Anovos ⭐ 78
Anovos - An Open Source Library for Scalable feature engineering Using Apache-Spark
total releases 5latest release May 28, 2022most recent commit 3 months ago
Sparkmagic ⭐ 1,210
Jupyter magics and kernels for working with remote Spark clusters
dependent packages 5total releases 47latest release May 02, 2022most recent commit 2 months agopypi hdijupyterutils} Downloads
Ibis ⭐ 2,773
The flexibility of Python with the scale and performance of modern SQL.
dependent packages 16total releases 32latest release April 28, 2022most recent commit 16 hours agopypi ibis-framework} Downloads
Datacompy ⭐ 289
Pandas and Spark DataFrame comparison for humans
dependent packages 1total releases 10latest release April 19, 2022most recent commit 7 days agopypi datacompy} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Classification
Data Science
Scraper
Transaction
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2023 Awesome Open Source.  All rights reserved.