Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Python Apache Spark Open Source Projects

Open source projects categorized as Python Apache Spark
Categories > Data Processing > Apache Spark
Spark ⭐ 37,661
Apache Spark - A unified analytics engine for large-scale data processing
dependent packages 939total releases 46most recent commit 3 months ago
Mlflow ⭐ 16,343
Open source platform for the machine learning lifecycle
dependent packages 440total releases 86most recent commit 3 months agopypi mlflow} Downloads
Bigdl ⭐ 4,728
Accelerate LLM with low-bit (FP4 / INT4 / FP8 / INT8) optimizations using bigdl-llm
dependent packages 10total releases 16most recent commit 3 months ago
Koalas ⭐ 3,291
Koalas: pandas API on Apache Spark
dependent packages 16total releases 47most recent commit 7 months agopypi koalas} Downloads
Analytics Zoo ⭐ 2,592
Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
dependent packages 3total releases 1most recent commit 4 months agopypi bigdl-orca-spark321} Downloads
Sparkit Learn ⭐ 1,054
PySpark + Scikit-learn = Sparkit-learn
total releases 13most recent commit 3 years ago
Spark Sklearn ⭐ 1,039
(Deprecated) Scikit-learn integration package for Apache Spark
total releases 8most recent commit 4 years ago
Splink ⭐ 939
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
dependent packages 2total releases 119most recent commit 3 months agopypi splink} Downloads
Incubator Livy ⭐ 840
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
dependent packages 8total releases 6most recent commit 3 months ago
Flintrock ⭐ 627
A command-line tool for launching Apache Spark clusters.
total releases 14most recent commit 5 months ago
Dist Keras ⭐ 611
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
total releases 2most recent commit 6 years ago
Sparkmeasure ⭐ 603
This is the development repository for sparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task and stage metrics data.
dependent packages 2total releases 9most recent commit 5 months ago
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 4 years ago
Agile_data_code_2 ⭐ 435
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
most recent commit a year ago
Sparkflow ⭐ 301
Easy to use library to bring Tensorflow on Apache Spark
total releases 13most recent commit 6 months ago
Sparktorch ⭐ 297
Train and run Pytorch models on Apache Spark.
total releases 11most recent commit a year ago
Spark Programming In Python ⭐ 269
Apache Spark 3 - Spark Programming in Python for Beginners
most recent commit 4 months ago
Pysparkling ⭐ 253
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
dependent packages 1total releases 69most recent commit a year agopypi pysparkling} Downloads
Sql Data Analysis And Visualization Projects ⭐ 200
SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
most recent commit 2 years ago
Bigdata Playground ⭐ 154
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
most recent commit 5 years ago
Albedo ⭐ 142
A recommender system for discovering GitHub repos, built with Apache Spark
most recent commit 4 years ago
Sparknotebook ⭐ 142
An example of running Apache Spark using Scala in ipython notebook
most recent commit 9 years ago
Pyspark Cheatsheet ⭐ 140
PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
most recent commit 2 years ago
The 10 Most Depended On Python Apache Spark Open Source Projects
Spark ⭐ 37,661
Apache Spark - A unified analytics engine for large-scale data processing
dependent packages 939total releases 46latest release May 09, 2021most recent commit 3 months ago
Mlflow ⭐ 16,343
Open source platform for the machine learning lifecycle
dependent packages 440total releases 86latest release December 07, 2023most recent commit 3 months agopypi mlflow} Downloads
Koalas ⭐ 3,291
Koalas: pandas API on Apache Spark
dependent packages 16total releases 47latest release October 19, 2021most recent commit 7 months agopypi koalas} Downloads
Bigdl ⭐ 4,728
Accelerate LLM with low-bit (FP4 / INT4 / FP8 / INT8) optimizations using bigdl-llm
dependent packages 10total releases 16latest release April 19, 2021most recent commit 3 months ago
Incubator Livy ⭐ 840
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
dependent packages 8total releases 6latest release September 25, 2023most recent commit 3 months ago
Pyspark Stubs ⭐ 116
Apache (Py)Spark type annotations (stub files).
dependent packages 4total releases 38latest release January 06, 2022most recent commit 2 years agopypi pyspark-stubs} Downloads
Analytics Zoo ⭐ 2,592
Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
dependent packages 3total releases 1latest release July 29, 2022most recent commit 4 months agopypi bigdl-orca-spark321} Downloads
Splink ⭐ 939
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
dependent packages 2total releases 119latest release November 14, 2023most recent commit 3 months agopypi splink} Downloads
The 10 Latest Releases In Python Apache Spark Open Source Projects
Mlflow ⭐ 16,343
Open source platform for the machine learning lifecycle
dependent packages 440total releases 86latest release December 07, 2023most recent commit 3 months agopypi mlflow} Downloads
Flintrock ⭐ 627
A command-line tool for launching Apache Spark clusters.
total releases 14latest release November 27, 2023most recent commit 5 months ago
Splink ⭐ 939
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
dependent packages 2total releases 119latest release November 14, 2023most recent commit 3 months agopypi splink} Downloads
Incubator Livy ⭐ 840
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
dependent packages 8total releases 6latest release September 25, 2023most recent commit 3 months ago
Sparkql ⭐ 10
sparkql: Apache Spark SQL DataFrame schema management for sensible humans
total releases 20latest release September 18, 2023most recent commit 7 months ago
Dataproc Templates ⭐ 103
Dataproc templates and pipelines for solving simple in-cloud data tasks
total releases 8latest release August 07, 2023most recent commit 3 months ago
Chopin2 ⭐ 6
Domain-Agnostic Supervised Learning with Hyperdimensional Computing
total releases 8latest release January 27, 2023most recent commit 7 months ago
Aut ⭐ 128
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
total releases 27latest release November 17, 2022most recent commit 9 months ago
Pysparkling ⭐ 253
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
dependent packages 1total releases 69latest release November 13, 2022most recent commit a year agopypi pysparkling} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.