Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Spark Data Engineering Open Source Projects

Open source projects categorized as Spark Data Engineering
Categories > Data Processing > Data Engineering
Data Engineering Zoomcamp ⭐ 19,461
Free Data Engineering course!
most recent commit 3 months ago
Cookbook ⭐ 12,557
The Data Engineering Cookbook
most recent commit 4 months ago
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585most recent commit 3 months agopypi dagster} Downloads
Mage Ai ⭐ 6,324
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
total releases 314most recent commit 3 months ago
Risingwave ⭐ 5,799
The distributed streaming database. Engineered to offer the simplest and most cost-efficient way for stream processing and management.
total releases 14most recent commit 3 months ago
Awesome Opensource Data Engineering ⭐ 1,331
An Awesome List of Open-Source Data Engineering Projects
most recent commit 4 months ago
Pyspark Example Project ⭐ 1,034
Example project implementing best practices for PySpark ETL jobs and applications.
most recent commit a year ago
Around Dataengineering ⭐ 926
A Data Engineering & Machine Learning Knowledge Hub
most recent commit 2 years ago
Zingg ⭐ 828
Scalable identity resolution, entity resolution, data mastering and deduplication using ML
total releases 1most recent commit 3 months ago
Blaze ⭐ 784
Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.
most recent commit 3 months ago
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 4 years ago
Data Engineering Interview Questions ⭐ 554
More than 2000+ Data engineer interview questions.
most recent commit 7 months ago
Data Engineering Projects ⭐ 322
Personal Data Engineering Projects
most recent commit a year ago
Every Single Day I Tldr ⭐ 311
A daily digest of the articles or videos I've found interesting, that I want to share with you.
most recent commit 3 months ago
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35most recent commit 5 months agopypi butterfree} Downloads
Geni ⭐ 268
A Clojure dataframe library that runs on Spark
total releases 33most recent commit 5 months ago
Setl ⭐ 173
A simple Spark-powered ETL framework that just works 🍺
total releases 4most recent commit 5 months ago
Spark Alchemy ⭐ 169
Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive
total releases 4most recent commit a year ago
Lakehouse Engine ⭐ 154
The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.
most recent commit 3 months ago
Scalable Data Science Platform ⭐ 153
Content for architecting a data science platform for products using Luigi, Spark & Flask.
most recent commit 4 years ago
Big Data Mapreduce Course ⭐ 135
Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University
most recent commit 5 months ago
Movalytics Data Warehouse ⭐ 116
Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
most recent commit 4 years ago
De Zoomcamp Ui ⭐ 107
🎨 UI for the Free Data Engineering Zoomcamp 2023 Course provided by DataTalksClub
most recent commit 3 months ago
The 6 Most Depended On Spark Data Engineering Open Source Projects
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35latest release November 14, 2023most recent commit 5 months agopypi butterfree} Downloads
Soda Spark ⭐ 49
Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes
dependent packages 1total releases 11latest release May 11, 2022most recent commit 2 years agopypi soda-spark} Downloads
Waimak ⭐ 73
Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
dependent packages 1total releases 31latest release August 16, 2021most recent commit a year ago
Gallia Core ⭐ 79
A schema-aware Scala library for data transformation
dependent packages 1total releases 6latest release September 19, 2023most recent commit 4 months ago
The 10 Latest Releases In Spark Data Engineering Open Source Projects
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Sageworks ⭐ 36
SageWorks: An easy to use Python API for creating and deploying SageMaker Models
total releases 60latest release November 29, 2023most recent commit 3 months ago
Yaetos ⭐ 32
Write data & AI pipelines in (SQL, Spark, Pandas) and deploy to the cloud, simplified
total releases 32latest release November 27, 2023most recent commit 3 months ago
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35latest release November 14, 2023most recent commit 5 months agopypi butterfree} Downloads
Spark Ai ⭐ 31
Toolbox for building Generative AI applications on top of Apache Spark.
total releases 6latest release October 17, 2023most recent commit 3 months ago
Gallia Core ⭐ 79
A schema-aware Scala library for data transformation
dependent packages 1total releases 6latest release September 19, 2023most recent commit 4 months ago
Marshmallow Pyspark ⭐ 12
Marshmallow serializer integration with pyspark
total releases 5latest release November 11, 2022most recent commit 4 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.