Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Data Engineering Open Source Projects

Open source projects categorized as Data Engineering
Categories > Data Processing > Data Engineering
Superset ⭐ 58,778
Apache Superset is a Data Visualization and Data Exploration Platform
dependent packages 21total releases 6most recent commit 3 hours agonpm @superset-ui/switchboard} Downloads
Made With Ml ⭐ 35,496
Learn how to design, develop, deploy and iterate on production-grade ML applications.
total releases 5most recent commit 5 months ago
Airflow ⭐ 34,299
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169most recent commit 15 days agopypi apache-airflow} Downloads
Applied Ml ⭐ 24,828
📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
most recent commit 6 months ago
Data Engineering Zoomcamp ⭐ 19,461
Free Data Engineering course!
most recent commit 3 months ago
Prefect ⭐ 14,603
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
dependent packages 152total releases 249most recent commit 2 days agopypi prefect} Downloads
Argo Workflows ⭐ 14,264
Workflow Engine for Kubernetes
dependent packages 51total releases 449most recent commit 6 days ago
Airbyte ⭐ 12,918
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
dependent packages 11total releases 311most recent commit 3 months agopypi airbyte-cdk} Downloads
Cookbook ⭐ 12,557
The Data Engineering Cookbook
most recent commit 4 months ago
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585most recent commit 3 months agopypi dagster} Downloads
Great_expectations ⭐ 9,179
Always know what to expect from your data.
dependent packages 53total releases 256most recent commit 3 months agopypi great-expectations} Downloads
Data Engineer Roadmap ⭐ 9,131
Roadmap to becoming a data engineer in 2021
most recent commit 2 years ago
Xonsh ⭐ 7,986
🐚 Python-powered, cross-platform, Unix-gazing shell.
dependent packages 80total releases 125most recent commit 5 days agopypi xonsh} Downloads
Benthos ⭐ 7,564
Fancy stream processing made operationally mundane
dependent packages 12total releases 51most recent commit 2 days ago
Mage Ai ⭐ 6,324
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
total releases 314most recent commit 3 months ago
Risingwave ⭐ 5,799
The distributed streaming database. Engineered to offer the simplest and most cost-efficient way for stream processing and management.
total releases 14most recent commit 3 months ago
Data Engineer Handbook ⭐ 5,650
This is a repo with links to everything you'd ever want to learn about data engineering
most recent commit 3 months ago
Cloudquery ⭐ 5,380
The open source high performance data integration platform built for developers.
dependent packages 7total releases 345most recent commit 3 months ago
Growthbook ⭐ 5,285
Open Source Feature Flagging and A/B Testing Platform
dependent packages 11total releases 49most recent commit 3 months agonpm @growthbook/growthbook} Downloads
Kestra ⭐ 5,257
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
dependent packages 4total releases 58most recent commit 3 months ago
Feast ⭐ 5,053
Feature Store for Machine Learning
dependent packages 28total releases 116most recent commit 3 months agopypi feast} Downloads
Taipy ⭐ 4,311
Turns Data and AI algorithms into production-ready web applications in no time.
total releases 26most recent commit 3 months ago
Lakefs ⭐ 3,900
lakeFS - Data version control for your data lake | Git for data
dependent packages 5total releases 83most recent commit 3 months ago
The 10 Most Depended On Data Engineering Open Source Projects
Airflow ⭐ 34,299
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169latest release November 27, 2023most recent commit 15 days agopypi apache-airflow} Downloads
Egeria ⭐ 742
Egeria core
dependent packages 234total releases 37latest release September 11, 2023most recent commit 3 months ago
Prefect ⭐ 14,603
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
dependent packages 152total releases 249latest release December 08, 2023most recent commit 2 days agopypi prefect} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Xonsh ⭐ 7,986
:shell: Python-powered, cross-platform, Unix-gazing shell.
dependent packages 80total releases 125latest release December 05, 2023most recent commit 5 days agopypi xonsh} Downloads
Great_expectations ⭐ 9,179
Always know what to expect from your data.
dependent packages 53total releases 256latest release December 08, 2023most recent commit 3 months agopypi great-expectations} Downloads
Argo Workflows ⭐ 14,264
Workflow Engine for Kubernetes
dependent packages 51total releases 449latest release November 27, 2023most recent commit 6 days ago
Quilt ⭐ 1,299
Quilt is a data mesh for connecting people with actionable data
dependent packages 38total releases 60latest release November 29, 2023most recent commit 3 months agopypi quilt3} Downloads
Grai Core ⭐ 254
dependent packages 29total releases 34latest release November 10, 2023most recent commit 3 months agopypi grai-schemas} Downloads
The 10 Latest Releases In Data Engineering Open Source Projects
Feathub ⭐ 255
FeatHub - A stream-batch unified feature store for real-time machine learning
total releases 371latest release December 12, 2023most recent commit 7 months ago
Jupysql ⭐ 261
Better SQL in Jupyter. 📊
dependent packages 2total releases 38latest release December 11, 2023most recent commit a year agopypi jupysql} Downloads
Great_expectations ⭐ 9,179
Always know what to expect from your data.
dependent packages 53total releases 256latest release December 08, 2023most recent commit 3 months agopypi great-expectations} Downloads
Prefect ⭐ 14,603
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
dependent packages 152total releases 249latest release December 08, 2023most recent commit 2 days agopypi prefect} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Snowpark Python ⭐ 215
Snowflake Snowpark Python API
dependent packages 22total releases 22latest release December 07, 2023most recent commit 3 months agopypi snowflake-snowpark-python} Downloads
Conduit ⭐ 321
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
total releases 446latest release December 07, 2023most recent commit 3 months ago
Dataform ⭐ 757
Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
dependent packages 15total releases 222latest release December 07, 2023most recent commit 3 months agonpm @dataform/core} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.