Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Pipeline Data Engineering Open Source Projects

Open source projects categorized as Pipeline Data Engineering
Categories > Data Processing > Data Engineering
Categories > Data Processing > Pipeline
Prefect ⭐ 14,339
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
dependent packages 152total releases 249most recent commit 22 days agopypi prefect} Downloads
Argo Workflows ⭐ 14,227
Workflow Engine for Kubernetes
dependent packages 51total releases 449most recent commit 7 days ago
Airbyte ⭐ 12,918
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
dependent packages 11total releases 311most recent commit 3 months agopypi airbyte-cdk} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585most recent commit 3 months agopypi dagster} Downloads
Great_expectations ⭐ 9,179
Always know what to expect from your data.
dependent packages 53total releases 256most recent commit 3 months agopypi great-expectations} Downloads
Mage Ai ⭐ 6,324
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
total releases 314most recent commit 3 months ago
Kestra ⭐ 5,257
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
dependent packages 4total releases 58most recent commit 3 months ago
Taipy ⭐ 4,311
Turns Data and AI algorithms into production-ready web applications in no time.
total releases 26most recent commit 3 months ago
Ploomber ⭐ 3,318
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
dependent packages 7total releases 115most recent commit 3 months agopypi ploomber} Downloads
Meltano ⭐ 1,460
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
dependent packages 1total releases 263most recent commit 3 months agopypi meltano} Downloads
Mlrun ⭐ 1,177
Machine Learning automation and tracking
total releases 486most recent commit 3 months ago
Neumai ⭐ 693
Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.
most recent commit 3 months ago
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 4 years ago
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181most recent commit 3 months agopypi vdk-core} Downloads
Data Engineering With Python ⭐ 302
Data Engineering with Python, published by Packt
most recent commit a year ago
Dataengineering Roadmap ⭐ 297
Un repositorio más con conceptos básicos, desafíos técnicos y recursos sobre ingeniería de datos en español 🧙✨
most recent commit 4 months ago
Yuniql ⭐ 292
Free and open source schema versioning and database migration made natively with .NET/6. NEW THIS MAY 2022! v1.3.15 released!
dependent packages 7total releases 25most recent commit 2 years agonuget Yuniql.Extensibility} Downloads
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35most recent commit 5 months agopypi butterfree} Downloads
Cuelake ⭐ 266
Use SQL to build ELT pipelines on a data lakehouse.
most recent commit 2 years ago
Pipelinex ⭐ 212
PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more
total releases 48most recent commit 5 months ago
Pureml ⭐ 174
Developer platform for production ML.
dependent packages 1total releases 29most recent commit 10 months agopypi pureml} Downloads
Setl ⭐ 173
A simple Spark-powered ETL framework that just works 🍺
total releases 4most recent commit 4 months ago
Dataplane ⭐ 171
Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
total releases 30most recent commit 4 months ago
The 10 Most Depended On Pipeline Data Engineering Open Source Projects
Prefect ⭐ 14,339
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
dependent packages 152total releases 249latest release December 08, 2023most recent commit 22 days agopypi prefect} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Great_expectations ⭐ 9,179
Always know what to expect from your data.
dependent packages 53total releases 256latest release December 08, 2023most recent commit 3 months agopypi great-expectations} Downloads
Argo Workflows ⭐ 14,227
Workflow Engine for Kubernetes
dependent packages 51total releases 449latest release November 27, 2023most recent commit 7 days ago
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181latest release November 28, 2023most recent commit 3 months agopypi vdk-core} Downloads
Yuniql ⭐ 292
Free and open source schema versioning and database migration made natively with .NET/6. NEW THIS MAY 2022! v1.3.15 released!
dependent packages 7total releases 25latest release May 25, 2022most recent commit 2 years agonuget Yuniql.Extensibility} Downloads
Ploomber ⭐ 3,318
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
dependent packages 7total releases 115latest release November 29, 2023most recent commit 3 months agopypi ploomber} Downloads
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35latest release November 14, 2023most recent commit 5 months agopypi butterfree} Downloads
The 10 Latest Releases In Pipeline Data Engineering Open Source Projects
Great_expectations ⭐ 9,179
Always know what to expect from your data.
dependent packages 53total releases 256latest release December 08, 2023most recent commit 3 months agopypi great-expectations} Downloads
Prefect ⭐ 14,339
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
dependent packages 152total releases 249latest release December 08, 2023most recent commit 22 days agopypi prefect} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Mlrun ⭐ 1,177
Machine Learning automation and tracking
total releases 486latest release December 07, 2023most recent commit 3 months ago
Ploomber ⭐ 3,318
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
dependent packages 7total releases 115latest release November 29, 2023most recent commit 3 months agopypi ploomber} Downloads
Drivers ⭐ 53
Low-code Python library enabling access to APIs, tools, data sources in seconds.
total releases 403latest release November 29, 2023most recent commit 3 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.