Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Workflow Data Engineering Open Source Projects

Open source projects categorized as Workflow Data Engineering
Categories > Data Processing > Data Engineering
Airflow ⭐ 34,468
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169most recent commit 2 days agopypi apache-airflow} Downloads
Prefect ⭐ 14,603
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
dependent packages 152total releases 249most recent commit 4 days agopypi prefect} Downloads
Argo Workflows ⭐ 14,264
Workflow Engine for Kubernetes
dependent packages 51total releases 449most recent commit 8 days ago
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585most recent commit 3 months agopypi dagster} Downloads
Kestra ⭐ 5,257
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
dependent packages 4total releases 58most recent commit 3 months ago
Taipy ⭐ 4,311
Turns Data and AI algorithms into production-ready web applications in no time.
total releases 26most recent commit 3 months ago
Ploomber ⭐ 3,318
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
dependent packages 7total releases 115most recent commit 3 months agopypi ploomber} Downloads
Mlrun ⭐ 1,177
Machine Learning automation and tracking
total releases 486most recent commit 3 months ago
Active_workflow ⭐ 675
Polyglot workflows without leaving the comfort of your technology stack.
most recent commit a year ago
Ui ⭐ 176
The home of the Prefect 1 UI
most recent commit 4 months ago
Dataplane ⭐ 171
Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
total releases 30most recent commit 4 months ago
D6t Python ⭐ 118
Accelerate data science
most recent commit 3 years ago
Viewflow ⭐ 84
Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.
most recent commit 3 years ago
Soorgeon ⭐ 73
Convert monolithic Jupyter notebooks 📙 into maintainable Ploomber pipelines. 📊
total releases 19most recent commit 3 months ago
Doltpy ⭐ 53
A Python API for Dolt
dependent packages 2total releases 60most recent commit a year agopypi doltpy} Downloads
Pyjaws ⭐ 36
PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows
most recent commit 5 months ago
Didact Engine ⭐ 34
The REST API and execution engine for the Didact Platform.
most recent commit 4 months ago
Debussy_concert ⭐ 29
Debussy is an opinionated Data Architecture and Engineering framework, enabling data analysts and engineers to build better platforms and pipelines.
total releases 2most recent commit a year ago
Airflow Docker ⭐ 19
This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.
most recent commit 5 months ago
Prefect Saturn ⭐ 15
Python client for using Prefect Cloud with Saturn Cloud
most recent commit 2 years ago
Thepipelinetool ⭐ 13
A pipeline orchestration tool
most recent commit 3 months ago
Dagger ⭐ 9
Define sophisticated data pipelines with Python and run them on different distributed systems (such as Argo Workflows).
dependent packages 1total releases 11most recent commit 2 years agopypi py-dagger} Downloads
Data Engineering Onboarding Starter ⭐ 8
This repository contains a 10 step program to enter the world of Data Engineering
most recent commit 7 months ago
The 8 Most Depended On Workflow Data Engineering Open Source Projects
Airflow ⭐ 34,468
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169latest release November 27, 2023most recent commit 2 days agopypi apache-airflow} Downloads
Prefect ⭐ 14,603
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
dependent packages 152total releases 249latest release December 08, 2023most recent commit 4 days agopypi prefect} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Argo Workflows ⭐ 14,264
Workflow Engine for Kubernetes
dependent packages 51total releases 449latest release November 27, 2023most recent commit 8 days ago
Ploomber ⭐ 3,318
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
dependent packages 7total releases 115latest release November 29, 2023most recent commit 3 months agopypi ploomber} Downloads
Doltpy ⭐ 53
A Python API for Dolt
dependent packages 2total releases 60latest release January 30, 2023most recent commit a year agopypi doltpy} Downloads
Dagger ⭐ 9
Define sophisticated data pipelines with Python and run them on different distributed systems (such as Argo Workflows).
dependent packages 1total releases 11latest release September 30, 2021most recent commit 2 years agopypi py-dagger} Downloads
The 10 Latest Releases In Workflow Data Engineering Open Source Projects
Prefect ⭐ 14,603
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
dependent packages 152total releases 249latest release December 08, 2023most recent commit 4 days agopypi prefect} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Mlrun ⭐ 1,177
Machine Learning automation and tracking
total releases 486latest release December 07, 2023most recent commit 3 months ago
Ploomber ⭐ 3,318
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
dependent packages 7total releases 115latest release November 29, 2023most recent commit 3 months agopypi ploomber} Downloads
Argo Workflows ⭐ 14,264
Workflow Engine for Kubernetes
dependent packages 51total releases 449latest release November 27, 2023most recent commit 8 days ago
Airflow ⭐ 34,468
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169latest release November 27, 2023most recent commit 2 days agopypi apache-airflow} Downloads
Taipy ⭐ 4,311
Turns Data and AI algorithms into production-ready web applications in no time.
total releases 26latest release October 27, 2023most recent commit 3 months ago
Doltpy ⭐ 53
A Python API for Dolt
dependent packages 2total releases 60latest release January 30, 2023most recent commit a year agopypi doltpy} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.