Awesome Open Source
Awesome Open Source
Combined Topics
airflow x

The Top 1,082 Airflow Open Source Projects

Categories > Control Flow > Airflow
Airflow ⭐ 26,926
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 336total releases 119most recent commit 40 minutes agopypi apache-airflow} Downloads
Diagrams ⭐ 25,319
🎨 Diagram as Code for prototyping cloud system architectures
total releases 14most recent commit 3 days ago
Argo Workflows ⭐ 11,502
Workflow engine for Kubernetes
dependent packages 31total releases 423most recent commit 2 hours ago
Dolphinscheduler ⭐ 8,409
Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.
dependent packages 1total releases 14most recent commit an hour ago
Data Engineering Zoomcamp ⭐ 5,310
Free Data Engineering course!
most recent commit 5 days ago
Pipeline ⭐ 4,113
PipelineAI Kubeflow Distribution
total releases 85most recent commit 2 years ago
Amundsen ⭐ 3,458
Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.
dependent packages 10total releases 67most recent commit 2 days agopypi amundsen-common} Downloads
Docker Airflow ⭐ 3,403
Docker Apache Airflow
most recent commit 6 months ago
Awesome Apache Airflow ⭐ 3,037
Curated list of resources about Apache Airflow
most recent commit 4 days ago
Ploomber ⭐ 2,602
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
dependent packages 5total releases 92most recent commit 2 days agopypi ploomber} Downloads
Awesome Etl ⭐ 2,514
A curated list of awesome ETL frameworks, libraries, and software.
most recent commit 7 months ago
Dataspherestudio ⭐ 2,225
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
dependent packages 1total releases 4most recent commit 7 days ago
Data Engineering Howto ⭐ 2,106
A list of useful resources to learn Data Engineering from scratch
most recent commit 6 months ago
Mara Pipelines ⭐ 1,927
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
total releases 11most recent commit 21 days ago
Elyra ⭐ 1,406
Elyra extends JupyterLab with an AI centric approach.
dependent packages 10total releases 89most recent commit 3 days agonpm @elyra/ui-components} Downloads
Airflow Maintenance Dags ⭐ 1,149
A series of DAGs/Workflows to help maintain the operation of Airflow
most recent commit 6 months ago
Etl With Airflow ⭐ 1,053
ETL best practices with airflow, with examples
most recent commit 8 months ago
Soda Core ⭐ 943
Data reliability tools for SQL- and Spark-accessible data
most recent commit 3 days ago
Around Dataengineering ⭐ 882
A Data Engineering & Machine Learning Knowledge Hub
most recent commit 19 days ago
Airflow Guides ⭐ 774
Guides and docs to help you get up and running with Apache Airflow.
most recent commit 4 days ago
Udacity Data Engineering Projects ⭐ 771
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
most recent commit 2 years ago
Dag Factory ⭐ 741
Dynamically generate Apache Airflow DAGs from YAML configuration files
dependent packages 1total releases 26most recent commit a month agopypi dag-factory} Downloads
Couler ⭐ 694
Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
most recent commit 4 months ago
Airflow Tutorial ⭐ 669
Apache Airflow tutorial
most recent commit 5 months ago
Optimus ⭐ 632
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
dependent packages 1total releases 69most recent commit 43 minutes ago
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 2 years ago
Kube Airflow ⭐ 556
A docker image and kubernetes config files to run Airflow on Kubernetes
most recent commit 3 years ago
Taier ⭐ 440
大数据平台-分布式任务调度系统
total releases 2most recent commit 6 hours ago
Agile_data_code_2 ⭐ 423
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
most recent commit 9 months ago
Astronomer ⭐ 416
Helm Charts for the Astronomer Platform, Apache Airflow as a Service on Kubernetes
most recent commit 2 hours ago
Dataengineeringproject ⭐ 385
Example end to end data engineering project.
most recent commit 6 months ago
Charts ⭐ 370
The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2018, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
most recent commit 2 months ago
Aws Airflow Stack ⭐ 370
Turbine: the bare metals that gets you Airflow
most recent commit 10 months ago
Airflow Tutorial ⭐ 345
Airflow basics tutorial
most recent commit a year ago
Gather Deployment ⭐ 345
Gathers scalable Tensorflow, Python infrastructure deployment and practices, 100% Docker.
most recent commit 4 months ago
Ethereum Etl Airflow ⭐ 277
Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethe
most recent commit 4 days ago
Dagu ⭐ 262
A just another Cron alternative with a Web UI, but with much more capabilities
total releases 57most recent commit 3 days ago
Boundary Layer ⭐ 249
Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform
total releases 59most recent commit a month ago
Airflow Dbt ⭐ 245
Apache Airflow integration for dbt
total releases 7most recent commit 2 months ago
Airflow Rest Api Plugin ⭐ 243
A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces
most recent commit 2 years ago
Airflow Operator ⭐ 241
Kubernetes custom controller and CRDs to managing Airflow
most recent commit 3 years ago
Airflow Chart ⭐ 232
A Helm chart to install Apache Airflow on Kubernetes
most recent commit 3 days ago
Paperboy ⭐ 231
A web frontend for scheduling Jupyter notebook reports
total releases 5most recent commit 6 months ago
Airflow Testing ⭐ 227
Airflow Unit Tests and Integration Tests
most recent commit 5 months ago
Airflow Code Editor ⭐ 226
A plugin for Apache Airflow that allows you to edit DAGs in browser
total releases 37most recent commit 5 days ago
Beginner_de_project ⭐ 222
Beginner data engineering project - batch edition
most recent commit 2 months ago
Example Airflow Dags ⭐ 204
Example DAGs using hooks and operators from Airflow Plugins
most recent commit 4 years ago
Airflow Scheduler Failover Controller ⭐ 201
A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability
most recent commit 2 years ago
Airflow Diagrams ⭐ 192
Auto-generated Diagrams from Airflow DAGs. 🔮 🪄
total releases 15most recent commit 6 days ago
Airflow Exporter ⭐ 190
Airflow plugin to export dag and task based metrics to Prometheus.
total releases 12most recent commit 8 months ago
Airflow Dag Creation Manager Plugin ⭐ 188
A plugin for Airflow that create and manage your DAG with web UI.
most recent commit 3 years ago
Astro Cli ⭐ 187
CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer
total releases 266most recent commit 2 days ago
Data Engineering Projects ⭐ 182
Personal Data Engineering Projects
most recent commit a month ago
Airflow Doc Zh ⭐ 173
📖 [译] Airflow 中文文档
total releases 2most recent commit a year ago
Data Pipelines With Apache Airflow ⭐ 168
Code for Data Pipelines with Apache Airflow
most recent commit a year ago
Airflow_for_beginners ⭐ 166
most recent commit 5 months ago
Whirl ⭐ 163
Fast iterative local development and testing of Apache Airflow workflows
most recent commit 13 days ago
Bitnami Docker Airflow ⭐ 163
Bitnami Docker Image for Apache Airflow
most recent commit 11 days ago
Torchx ⭐ 157
TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.
dependent packages 4total releases 316most recent commit 3 days agopypi torchx-nightly} Downloads
Airflow Pipeline ⭐ 155
An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR
most recent commit 24 days ago
Airflow_in_docker_compose ⭐ 152
Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)
most recent commit 3 months ago
Cwl Airflow ⭐ 151
Python package to extend Airflow functionality with CWL1.1 support
total releases 38most recent commit 5 months ago
Airflow Client Python ⭐ 149
Apache Airflow - OpenApi Client for Python
total releases 6most recent commit 5 days ago
Starthinker ⭐ 143
Reference framework for building data workflows provided by Google. Accelerates authentication, logging, scheduling, and deployment of solutions using GCP. To borrow a tagline.. "The framework for professionals with deadlines."
total releases 15most recent commit 3 days ago
Telemetry Airflow ⭐ 142
Airflow configuration for Telemetry
most recent commit 4 days ago
Tensorflow Lifetime Value ⭐ 140
Predict customer lifetime value using AutoML Tables, or ML Engine with a TensorFlow neural network and the Lifetimes Python library.
most recent commit 2 months ago
Skills Ml ⭐ 136
Data Processing and Machine learning methods for the Open Skills Project
total releases 3most recent commit 9 months ago
Data Testing With Airflow ⭐ 129
most recent commit 3 years ago
Airflow Gcp Examples ⭐ 129
Repository with examples and smoke tests for the GCP Airflow operators and hooks
most recent commit 6 years ago
Microservices Based Algorithmic Trading System ⭐ 128
MBATS is a docker based platform for developing, testing and deploying Algorthmic Trading strategies with a focus on Machine Learning based algorithms.
most recent commit a year ago
Tensorflow Recommendation Wals ⭐ 126
An end-to-end solution for website article recommendations based on Google Analytics data. Uses WALS matrix-factorization in TensorFlow, trained on Cloud ML Engine. Recommendations served with App Engine Flex and Cloud Endpoints. Orchestration is performed using Airflow on Cloud Composer. See the solution tutorials at:
most recent commit 2 years ago
Beyond Jupyter ⭐ 125
🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)
most recent commit 4 years ago
Kedro Airflow ⭐ 120
Kedro-Airflow makes it easy to deploy Kedro projects to Airflow.
most recent commit 4 months ago
Airflow Kube Helm ⭐ 120
most recent commit 2 years ago
Data Science Stack Cookiecutter ⭐ 120
🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)
most recent commit 2 years ago
Airflow On Kubernetes ⭐ 119
A guide to running Airflow on Kubernetes
most recent commit 3 years ago
Gusty ⭐ 115
Making DAG construction easier
total releases 35most recent commit a day ago
Aws Ecs Airflow ⭐ 110
Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
most recent commit a year ago
Kyle School ⭐ 110
쏘카 데이터 그룹 사내 신입/인턴을 대상으로 한 카일 스쿨
most recent commit 2 years ago
Astro Sdk ⭐ 109
Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
most recent commit 3 days ago
Incubator Liminal ⭐ 107
Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.
total releases 25most recent commit a month ago
Airflow Extended Api Plugin ⭐ 105
Yet another Airflow plugin using CLI command as RESTful api, supports Airflow v2.X.
total releases 5most recent commit 3 months ago
Sagemaker Ml Workflow With Apache Airflow ⭐ 104
This repository shows a sample example to build, manage and orchestrate Machine Learning workflows using Amazon Sagemaker and Apache Airflow.
most recent commit a year ago
Hark Lang ⭐ 104
A serverless virtual machine
most recent commit a month ago
Afctl ⭐ 103
afctl helps to manage and deploy Apache Airflow projects faster and smoother.
total releases 5most recent commit 2 years ago
Airflow Prometheus Exporter ⭐ 100
Prometheus Exporter for Airflow
most recent commit 2 years ago
Airflow Repo Template ⭐ 98
The easiest way to run Airflow locally, with linting & tests for valid DAGs and Plugins.
most recent commit a year ago
Streamify ⭐ 97
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
most recent commit 4 months ago
Public Datasets Pipelines ⭐ 97
Cloud-native, data onboarding architecture for Google Cloud Datasets
most recent commit 4 days ago
Incremental_training ⭐ 96
Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and MLFlow'
most recent commit 5 months ago
Ap Airflow ⭐ 89
Astronomer Core Docker Images
most recent commit 2 days ago
Airflow Site ⭐ 88
Apache Airflow Website
dependent packages 3total releases 6most recent commit 5 days agopypi sphinx-airflow-theme} Downloads
Airflow Autoscaling Ecs ⭐ 87
Airflow Deployment on AWS ECS Fargate Using Cloudformation
most recent commit 2 years ago
Airflow Provider Great Expectations ⭐ 86
Great Expectations Airflow operator
dependent packages 2total releases 13most recent commit 5 months agopypi airflow-provider-great-expectations} Downloads
Airflow Tutorial ⭐ 86
Use Airflow to move data from multiple MySQL databases to BigQuery
most recent commit 2 years ago
Ge_tutorials ⭐ 85
Learn how to add data validation and documentation to a data pipeline built with dbt and Airflow.
most recent commit a year ago
Datascience Group ⭐ 84
AWSKRUG Datascience Group
most recent commit 8 days ago
Viewflow ⭐ 84
Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.
most recent commit a year ago
Udacity Data Eng Proj 1 ⭐ 81
Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3
most recent commit 9 months ago
Skills Api ⭐ 81
Skills API
most recent commit 4 years ago
1-100 of 1,082 projects
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 71
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 343
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 115
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,072,185)
Python  (819,783)
Java  (391,684)
Php  (287,417)
Typescript  (254,498)
C Plus Plus  (241,642)
Ruby  (222,126)
C  (183,493)
Shell  (172,974)
C Sharp  (166,272)
Golang  (160,486)
Swift  (65,800)
Rust  (59,273)
R  (58,650)
Objective C  (57,122)
Dart  (53,962)
Kotlin  (47,557)
Lua  (34,573)
Matlab  (31,397)
Perl  (30,575)
Language  (29,995)
Scala  (28,782)
Haskell  (20,487)
Powershell  (19,264)
Clojure  (19,187)
Bash  (18,782)
Coffeescript  (17,262)
Elixir  (16,765)
Assembly  (14,766)
Processing  (13,289)
Julia  (12,999)
Basic  (10,723)
Flow  (10,535)
Emacs Lisp  (10,457)
Erlang  (9,062)
Groovy  (8,461)
Scheme  (8,414)
Solidity  (8,315)
Ocaml  (6,744)
Pascal  (6,484)
Hack  (6,420)
Lisp  (6,337)
Fortran  (5,581)
Elm  (5,475)
Common Lisp  (5,323)
Dlang  (4,746)
Shell Script  (4,684)
Python Library  (4,467)
Crystal  (4,388)
Scripting  (4,216)
Top Projects
Freecodecamp ⭐ 351,075
996.icu ⭐ 263,365
Free Programming Books ⭐ 243,838
Coding Interview University ⭐ 228,177
Awesome ⭐ 211,770
Developer Roadmap ⭐ 205,172
Public Apis ⭐ 204,370
Vue ⭐ 198,403
React ⭐ 192,810
System Design Primer ⭐ 191,492
Tensorflow ⭐ 166,925
Build Your Own X ⭐ 160,418
Bootstrap ⭐ 158,868
You Dont Know Js ⭐ 157,035
Cs Notes ⭐ 153,924
Ohmyzsh ⭐ 148,511
Javascript Algorithms ⭐ 146,946
Flutter ⭐ 143,329
Python ⭐ 141,094
Gitignore ⭐ 136,695
Awesome Python ⭐ 136,585
Linux ⭐ 135,931
Vscode ⭐ 135,067
Javaguide ⭐ 125,952
Javascript ⭐ 125,105
Python 100 Days ⭐ 121,533
Computer Science ⭐ 121,209
Youtube Dl ⭐ 112,146
The Art Of Command Line ⭐ 108,745
Fucking Algorithm ⭐ 108,035
React Native ⭐ 104,167
Electron ⭐ 102,951
Go ⭐ 102,514
D3 ⭐ 101,796
30 Seconds Of Code ⭐ 100,507
Awesome Selfhosted ⭐ 97,785
Create React App ⭐ 96,344
Axios ⭐ 94,983
Free Programming Books Zh_cn ⭐ 94,762
Kubernetes ⭐ 90,903
Next.js ⭐ 90,713
Node ⭐ 89,307
Awesome Go ⭐ 85,358
Terminal ⭐ 84,440
Deno ⭐ 84,421
Three.js ⭐ 84,182
Angular ⭐ 83,091
Typescript ⭐ 83,014
Ant Design ⭐ 81,341
Material Ui ⭐ 80,305
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.