Awesome Open Source
Awesome Open Source
Combined Topics
airflow
x
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
The Top 384 Airflow Open Source Projects on Github
Categories
>
Control Flow
>
Airflow
Airflow
⭐
23,500
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Argo Workflows
⭐
9,493
Workflow engine for Kubernetes
Dolphinscheduler
⭐
6,579
Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.
Pipeline
⭐
4,113
PipelineAI Kubeflow Distribution
Docker Airflow
⭐
3,301
Docker Apache Airflow
Awesome Apache Airflow
⭐
2,667
Curated list of resources about Apache Airflow
Dataspherestudio
⭐
1,612
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Elyra
⭐
1,112
Elyra extends JupyterLab Notebooks with an AI centric approach.
Airflow Maintenance Dags
⭐
1,048
A series of DAGs/Workflows to help maintain the operation of Airflow
Around Dataengineering
⭐
698
A Data Engineering & Machine Learning Knowledge Hub
Ploomber
⭐
638
Write maintainable, production-ready pipelines using Jupyter or your favorite text editor. Develop locally, deploy to the cloud. ☁️
Goodreads_etl_pipeline
⭐
593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Dag Factory
⭐
572
Dynamically generate Apache Airflow DAGs from YAML configuration files
Optimus
⭐
548
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Soda Sql
⭐
450
Data profiling, testing, and monitoring for SQL accessible data.
Agile_data_code_2
⭐
420
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
Aws Airflow Stack
⭐
370
Turbine: the bare metals that gets you Airflow
Airflow Tutorial
⭐
345
Airflow basics tutorial
Gather Deployment
⭐
339
Gathers scalable Tensorflow and Python infrastructure deployment, Husein Go-To for development, 100% Docker.
Dataengineeringproject
⭐
265
Example end to end data engineering project.
Airflow Rest Api Plugin
⭐
243
A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces
Airflow Operator
⭐
241
Kubernetes custom controller and CRDs to managing Airflow
Charts
⭐
237
Airflow Helm Chart (User Community) - used to deploy Apache Airflow on Kubernetes
Paperboy
⭐
228
A web frontend for scheduling Jupyter notebook reports
Example Airflow Dags
⭐
204
Example DAGs using hooks and operators from Airflow Plugins
Airflow Scheduler Failover Controller
⭐
201
A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability
Airflow Testing
⭐
195
Airflow Unit Tests and Integration Tests
Udacity Data Engineering Projects
⭐
191
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
Airflow Chart
⭐
189
A Helm chart to install Apache Airflow on Kubernetes
Airflow Exporter
⭐
181
Airflow plugin to export dag and task based metrics to Prometheus.
Airflow Doc Zh
⭐
173
📖 [译] Airflow 中文文档
Airflow Dbt
⭐
165
Apache Airflow integration for dbt
Airflow Code Editor
⭐
153
A plugin for Apache Airflow that allows you to edit DAGs in browser
Airflow Pipeline
⭐
139
An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR
Telemetry Airflow
⭐
133
Airflow configuration for Telemetry
Whirl
⭐
131
Fast iterative local development and testing of Apache Airflow workflows
Beyond Jupyter
⭐
125
🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)
Bitnami Docker Airflow
⭐
122
Bitnami Docker Image for Apache Airflow
Data Science Stack Cookiecutter
⭐
120
🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)
Aws Ecs Airflow
⭐
110
Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
Afctl
⭐
103
afctl helps to manage and deploy Apache Airflow projects faster and smoother.
Kedro Airflow
⭐
94
Kedro-Airflow makes it easy to deploy Kedro projects to Airflow.
Airflow_in_docker_compose
⭐
88
Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)
Airflow Autoscaling Ecs
⭐
87
Airflow Deployment on AWS ECS Fargate Using Cloudformation
Airflow Tutorial
⭐
86
Use Airflow to move data from multiple MySQL databases to BigQuery
Gusty
⭐
85
Making DAG construction easier
Incubator Liminal
⭐
84
Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.
Viewflow
⭐
84
Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.
Incremental_training
⭐
72
Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and MLFlow'
Terraform Aws Airflow
⭐
72
Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor
Beginner_de_project
⭐
67
Beginner data engineering project - batch edition
Airflow Spark Operator Plugin
⭐
66
A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator
Ap Airflow
⭐
65
Astronomer Core Docker Images
Airflow Toolkit
⭐
62
Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥 >> [ 🚀, 🚢 ]
Discreetly
⭐
62
ETLy is an add-on dashboard service on top of Apache Airflow.
Airflow_multi_dagrun
⭐
62
triggering a DAG run multiple times
Public Datasets Pipelines
⭐
61
Cloud-native, data onboarding architecture for Google Cloud Datasets
Airflow Role
⭐
56
Ansible role to install Apache Airflow
Airflow Cookbook
⭐
55
Airflow workflow management platform chef cookbook.
Cccatalog
⭐
54
Note: Project is discontinued. Mapping the commons towards an open ledger and cc search.
Airflow Site
⭐
53
Apache Airflow Website
Objinsync
⭐
52
Continuously synchronize directories from remote object store to local filesystem
Xene
⭐
52
A distributed workflow runner focusing on performance and simplicity.
Airflow Training
⭐
51
Airflow training for the crunch conf
Aircal
⭐
50
Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.
Udacity Data Engineering
⭐
42
Udacity Data Engineering Nano Degree (DEND)
Ansible Role Airflow
⭐
41
Ansible role to deploy and configure Airflow
Airflow Boilerplate
⭐
40
A complete development environment setup for working with Airflow
Airflow Clickhouse Plugin
⭐
39
Airflow ClickHouse Plugin based on clickhouse-driver
Airflow Client Go
⭐
37
Apache Airflow - OpenApi Client for Go
Ecs Airflow
⭐
32
Cloudformation templates for deploying Airflow in ECS
Autohub
⭐
32
基于Python开源技术生态的企业级自动化测试集成解决方案
Kubernetesjoboperator
⭐
31
An airflow operator that executes a task in a kubernetes cluster, given a kubernetes yaml configuration or an image refrence.
Airflow Client Python
⭐
30
Apache Airflow - OpenApi Client for Python
Fairflow
⭐
29
Functional Airflow DAG definitions.
Automating Your Data Pipeline With Apache Airflow
⭐
28
Automating Your Data Pipeline with Apache Airflow
Docker Airflow
⭐
28
Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/
Movalytics Data Warehouse
⭐
27
Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
Databook
⭐
26
A facebook for data
Airflow Testing Ci Workflow
⭐
26
(project & tutorial) dag pipeline tests + ci/cd setup
Dbt Airflow Docker Compose
⭐
26
Execution of DBT models using Apache Airflow through Docker Compose
Airflow Ms Teams Operator
⭐
25
Airflow operator that can send messages to MS Teams
Fastetl
⭐
25
Plugins do Airflow para implementação de pipelines de dados
Airflow Kube Setup
⭐
25
How to deploy airflow on Kubernetes
Airflow Prometheus Exporter
⭐
25
Export Airflow metrics (from mysql) in prometheus format
Bitnami Docker Airflow Worker
⭐
23
Bitnami Docker Image for Apache Airflow Worker
Simple Dag Editor
⭐
23
Zero configuration Airflow plugin that let you manage your DAG files.
Fancontrol
⭐
23
Arduino Leonardo (ATmega32U4) PWM fan control
Airflow Tutorial
⭐
23
Basic tutorial of using Apache Airflow
Dataops Platform Airflow Dbt
⭐
21
Build DataOps platform with Apache Airflow and dbt on AWS
Airflow Scraping
⭐
21
Using Apache Airflow to schedule web scrapers
Terraform Aws Airflow
⭐
20
Terraform module for a PostgreSQL-backed Apache Airflow instance
Airdatacomputer
⭐
20
Air Data Computer
Aircan
⭐
19
💨🥫 A Data Factory system for running data processing pipelines built on AirFlow and tailored to CKAN. Includes evolution of DataPusher and Xloader for loading data to DataStore.
Dbt Cloud Plugin
⭐
19
DBT Cloud Plugin for Airflow
Bitnami Docker Airflow Scheduler
⭐
19
Bitnami Docker Image for Apache Airflow Scheduler
Insight Gdelt Feed
⭐
19
A way for home buyers to know about factors affecting a state
Airflow Pentaho Plugin
⭐
19
Pentaho plugin for Apache Airflow - Orquestate pentaho transformations and jobs from Airflow
Fab Oidc
⭐
18
Flask-AppBuilder SecurityManager for OpenIDConnect
Ansible Airflow
⭐
18
Ansible role for the installation of Apache Airflow
1-100 of 384 projects
Next >
Related Projects
Python Airflow Projects (258)
Docker Airflow Projects (91)
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.
