Awesome Open Source
Awesome Open Source
Combined Topics
airflow
x
Advertising
📦 10
All Projects
Application Programming Interfaces
📦 124
Applications
📦 192
Artificial Intelligence
📦 78
Blockchain
📦 73
Build Tools
📦 113
Cloud Computing
📦 80
Code Quality
📦 28
Collaboration
📦 32
Command Line Interface
📦 49
Community
📦 83
Companies
📦 60
Compilers
📦 63
Computer Science
📦 80
Configuration Management
📦 42
Content Management
📦 175
Control Flow
📦 213
Data Formats
📦 78
Data Processing
📦 276
Data Storage
📦 135
Economics
📦 64
Frameworks
📦 215
Games
📦 129
Graphics
📦 110
Hardware
📦 152
Integrated Development Environments
📦 49
Learning Resources
📦 166
Legal
📦 29
Libraries
📦 129
Lists Of Projects
📦 22
Machine Learning
📦 347
Mapping
📦 64
Marketing
📦 15
Mathematics
📦 55
Media
📦 239
Messaging
📦 98
Networking
📦 315
Operating Systems
📦 89
Operations
📦 121
Package Managers
📦 55
Programming Languages
📦 245
Runtime Environments
📦 100
Science
📦 42
Security
📦 396
Social Media
📦 27
Software Architecture
📦 72
Software Development
📦 72
Software Performance
📦 58
Software Quality
📦 133
Text Editors
📦 49
Text Processing
📦 136
User Interface
📦 330
User Interface Components
📦 514
Version Control
📦 30
Virtualization
📦 71
Web Browsers
📦 42
Web Servers
📦 26
Web User Interface
📦 210
The Top 59 Airflow Open Source Projects
Categories
>
Control Flow
>
Airflow
Airflow
⭐
21,782
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Argo Workflows
⭐
8,671
Workflow engine for Kubernetes
Dolphinscheduler
⭐
5,932
Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available `out of the box`.
Pipeline
⭐
4,113
PipelineAI Kubeflow Distribution
Docker Airflow
⭐
3,224
Docker Apache Airflow
Awesome Apache Airflow
⭐
2,478
Curated list of resources about Apache Airflow
Dataspherestudio
⭐
1,364
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Airflow Maintenance Dags
⭐
976
A series of DAGs/Workflows to help maintain the operation of Airflow
Elyra
⭐
945
Elyra extends JupyterLab Notebooks with an AI centric approach.
Goodreads_etl_pipeline
⭐
826
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Udacity Data Engineering Projects
⭐
531
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
Around Dataengineering
⭐
463
A Data Engineering & Machine Learning Knowledge Hub
Dag Factory
⭐
452
Dynamically generate Apache Airflow DAGs from YAML configuration files
Agile_data_code_2
⭐
415
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
Aws Airflow Stack
⭐
361
Turbine: the bare metals that gets you Airflow
Soda Sql
⭐
334
Data testing and monitoring for SQL accessible data
Airflow Tutorial
⭐
324
Airflow basics tutorial
Ploomber
⭐
298
Lean Data Science workflows: develop and test locally. Deploy to Kubernetes, AWS Batch, or Airflow.
Airflow Rest Api Plugin
⭐
289
A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces
Airflow Operator
⭐
283
Kubernetes custom controller and CRDs to managing Airflow
Example Airflow Dags
⭐
261
Example DAGs using hooks and operators from Airflow Plugins
Paperboy
⭐
225
A web frontend for scheduling Jupyter notebook reports
Airflow Scheduler Failover Controller
⭐
204
A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability
Airflow Testing
⭐
189
Airflow Unit Tests and Integration Tests
Airflow Doc Zh
⭐
177
📖 [译] Airflow 中文文档
Charts
⭐
174
the community-maintained descendant of the `stable/airflow` helm chart
Airflow Exporter
⭐
166
Airflow plugin to export dag and task based metrics to Prometheus.
Airflow Chart
⭐
163
A Helm chart to install Apache Airflow on Kubernetes
Data Science Stack Cookiecutter
⭐
159
🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)
Airflow Autoscaling Ecs
⭐
144
Airflow Deployment on AWS ECS Fargate Using Cloudformation
Dataengineeringproject
⭐
144
Example end to end data engineering project.
Beyond Jupyter
⭐
137
🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)
Airflow Pipeline
⭐
136
An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR
Telemetry Airflow
⭐
128
Airflow configuration for Telemetry
Whirl
⭐
124
Fast iterative local development and testing of Apache Airflow workflows
Airflow Dbt
⭐
123
Apache Airflow integration for dbt
Airflow_in_docker_compose
⭐
119
Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)
Airflow Code Editor
⭐
119
A plugin for Apache Airflow that allows you to edit DAGs in browser
Afctl
⭐
118
afctl helps to manage and deploy Apache Airflow projects faster and smoother.
Aws Ecs Airflow
⭐
114
Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
Bitnami Docker Airflow
⭐
101
Bitnami Docker Image for Apache Airflow
Udacity Data Engineering
⭐
97
Udacity Data Engineering Nano Degree (DEND)
Beginner_de_project
⭐
92
Beginner data engineering project - batch edition
Airflow Training
⭐
87
Airflow training for the crunch conf
Gusty
⭐
77
Making DAG construction easier
Terraform Aws Airflow
⭐
74
Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor
Discreetly
⭐
62
ETLy is an add-on dashboard service on top of Apache Airflow.
Data Pipelines With Apache Airflow
⭐
62
Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3
Airflow Cookbook
⭐
58
Airflow workflow management platform chef cookbook.
Airflow_multi_dagrun
⭐
58
triggering a DAG run multiple times
Xene
⭐
57
A distributed workflow runner focusing on performance and simplicity.
Airflow Toolkit
⭐
57
Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥 >> [ 🚀, 🚢 ]
Objinsync
⭐
46
Continuously synchronize directories from remote object store to local filesystem
Airflow On Kubernetes
⭐
40
Bare minimal Airflow on Kubernetes (Local, EKS, AKS)
Docker Airflow
⭐
30
Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/
Databook
⭐
26
A facebook for data
Automating Your Data Pipeline With Apache Airflow
⭐
24
Automating Your Data Pipeline with Apache Airflow
Docker Compose Airflow
⭐
20
Docker Compose Apache Airflow (Official Docker Images) with CeleryExecutor, InitDB and InitUser
Phila Airflow
⭐
16
1-59 of 59 projects
Advertising
📦 10
All Projects
Application Programming Interfaces
📦 124
Applications
📦 192
Artificial Intelligence
📦 78
Blockchain
📦 73
Build Tools
📦 113
Cloud Computing
📦 80
Code Quality
📦 28
Collaboration
📦 32
Command Line Interface
📦 49
Community
📦 83
Companies
📦 60
Compilers
📦 63
Computer Science
📦 80
Configuration Management
📦 42
Content Management
📦 175
Control Flow
📦 213
Data Formats
📦 78
Data Processing
📦 276
Data Storage
📦 135
Economics
📦 64
Frameworks
📦 215
Games
📦 129
Graphics
📦 110
Hardware
📦 152
Integrated Development Environments
📦 49
Learning Resources
📦 166
Legal
📦 29
Libraries
📦 129
Lists Of Projects
📦 22
Machine Learning
📦 347
Mapping
📦 64
Marketing
📦 15
Mathematics
📦 55
Media
📦 239
Messaging
📦 98
Networking
📦 315
Operating Systems
📦 89
Operations
📦 121
Package Managers
📦 55
Programming Languages
📦 245
Runtime Environments
📦 100
Science
📦 42
Security
📦 396
Social Media
📦 27
Software Architecture
📦 72
Software Development
📦 72
Software Performance
📦 58
Software Quality
📦 133
Text Editors
📦 49
Text Processing
📦 136
User Interface
📦 330
User Interface Components
📦 514
Version Control
📦 30
Virtualization
📦 71
Web Browsers
📦 42
Web Servers
📦 26
Web User Interface
📦 210
