Airflow Operator
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Argo Workflows | 13,468 | 24 | 45 | 6 hours ago | 439 | July 20, 2023 | 897 | apache-2.0 | Go | |
| Workflow Engine for Kubernetes | ||||||||||
| Pipeline | 4,159 | a year ago | 85 | July 18, 2017 | 1 | apache-2.0 | Jsonnet | |||
| PipelineAI Kubeflow Distribution | ||||||||||
| Orchest | 3,876 | 4 months ago | 19 | December 13, 2022 | 125 | apache-2.0 | TypeScript | |||
| Build data pipelines, the easy way 🛠️ | ||||||||||
| Ploomber | 3,164 |  | 6 | a month ago | 113 | August 03, 2023 | 97 | apache-2.0 | Python | |
| The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️ | ||||||||||
| Awesome Apache Airflow | 3,060 | a year ago | 2 | Shell | ||||||
| Curated list of resources about Apache Airflow | ||||||||||
| Couler | 834 | 2 months ago | 33 | apache-2.0 | Python | |||||
| Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow. | ||||||||||
| Charts | 556 | 8 days ago | 75 | apache-2.0 | Shell | |||||
| The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes. | ||||||||||
| Kube Airflow | 556 | 4 years ago | 30 | apache-2.0 | Python | |||||
| A docker image and kubernetes config files to run Airflow on Kubernetes | ||||||||||
| Astronomer | 449 | 6 days ago | 21 | other | Python | |||||
| Helm Charts for the Astronomer Platform, Apache Airflow as a Service on Kubernetes | ||||||||||
| Astro Cli | 277 | a day ago | 279 | December 22, 2022 | 103 | other | Go | |||
| CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer | ||||||||||
This is not an officially supported Google product.
Community
- Join our Slack channel.
Project Status
Alpha
The Airflow Operator is still under active development and has not been extensively tested in production environment. Backward compatibility of the APIs is not guaranteed for alpha releases.
Prerequisites
- Version >= 1.9 of Kubernetes.
- Uses 1.9 of Airflow (1.10.1+ for k8s executor)
- Uses 4.0.x of Redis (for celery operator)
- Uses 5.7 of MySQL
Get Started
One Click Deployment from Google Cloud Marketplace to your GKE cluster
Get started quickly with the Airflow Operator using the Quick Start Guide
For more information check the Design and detailed User Guide
Airflow Operator Overview
Airflow Operator is a custom Kubernetes operator that makes it easy to deploy and manage Apache Airflow on Kubernetes. Apache Airflow is a platform to programmatically author, schedule and monitor workflows. Using the Airflow Operator, an Airflow cluster is split into 2 parts represented by the AirflowBase and AirflowCluster custom resources.
The Airflow Operator performs these jobs:
- Creates and manages the necessary Kubernetes resources for an Airflow deployment.
- Updates the corresponding Kubernetes resources when the
AirflowBaseorAirflowClusterspecification changes. - Restores managed Kubernetes resources that are deleted.
- Supports creation of Airflow schedulers with different Executors
- Supports sharing of the
AirflowBaseacross mulitpleAirflowClusters
Checkout out the Design
Development
Refer to the Design and Development Guide.
Managed Airflow solution
Google Cloud Composer is a fully managed workflow orchestration service targeting customers that need a workflow manager in the cloud.
