Awesome Open Source

Programming Languages

Search results for dag airflow

321 search results found

Airflow ⭐ 31,852

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Argo Workflows ⭐ 13,481

Workflow Engine for Kubernetes

Orchest ⭐ 3,876

Build data pipelines, the easy way 🛠️

Awesome Etl ⭐ 2,874

A curated list of awesome ETL frameworks, libraries, and software.

Airflow Maintenance Dags ⭐ 1,484

A series of DAGs/Workflows to help maintain the operation of Airflow

Dag Factory ⭐ 968

Dynamically generate Apache Airflow DAGs from YAML configuration files

Goodreads_etl_pipeline ⭐ 593

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Kube Airflow ⭐ 556

A docker image and kubernetes config files to run Airflow on Kubernetes

Aws Airflow Stack ⭐ 370

Turbine: the bare metals that gets you Airflow

Ethereum Etl Airflow ⭐ 369

Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethe

Airflow Tutorial ⭐ 345

Airflow basics tutorial

Airflow Code Editor ⭐ 323

A plugin for Apache Airflow that allows you to edit DAGs in browser

Astro Sdk ⭐ 285

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Astro Cli ⭐ 277

CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer

Microservices Based Algorithmic Trading System ⭐ 267

MBATS is a docker based platform for developing, testing and deploying Algorthmic Trading strategies with a focus on Machine Learning based algorithms.

Airflow Diagrams ⭐ 263

Auto-generated Diagrams from Airflow DAGs. 🔮 🪄

Boundary Layer ⭐ 255

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform

Airflow Testing ⭐ 227

Airflow Unit Tests and Integration Tests

Example Airflow Dags ⭐ 204

Example DAGs using hooks and operators from Airflow Plugins

Airflow Exporter ⭐ 190

Airflow plugin to export dag and task based metrics to Prometheus.

Data Testing With Airflow ⭐ 188

Airflow Dag Creation Manager Plugin ⭐ 188

A plugin for Airflow that create and manage your DAG with web UI.

Fast iterative local development and testing of Apache Airflow workflows

Cwl Airflow ⭐ 161

Python package to extend Airflow functionality with CWL1.1 support

Telemetry Airflow ⭐ 159

Airflow configuration for Telemetry

Airflow Gcp Examples ⭐ 129

Repository with examples and smoke tests for the GCP Airflow operators and hooks

Airflow Dbt Demo ⭐ 120

A repository of sample code to accompany our blog post on Airflow and dbt.

Airflow Kube Helm ⭐ 120

Kedro Airflow ⭐ 120

Kedro-Airflow makes it easy to deploy Kedro projects to Airflow.

Airflow On Kubernetes ⭐ 119

A guide to running Airflow on Kubernetes

Airflowctl ⭐ 114

A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

Aws Ecs Airflow ⭐ 110

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

Sagemaker Ml Workflow With Apache Airflow ⭐ 104

This repository shows a sample example to build, manage and orchestrate Machine Learning workflows using Amazon Sagemaker and Apache Airflow.

afctl helps to manage and deploy Apache Airflow projects faster and smoother.

Airflow Toolkit ⭐ 101

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥 >> [ 🚀, 🚢 ]

Airflow Prometheus Exporter ⭐ 100

Prometheus Exporter for Airflow

Airflow Repo Template ⭐ 98

The easiest way to run Airflow locally, with linting & tests for valid DAGs and Plugins.

Airflow Autoscaling Ecs ⭐ 87

Airflow Deployment on AWS ECS Fargate Using Cloudformation

Airflow Tutorial ⭐ 86

Use Airflow to move data from multiple MySQL databases to BigQuery

Oozie To Airflow ⭐ 80

Oozie Workflow to Airflow DAGs migration tool

Airflow Declarative ⭐ 77

Airflow declarative DAGs via YAML

Airflow_multi_dagrun ⭐ 76

triggering a DAG run multiple times

Discreetly ⭐ 70

ETLy is an add-on dashboard service on top of Apache Airflow.

Getting Started ⭐ 69

A Getting Started Guide for developing and using Airflow Plugins

Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.

Airflint ⭐ 68

Enforce Best Practices for all your Airflow DAGs. ⭐

Aws Concurrent Data Orchestration Pipeline Emr Livy ⭐ 66

This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.

Dbt Airflow Docker Compose ⭐ 64

Execution of DBT models using Apache Airflow through Docker Compose

Airflow Spark ⭐ 64

Docker with Airflow and Spark standalone cluster

Airflow Gcp K8s ⭐ 56

Production Tools ⭐ 56

A bare-bones repository demonstrating how to set up tools for data science projects that will help you write higher quality code.

A distributed workflow runner focusing on performance and simplicity.

Airflow Tutorial ⭐ 51

🐍💨 Airflow tutorial for PyCon 2019

Airflow Backfill Util ⭐ 47

Airflow Backfill UI based plugin for existing / new Airflow environment

Terraform Aws Ecs Airflow ⭐ 46

A terraform module that creates an airflow instance in AWS ECS.

Airflow Plugin Demo ⭐ 43

Example for an airflow plugin

Airflow_api_plugin ⭐ 41

REST-like API exposing Airflow data and operations

Airflow Dag Dependencies ⭐ 39

Visualize dependencies between Airflow DAGs

Alphasql ⭐ 39

AlphaSQL provides Integrated Type and Schema Check and Parallelization for SQL file set mainly for BigQuery

Blockchain Etl Architecture ⭐ 36

Blockchain ETL Architecture

Pylint Airflow ⭐ 34

Pylint plugin for static code analysis on Airflow code

Airflow Gke K8sexecutor Helm ⭐ 33

Quickly get a kubernetes executor airflow environment provisioned on GKE. Azure Kubernetes Service instructions included also as are instructions for docker-for-mac.

Fairflow ⭐ 29

Functional Airflow DAG definitions.

Stellar Etl Airflow ⭐ 29

Airflow DAGs for the Stellar ETL project

Dea Airflow ⭐ 28

Docker Airflow ⭐ 28

Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

Yet Another Workflow Engine: a subprocess-based DAG execution system

Spot Price Reporter ⭐ 24

Fetch and plot AWS spot pricing history

💨🥫 A Data Factory system for running data processing pipelines built on AirFlow and tailored to CKAN. Includes evolution of DataPusher and Xloader for loading data to DataStore.

Data Pipeline ⭐ 23

Heroku Airflow ⭐ 21

Demo of Author data workflows with Airflow on Heroku (not maintained)

Terraform Aws Airflow ⭐ 20

Terraform module for a PostgreSQL-backed Apache Airflow instance

Airflow Chart ⭐ 20

Helm chart for deploying Apache Airflow in kubernetes

Sf Evictiontracker ⭐ 20

Tracking and measuring neighborhood and district-level eviction rates in the city of San Francisco.

Superset Trading Dashboard ⭐ 19

Superset Trading Dashboard

Airflow Livy Operators ⭐ 17

Lets Airflow DAGs run Spark jobs via Livy: sessions and/or batches.

Dbimport ⭐ 16

DBImport ingestion tool. Handle import, export and standard ETL flows in Hadoop/Hive

Phila Airflow ⭐ 16

Airflow Valohai Plugin ⭐ 15

🦈 Airflow plugin to scale machine learning tasks with Valohai and get automatic version control

Db Airflow Dags ⭐ 15

Collection of DAGS for DCP database development and generation

Airflowdatapipeline ⭐ 15

Example of an ETL Pipeline using Airflow

Opentrials Airflow ⭐ 15

Configuration and definitions of Airflow for OpenTrials

Airflow Docker ⭐ 14

An opinionated implementation of exclusively using airflow DockerOperators for all Operators

Airflow Metrics ⭐ 14

Metrics for airflow

Airflow Imaging Plugins ⭐ 14

Set of plugins helping to work with imaging data in Airflow.

Airflow Helm ⭐ 14

A Helm Chart for Apache Airflow

Etl Airflow S3 ⭐ 13

ETL of newspaper article keywords using Apache Airflow, Newspaper3k, Quilt T4 and AWS S3

Airflow Ml Prediction ⭐ 13

Running ECS task for ML prediction orchestrated by Airflow

Ml Airflow ⭐ 13

Generalized project for running Airflow DAGs, with possibility of skipping tasks already done for some set of input parameters.

Apache Airflow Cloudera Csd ⭐ 12

CSD for Apache Airflow

Astro Provider Databricks ⭐ 12

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows

Rb_status_plugin ⭐ 11

rb_status_plugin : Data confidence tool for Airflow

Airflow Cdk ⭐ 11

Constructs to deploy airflow via the aws cdk

Geo Openstreetmap ⭐ 11

Theeye Supervisor ⭐ 11

TheEye supervisor

Airflowjob ⭐ 11

Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE

Windmill ⭐ 10

Drag N Drop WepApp to Build and Manage Airflow DAGs

Cicd Cloud Composer ⭐ 10

Example script to deploy DAGs to Google Cloud Composer.

Quantifying interactions with government services to support delivery teams to improve their own products and services

Data Rivers ⭐ 9

Apache Airflow and Beam ETL scripts for the City of Pittsburgh's data analysis pipelines

Related Searches

Python3 Airflow (678)

Python Airflow (633)

Python Dag (455)

1-100 of 321 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2023 Awesome Open Source. All rights reserved.