Awesome Open Source

Programming Languages

Search results for python airflow

441 search results found

Airflow ⭐ 31,852

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Diagrams ⭐ 31,095

🎨 Diagram as Code for prototyping cloud system architectures

Amundsen ⭐ 4,061

Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.

Orchest ⭐ 3,876

Build data pipelines, the easy way 🛠️

Ploomber ⭐ 3,164

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

Mara Pipelines ⭐ 2,019

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Elyra ⭐ 1,672

Elyra extends JupyterLab with an AI centric approach.

Airflow Maintenance Dags ⭐ 1,484

A series of DAGs/Workflows to help maintain the operation of Airflow

Dag Factory ⭐ 968

Dynamically generate Apache Airflow DAGs from YAML configuration files

Around Dataengineering ⭐ 926

A Data Engineering & Machine Learning Knowledge Hub

Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

Airflow Tutorial ⭐ 669

Apache Airflow tutorial

Dataengineeringproject ⭐ 644

Example end to end data engineering project.

Goodreads_etl_pipeline ⭐ 593

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Kube Airflow ⭐ 556

A docker image and kubernetes config files to run Airflow on Kubernetes

Data Pipelines With Apache Airflow ⭐ 503

Code for Data Pipelines with Apache Airflow

Energy Forecasting ⭐ 490

🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | Learn MLE & MLOps for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 & 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴

Astronomer ⭐ 450

Helm Charts for the Astronomer Platform, Apache Airflow as a Service on Kubernetes

Agile_data_code_2 ⭐ 435

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

Aws Airflow Stack ⭐ 370

Turbine: the bare metals that gets you Airflow

Ethereum Etl Airflow ⭐ 369

Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethe

Gather Deployment ⭐ 347

Gathers Python deployment, infrastructure and practices.

Airflow Tutorial ⭐ 345

Airflow basics tutorial

Airflow Code Editor ⭐ 323

A plugin for Apache Airflow that allows you to edit DAGs in browser

Airflow Dbt ⭐ 298

Apache Airflow integration for dbt

Astro Sdk ⭐ 285

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Beginner_de_project ⭐ 276

Beginner data engineering project - batch edition

Microservices Based Algorithmic Trading System ⭐ 267

MBATS is a docker based platform for developing, testing and deploying Algorthmic Trading strategies with a focus on Machine Learning based algorithms.

Airflow Diagrams ⭐ 263

Auto-generated Diagrams from Airflow DAGs. 🔮 🪄

Boundary Layer ⭐ 255

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform

Airflow Chart ⭐ 255

A Helm chart to install Apache Airflow on Kubernetes

Airflow Client Python ⭐ 253

Apache Airflow - OpenApi Client for Python

Astronomer Cosmos ⭐ 252

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

Airflow Rest Api Plugin ⭐ 243

A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

Paperboy ⭐ 231

A web frontend for scheduling Jupyter notebook reports

Airflow Testing ⭐ 227

Airflow Unit Tests and Integration Tests

Airflow Scheduler Failover Controller ⭐ 220

A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability

Example Airflow Dags ⭐ 204

Example DAGs using hooks and operators from Airflow Plugins

Airflow Exporter ⭐ 190

Airflow plugin to export dag and task based metrics to Prometheus.

Data Testing With Airflow ⭐ 188

Making DAG construction easier

Airflow_in_docker_compose ⭐ 167

Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

Airflow_for_beginners ⭐ 166

Cwl Airflow ⭐ 161

Python package to extend Airflow functionality with CWL1.1 support

Telemetry Airflow ⭐ 159

Airflow configuration for Telemetry

Starthinker ⭐ 156

Reference framework for building data workflows provided by Google. Accelerates authentication, logging, scheduling, and deployment of solutions using GCP. To borrow a tagline.. "The framework for professionals with deadlines."

Airflow Provider Great Expectations ⭐ 139

Great Expectations Airflow operator

Incubator Liminal ⭐ 131

Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.

Airflow Gcp Examples ⭐ 129

Repository with examples and smoke tests for the GCP Airflow operators and hooks

Tensorflow Recommendation Wals ⭐ 126

An end-to-end solution for website article recommendations based on Google Analytics data. Uses WALS matrix-factorization in TensorFlow, trained on Cloud ML Engine. Recommendations served with App Engine Flex and Cloud Endpoints. Orchestration is performed using Airflow on Cloud Composer. See the solution tutorials at:

Public Datasets Pipelines ⭐ 124

Cloud-native, data onboarding architecture for Google Cloud Datasets

Astronomer Providers ⭐ 122

Airflow Providers containing Deferrable Operators & Sensors from Astronomer

Airflow Dbt Demo ⭐ 120

A repository of sample code to accompany our blog post on Airflow and dbt.

Kedro Airflow ⭐ 120

Kedro-Airflow makes it easy to deploy Kedro projects to Airflow.

Data Science Stack Cookiecutter ⭐ 120

🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)

Airflow Docker ⭐ 120

Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j)

Airflow Dbt Python ⭐ 114

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Airflowctl ⭐ 114

A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

Cortex Data Foundation ⭐ 110

Data Foundation - Google Cloud Cortex Framework

Airflow Extended Api Plugin ⭐ 108

Yet another Airflow plugin using CLI command as RESTful api, supports Airflow v2.X.

Hark Lang ⭐ 104

A serverless virtual machine

afctl helps to manage and deploy Apache Airflow projects faster and smoother.

Airflow Toolkit ⭐ 101

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥 >> [ 🚀, 🚢 ]

Airflow Prometheus Exporter ⭐ 100

Prometheus Exporter for Airflow

Airflow Repo Template ⭐ 98

The easiest way to run Airflow locally, with linting & tests for valid DAGs and Plugins.

Streamify ⭐ 97

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Incremental_training ⭐ 96

Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and MLFlow'

Airflow Clickhouse Plugin ⭐ 95

The most popular ClickHouse plugin for Airflow. 🔝 Top-1% downloads on PyPI: https://pypi.org/project/airflow-clickhouse-plugin Based on mymarilyn/clickhouse-driver.

Movalytics Data Warehouse ⭐ 93

Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow

Polygon Etl ⭐ 90

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Viewflow ⭐ 84

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Skills Api ⭐ 81

Final Project Level3 Recsys 01 ⭐ 81

🍁 AI를 활용한 메이플스토리 코디 추천 서비스👚👕

Oozie To Airflow ⭐ 80

Oozie Workflow to Airflow DAGs migration tool

Get your MLOps (Level 1) platform started and going fast.

Airflow Declarative ⭐ 77

Airflow declarative DAGs via YAML

Airflow_multi_dagrun ⭐ 76

triggering a DAG run multiple times

Discreetly ⭐ 70

ETLy is an add-on dashboard service on top of Apache Airflow.

Fileflow ⭐ 70

Airflow plugin to transfer arbitrary files between operators

Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.

Airflint ⭐ 68

Enforce Best Practices for all your Airflow DAGs. ⭐

Airflow Ms Teams Operator ⭐ 68

Airflow operator that can send messages to MS Teams

Magniv Core ⭐ 67

Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.

Airflow Spark Operator Plugin ⭐ 66

A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator

Aws Concurrent Data Orchestration Pipeline Emr Livy ⭐ 66

This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.

Airflow Spark ⭐ 64

Docker with Airflow and Spark standalone cluster

Dbt Airflow Docker Compose ⭐ 64

Execution of DBT models using Apache Airflow through Docker Compose

Production Tools ⭐ 56

A bare-bones repository demonstrating how to set up tools for data science projects that will help you write higher quality code.

Airflow Gcp K8s ⭐ 56

Data Engineering made simple - An opinionated Data Engineering framework

Cccatalog ⭐ 55

[PROJECT TRANSFERRED] Mapping the commons towards an open ledger and cc search.

Airflow Cookbook ⭐ 55

Airflow workflow management platform chef cookbook.

Kubernetesjoboperator ⭐ 54

An airflow operator that executes a task in a kubernetes cluster, given a kubernetes yaml configuration or an image refrence.

Airflow Training ⭐ 51

Airflow training for the crunch conf

Airflow Tutorial ⭐ 51

🐍💨 Airflow tutorial for PyCon 2019

Ambari Airflow Mpack ⭐ 45

Ambari stack service for installing and managing Apache Airflow on HDP cluster

Airflow plugins for implementing data pipelines. | Plugins do Airflow para implementação de pipelines de dados.

Airflow Plugin Demo ⭐ 43

Example for an airflow plugin

Works With Determined ⭐ 43

This repository contains example integrations between Determined and other ML products

Related Searches

Python Python3 (857,414)

Python Django (28,897)

Python Ml (20,195)

Python Pytorch (17,959)

Python Flask (17,122)

Python Dataset (14,792)

Python Docker (14,113)

Python Machine Learning (14,099)

Python Tensorflow (13,736)

Python Command Line (13,351)

1-100 of 441 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2023 Awesome Open Source. All rights reserved.