Awesome Open Source

Programming Languages

Search results for python airflow

328 search results found

Airflow ⭐ 34,468

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Diagrams ⭐ 33,556

🎨 Diagram as Code for prototyping cloud system architectures

Orchest ⭐ 3,876

Build data pipelines, the easy way 🛠️

Ploomber ⭐ 3,318

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

Elyra ⭐ 1,721

Elyra extends JupyterLab with an AI centric approach.

Airflow Maintenance Dags ⭐ 1,567

A series of DAGs/Workflows to help maintain the operation of Airflow

Dag Factory ⭐ 1,001

Dynamically generate Apache Airflow DAGs from YAML configuration files

Around Dataengineering ⭐ 926

A Data Engineering & Machine Learning Knowledge Hub

Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

Airflow Tutorial ⭐ 669

Apache Airflow tutorial

Energy Forecasting ⭐ 663

🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | 𝗟𝗲𝗮𝗿𝗻 𝗠𝗟𝗘 & 𝗠𝗟𝗢𝗽𝘀 for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 & 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴

Dataengineeringproject ⭐ 644

Example end to end data engineering project.

Goodreads_etl_pipeline ⭐ 593

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Data Pipelines With Apache Airflow ⭐ 587

Code for Data Pipelines with Apache Airflow

Kube Airflow ⭐ 556

A docker image and kubernetes config files to run Airflow on Kubernetes

Astronomer ⭐ 454

Helm Charts for the Astronomer Platform, Apache Airflow as a Service on Kubernetes

Agile_data_code_2 ⭐ 435

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

Ethereum Etl Airflow ⭐ 378

Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethe

Aws Airflow Stack ⭐ 370

Turbine: the bare metals that gets you Airflow

Astronomer Cosmos ⭐ 368

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

Airflow Code Editor ⭐ 347

A plugin for Apache Airflow that allows you to edit DAGs in browser

Gather Deployment ⭐ 347

Gathers Python deployment, infrastructure and practices.

Airflow Tutorial ⭐ 345

Airflow basics tutorial

Astro Sdk ⭐ 303

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Microservices Based Algorithmic Trading System ⭐ 302

MBATS is a docker based platform for developing, testing and deploying Algorthmic Trading strategies with a focus on Machine Learning based algorithms.

Airflow Dbt ⭐ 298

Apache Airflow integration for dbt

Airflow Client Python ⭐ 289

Apache Airflow - OpenApi Client for Python

Beginner_de_project ⭐ 276

Beginner data engineering project - batch edition

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

Airflow Diagrams ⭐ 273

Auto-generated Diagrams from Airflow DAGs. 🔮 🪄

Airflow Chart ⭐ 261

A Helm chart to install Apache Airflow on Kubernetes

Boundary Layer ⭐ 255

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform

Airflow Rest Api Plugin ⭐ 243

A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces

Paperboy ⭐ 231

A web frontend for scheduling Jupyter notebook reports

Airflow Testing ⭐ 227

Airflow Unit Tests and Integration Tests

Airflow Scheduler Failover Controller ⭐ 220

A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability

Example Airflow Dags ⭐ 204

Example DAGs using hooks and operators from Airflow Plugins

Making DAG construction easier

Airflow Exporter ⭐ 190

Airflow plugin to export dag and task based metrics to Prometheus.

Data Testing With Airflow ⭐ 190

Airflow_in_docker_compose ⭐ 182

Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

Cwl Airflow ⭐ 178

Python package to extend Airflow functionality with CWL1.1 support

Airflow_for_beginners ⭐ 166

Telemetry Airflow ⭐ 161

Airflow configuration for Telemetry

Starthinker ⭐ 156

Reference framework for building data workflows provided by Google. Accelerates authentication, logging, scheduling, and deployment of solutions using GCP. To borrow a tagline.. "The framework for professionals with deadlines."

Airflow Provider Great Expectations ⭐ 147

Great Expectations Airflow operator

Airflow Dbt Python ⭐ 139

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Public Datasets Pipelines ⭐ 131

Cloud-native, data onboarding architecture for Google Cloud Datasets

Incubator Liminal ⭐ 131

Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.

Cortex Data Foundation ⭐ 129

Data Foundation - Google Cloud Cortex Framework

Airflow Gcp Examples ⭐ 129

Repository with examples and smoke tests for the GCP Airflow operators and hooks

Airflowctl ⭐ 128

A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

Ask Astro ⭐ 128

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

Tensorflow Recommendation Wals ⭐ 126

An end-to-end solution for website article recommendations based on Google Analytics data. Uses WALS matrix-factorization in TensorFlow, trained on Cloud ML Engine. Recommendations served with App Engine Flex and Cloud Endpoints. Orchestration is performed using Airflow on Cloud Composer. See the solution tutorials at:

Astronomer Providers ⭐ 123

Airflow Providers containing Deferrable Operators & Sensors from Astronomer

Kedro Airflow ⭐ 120

Kedro-Airflow makes it easy to deploy Kedro projects to Airflow.

Airflow Docker ⭐ 120

Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j)

Airflow Dbt Demo ⭐ 120

A repository of sample code to accompany our blog post on Airflow and dbt.

Data Science Stack Cookiecutter ⭐ 120

🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)

Movalytics Data Warehouse ⭐ 116

Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow

Airflow Extended Api Plugin ⭐ 108

Yet another Airflow plugin using CLI command as RESTful api, supports Airflow v2.X.

Airflow Clickhouse Plugin ⭐ 105

The most popular ClickHouse plugin for Airflow. 🔝 Top-1% downloads on PyPI: https://pypi.org/project/airflow-clickhouse-plugin Based on mymarilyn/clickhouse-driver.

Hark Lang ⭐ 104

A serverless virtual machine

afctl helps to manage and deploy Apache Airflow projects faster and smoother.

Airflow Toolkit ⭐ 101

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥️ >> [ 🚀, 🚢 ]

Airflow Prometheus Exporter ⭐ 100

Prometheus Exporter for Airflow

Airflow Repo Template ⭐ 98

The easiest way to run Airflow locally, with linting & tests for valid DAGs and Plugins.

Streamify ⭐ 97

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Incremental_training ⭐ 96

Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and MLFlow'

Polygon Etl ⭐ 93

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Viewflow ⭐ 84

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Final Project Level3 Recsys 01 ⭐ 81

🍁 AI를 활용한 메이플스토리 코디 추천 서비스👚👕

Skills Api ⭐ 81

Oozie To Airflow ⭐ 80

Oozie Workflow to Airflow DAGs migration tool

Get your MLOps (Level 1) platform started and going fast.

Airflow Declarative ⭐ 77

Airflow declarative DAGs via YAML

Airflow_multi_dagrun ⭐ 76

triggering a DAG run multiple times

Airflint ⭐ 72

Enforce Best Practices for all your Airflow DAGs. ⭐

Airflow Ms Teams Operator ⭐ 71

Airflow operator that can send messages to MS Teams

Fileflow ⭐ 70

Airflow plugin to transfer arbitrary files between operators

Discreetly ⭐ 70

ETLy is an add-on dashboard service on top of Apache Airflow.

Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.

Magniv Core ⭐ 67

Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.

Airflow Spark Operator Plugin ⭐ 66

A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator

Aws Concurrent Data Orchestration Pipeline Emr Livy ⭐ 66

This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.

Dbt Airflow Docker Compose ⭐ 64

Execution of DBT models using Apache Airflow through Docker Compose

Airflow Spark ⭐ 64

Docker with Airflow and Spark standalone cluster

Production Tools ⭐ 56

A bare-bones repository demonstrating how to set up tools for data science projects that will help you write higher quality code.

Data Engineering made simple - An opinionated Data Engineering framework

Airflow Gcp K8s ⭐ 56

Airflow Cookbook ⭐ 55

Airflow workflow management platform chef cookbook.

Cccatalog ⭐ 55

[PROJECT TRANSFERRED] Mapping the commons towards an open ledger and cc search.

Kubernetesjoboperator ⭐ 54

An airflow operator that executes a task in a kubernetes cluster, given a kubernetes yaml configuration or an image refrence.

Afancontrol ⭐ 53

Advanced Fan Control program, which controls PWM fans according to the current temperatures of the system components.

User friendly and open source platform for workflow creation and monitoring

Airflow Training ⭐ 51

Airflow training for the crunch conf

Airflow Tutorial ⭐ 51

🐍💨 Airflow tutorial for PyCon 2019

Algotrading_stack ⭐ 48

Microservices Stack for AlgoTrading

Ambari Airflow Mpack ⭐ 45

Ambari stack service for installing and managing Apache Airflow on HDP cluster

Airflow plugins for implementing data pipelines. | Plugins do Airflow para implementação de pipelines de dados.

Related Searches

Python Django (28,897)

Python Machine Learning (20,195)

Python Flask (17,643)

Python Dataset (14,792)

Python Docker (14,113)

Python Tensorflow (13,736)

Python Command Line (13,351)

Python Deep Learning (13,092)

Python Jupyter Notebook (12,976)

Python Testing (9,479)

1-100 of 328 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.