Awesome Open Source

Programming Languages

Search results for airflow

512 search results found

Ap Airflow ⭐ 100

Astronomer Core Docker Images

Airflow Prometheus Exporter ⭐ 100

Prometheus Exporter for Airflow

Airflow Repo Template ⭐ 98

The easiest way to run Airflow locally, with linting & tests for valid DAGs and Plugins.

Streamify ⭐ 97

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Incremental_training ⭐ 96

Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and MLFlow'

Deployml_course ⭐ 93

Репозиторий для открытого курса «Промышленная эксплуатация моделей машинного обучения»

Polygon Etl ⭐ 93

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Airflow Autoscaling Ecs ⭐ 87

Airflow Deployment on AWS ECS Fargate Using Cloudformation

Airflow Tutorial ⭐ 86

Use Airflow to move data from multiple MySQL databases to BigQuery

Viewflow ⭐ 84

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Final Project Level3 Recsys 01 ⭐ 81

🍁 AI를 활용한 메이플스토리 코디 추천 서비스👚👕

Skills Api ⭐ 81

Udacity Data Eng Proj 1 ⭐ 81

Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3

Oozie To Airflow ⭐ 80

Oozie Workflow to Airflow DAGs migration tool

Get your MLOps (Level 1) platform started and going fast.

Objinsync ⭐ 77

Continuously synchronize directories from remote object store to local filesystem

Airflow Declarative ⭐ 77

Airflow declarative DAGs via YAML

Airflow_multi_dagrun ⭐ 76

triggering a DAG run multiple times

Data Engineering Nanodegree ⭐ 76

Projects done in the Data Engineering Nanodegree by Udacity.com

Terraform Aws Airflow ⭐ 72

Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

Airflint ⭐ 72

Enforce Best Practices for all your Airflow DAGs. ⭐

Airflow Role ⭐ 72

Ansible role to install Apache Airflow

Airflow Ms Teams Operator ⭐ 71

Airflow operator that can send messages to MS Teams

Discreetly ⭐ 70

ETLy is an add-on dashboard service on top of Apache Airflow.

Fileflow ⭐ 70

Airflow plugin to transfer arbitrary files between operators

Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.

Getting Started ⭐ 69

A Getting Started Guide for developing and using Airflow Plugins

A lightweight tool to get an AI Infrastructure Stack up in minutes not days. K3ai will take care of setup K8s for You, deploy the AI tool of your choice and even run your code on it.

Magniv Core ⭐ 67

Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.

Aws Concurrent Data Orchestration Pipeline Emr Livy ⭐ 66

This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.

Airflow Spark Operator Plugin ⭐ 66

A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator

Airflow Spark ⭐ 64

Docker with Airflow and Spark standalone cluster

Dbt Airflow Docker Compose ⭐ 64

Execution of DBT models using Apache Airflow through Docker Compose

Diy Beatmungsgeraet ⭐ 60

CPAP Beatmungsgerät selbst gebaut - basierend auf einem Open Source und Open Hardware Entwurf

Production Tools ⭐ 56

A bare-bones repository demonstrating how to set up tools for data science projects that will help you write higher quality code.

Airflow Gcp K8s ⭐ 56

Data Engineering made simple - An opinionated Data Engineering framework

Cccatalog ⭐ 55

[PROJECT TRANSFERRED] Mapping the commons towards an open ledger and cc search.

Airflow Cookbook ⭐ 55

Airflow workflow management platform chef cookbook.

Kubernetesjoboperator ⭐ 54

An airflow operator that executes a task in a kubernetes cluster, given a kubernetes yaml configuration or an image refrence.

Afancontrol ⭐ 53

Advanced Fan Control program, which controls PWM fans according to the current temperatures of the system components.

Cloud Bigdata Book ⭐ 53

A distributed workflow runner focusing on performance and simplicity.

User friendly and open source platform for workflow creation and monitoring

Airflow Training ⭐ 51

Airflow training for the crunch conf

Airflow Tutorial ⭐ 51

🐍💨 Airflow tutorial for PyCon 2019

Dsgopipeline ⭐ 50

A short workshop on datascience pipelines using mlflow and airflow

Algotrading_stack ⭐ 48

Microservices Stack for AlgoTrading

Airflow Backfill Util ⭐ 47

Airflow Backfill UI based plugin for existing / new Airflow environment

Terraform Aws Ecs Airflow ⭐ 46

A terraform module that creates an airflow instance in AWS ECS.

Ambari Airflow Mpack ⭐ 45

Ambari stack service for installing and managing Apache Airflow on HDP cluster

Airflow plugins for implementing data pipelines. | Plugins do Airflow para implementação de pipelines de dados.

Works With Determined ⭐ 43

This repository contains example integrations between Determined and other ML products

Airflow Plugin Demo ⭐ 43

Example for an airflow plugin

Airflow Console ⭐ 42

Airflow Dag可视化编辑和管理

Udacity Data Engineering ⭐ 42

Udacity Data Engineering Nano Degree (DEND)

Terraform Google Composer ⭐ 42

Manages Cloud Composer v1 and v2 along with option to manage networking

Soopervisor ⭐ 42

☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.

Pyconde2019 Airflow Ml Workshop ⭐ 42

PyConDE & PyData Berlin 2019 Airflow Workshop: Airflow for machine learning pipelines.

Ansible Role Airflow ⭐ 42

Ansible role to deploy and configure Airflow

Waterwheel ⭐ 41

A workflow scheduler based on petri-nets

A Python package for building helm charts and Kubernetes templates in an object oriented fashion

Airflow_api_plugin ⭐ 41

REST-like API exposing Airflow data and operations

Airflow Boilerplate ⭐ 40

A complete development environment setup for working with Airflow

Ml In Production ⭐ 39

The practical use-cases of how to make your Machine Learning Pipelines robust and reliable using Apache Airflow.

Alphasql ⭐ 39

AlphaSQL provides Integrated Type and Schema Check and Parallelization for SQL file set mainly for BigQuery

Airflow Dag Dependencies ⭐ 39

Visualize dependencies between Airflow DAGs

Airflow Aws Executors ⭐ 37

Airflow Executor for both AWS ECS & AWS Fargate

Orchestra ⭐ 36

Advertising Data Lakes and Workflow Automation

PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows

Blockchain Etl Architecture ⭐ 36

Blockchain ETL Architecture

Airflow Kube Setup ⭐ 35

How to deploy airflow on Kubernetes

Cloud Data Lake ⭐ 34

Data lake, data warehouse on GCP

End-to-end DataOps platform deployed by Terraform.

Pylint Airflow ⭐ 34

Pylint plugin for static code analysis on Airflow code

Airflow Gke K8sexecutor Helm ⭐ 33

Quickly get a kubernetes executor airflow environment provisioned on GKE. Azure Kubernetes Service instructions included also as are instructions for docker-for-mac.

Ecs Airflow ⭐ 32

Cloudformation templates for deploying Airflow in ECS

Airflow Pentaho Plugin ⭐ 32

Pentaho plugin for Apache Airflow - Orquestate pentaho transformations and jobs from Airflow

基于Python开源技术生态的企业级自动化测试集成解决方案

Aws Airflow Demo ⭐ 31

Project files for the post: Running PySpark Applications on Amazon EMR using Apache Airflow: Using the new Amazon Managed Workflows for Apache Airflow (MWAA) on AWS.

Stellar Etl Airflow ⭐ 31

Airflow DAGs for the Stellar ETL project

Open drug data pipelines curated by pharmacists.

Evo Odas ⭐ 30

Code Repository for the EVO-ODAS

Debussy_concert ⭐ 29

Debussy is an opinionated Data Architecture and Engineering framework, enabling data analysts and engineers to build better platforms and pipelines.

Fairflow ⭐ 29

Functional Airflow DAG definitions.

Airflow Tutorial ⭐ 29

My self-learning about Apache Airflow

Kedro Airflow K8s ⭐ 29

Kedro Plugin to support running pipelines on Kubernetes using Airflow.

ADA is a microservice created to retrieve analytics metrics from an Airflow database instance.

Docker Airflow ⭐ 28

Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

Scalable Airflow Template ⭐ 28

fast and scalable Airflow on Kubernetes Setup.

Dea Airflow ⭐ 28

Airflow Provider Datarobot ⭐ 28

DataRobot provider for Apache Airflow

Yet Another Workflow Engine: a subprocess-based DAG execution system

Data Engineer Nanodegree Projects Udacity ⭐ 27

Projects done in the Data Engineer Nanodegree Program by Udacity.com

Data Engineering Nanodegree ⭐ 27

Solution to all projects of Udacity's Data Engineering Nanodegree: Data Modeling with Postgres & Cassandra, Data Warehouse with Redshift, Data Lake with Spark and Data Pipeline with Airflow.

Opencase ⭐ 26

Modular Open Source PC case designed for Mini-ITX motherboards.

Me7tuner ⭐ 26

ME7Tuner is software that provides tools to help calibrate the MAF, primary fueling and torque/load requests

Spreadsheets To Dataframes ⭐ 26

Pycon 2021 Tutorial to help Spreadsheet (Excel) Users learn Python

Python_mozetl ⭐ 26

ETL jobs for Firefox Telemetry

Airflow Testing Ci Workflow ⭐ 26

(project & tutorial) dag pipeline tests + ci/cd setup

101-200 of 512 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.