Awesome Open Source

Programming Languages

Search results for airflow

512 search results found

Airflow ⭐ 34,468

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Diagrams ⭐ 33,556

🎨 Diagram as Code for prototyping cloud system architectures

Argo Workflows ⭐ 14,264

Workflow Engine for Kubernetes

Dolphinscheduler ⭐ 11,613

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Pipeline ⭐ 4,158

Orchest ⭐ 3,876

Build data pipelines, the easy way 🛠️

Docker Airflow ⭐ 3,589

Docker Apache Airflow

Awesome Apache Airflow ⭐ 3,468

Curated list of resources about Apache Airflow

Ploomber ⭐ 3,318

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

Awesome Etl ⭐ 3,041

A curated list of awesome ETL frameworks, libraries, and software.

Data Engineering Howto ⭐ 2,949

A list of useful resources to learn Data Engineering from scratch

Dataspherestudio ⭐ 2,860

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Elyra ⭐ 1,721

Elyra extends JupyterLab with an AI centric approach.

Airflow Maintenance Dags ⭐ 1,567

A series of DAGs/Workflows to help maintain the operation of Airflow

Udacity Data Engineering Projects ⭐ 1,335

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Etl With Airflow ⭐ 1,053

ETL best practices with airflow, with examples

Dag Factory ⭐ 1,001

Dynamically generate Apache Airflow DAGs from YAML configuration files

Around Dataengineering ⭐ 926

A Data Engineering & Machine Learning Knowledge Hub

Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

Optimus ⭐ 707

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

Airflow Tutorial ⭐ 669

Apache Airflow tutorial

Energy Forecasting ⭐ 663

🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | 𝗟𝗲𝗮𝗿𝗻 𝗠𝗟𝗘 & 𝗠𝗟𝗢𝗽𝘀 for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 & 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴

Dataengineeringproject ⭐ 644

Example end to end data engineering project.

Goodreads_etl_pipeline ⭐ 593

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Data Pipelines With Apache Airflow ⭐ 587

Code for Data Pipelines with Apache Airflow

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.

Kube Airflow ⭐ 556

A docker image and kubernetes config files to run Airflow on Kubernetes

Data Engineering Interview Questions ⭐ 554

More than 2000+ Data engineer interview questions.

Astronomer ⭐ 454

Helm Charts for the Astronomer Platform, Apache Airflow as a Service on Kubernetes

Agile_data_code_2 ⭐ 435

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

Ethereum Etl Airflow ⭐ 378

Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethe

Aws Airflow Stack ⭐ 370

Turbine: the bare metals that gets you Airflow

Astronomer Cosmos ⭐ 368

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

Airflow Code Editor ⭐ 347

A plugin for Apache Airflow that allows you to edit DAGs in browser

Gather Deployment ⭐ 347

Gathers Python deployment, infrastructure and practices.

Airflow Tutorial ⭐ 345

Airflow basics tutorial

Data Engineering Projects ⭐ 322

Personal Data Engineering Projects

Astro Sdk ⭐ 303

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Microservices Based Algorithmic Trading System ⭐ 302

MBATS is a docker based platform for developing, testing and deploying Algorthmic Trading strategies with a focus on Machine Learning based algorithms.

Airflow Dbt ⭐ 298

Apache Airflow integration for dbt

Airflow Client Python ⭐ 289

Apache Airflow - OpenApi Client for Python

Compass ⭐ 284

Compass is a task diagnosis platform for bigdata

Beginner_de_project ⭐ 276

Beginner data engineering project - batch edition

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

Airflow Diagrams ⭐ 273

Auto-generated Diagrams from Airflow DAGs. 🔮 🪄

Airflow Chart ⭐ 261

A Helm chart to install Apache Airflow on Kubernetes

Boundary Layer ⭐ 255

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform

Airflow Rest Api Plugin ⭐ 243

A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces

Airflow Operator ⭐ 241

Kubernetes custom controller and CRDs to managing Airflow

Simple but powerful DAG scheduler and dashboard

Paperboy ⭐ 231

A web frontend for scheduling Jupyter notebook reports

Airflow Testing ⭐ 227

Airflow Unit Tests and Integration Tests

Airflow Scheduler Failover Controller ⭐ 220

A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability

Kyle School ⭐ 204

쏘카 데이터 그룹 사내 신입/인턴을 대상으로 한 카일스쿨

Example Airflow Dags ⭐ 204

Example DAGs using hooks and operators from Airflow Plugins

Airflow Doc Zh ⭐ 204

📖 [译] Airflow 中文文档

Making DAG construction easier

Data Testing With Airflow ⭐ 190

Airflow Exporter ⭐ 190

Airflow plugin to export dag and task based metrics to Prometheus.

Airflow Dag Creation Manager Plugin ⭐ 188

A plugin for Airflow that create and manage your DAG with web UI.

Fast iterative local development and testing of Apache Airflow workflows

Airflow_in_docker_compose ⭐ 182

Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

Cwl Airflow ⭐ 178

Python package to extend Airflow functionality with CWL1.1 support

Dataplane ⭐ 171

Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

Airflow Pipeline ⭐ 168

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

Airflow_for_beginners ⭐ 166

Telemetry Airflow ⭐ 161

Airflow configuration for Telemetry

Rill Flow ⭐ 160

Rill Flow is a high-performance, scalable distributed workflow orchestration service

Skills Ml ⭐ 159

Data Processing and Machine learning methods for the Open Skills Project

Starthinker ⭐ 156

Reference framework for building data workflows provided by Google. Accelerates authentication, logging, scheduling, and deployment of solutions using GCP. To borrow a tagline.. "The framework for professionals with deadlines."

Airflow Provider Great Expectations ⭐ 147

Great Expectations Airflow operator

🌻 FlorFlow: Flor, now with Dataflow

Tensorflow Lifetime Value ⭐ 140

Predict customer lifetime value using AutoML Tables, or ML Engine with a TensorFlow neural network and the Lifetimes Python library.

Airflow Dbt Python ⭐ 139

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Public Datasets Pipelines ⭐ 131

Cloud-native, data onboarding architecture for Google Cloud Datasets

Incubator Liminal ⭐ 131

Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.

Cortex Data Foundation ⭐ 129

Data Foundation - Google Cloud Cortex Framework

Airflow Gcp Examples ⭐ 129

Repository with examples and smoke tests for the GCP Airflow operators and hooks

Airflowctl ⭐ 128

A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

Ask Astro ⭐ 128

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

Tensorflow Recommendation Wals ⭐ 126

An end-to-end solution for website article recommendations based on Google Analytics data. Uses WALS matrix-factorization in TensorFlow, trained on Cloud ML Engine. Recommendations served with App Engine Flex and Cloud Endpoints. Orchestration is performed using Airflow on Cloud Composer. See the solution tutorials at:

Beyond Jupyter ⭐ 125

🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)

Astronomer Providers ⭐ 123

Airflow Providers containing Deferrable Operators & Sensors from Astronomer

A list of technologies similar to inner Yandex technologies

Data Science Stack Cookiecutter ⭐ 120

🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)

Airflow Kube Helm ⭐ 120

Kedro Airflow ⭐ 120

Kedro-Airflow makes it easy to deploy Kedro projects to Airflow.

Airflow Dbt Demo ⭐ 120

A repository of sample code to accompany our blog post on Airflow and dbt.

Airflow Docker ⭐ 120

Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j)

Airflow Site ⭐ 119

Apache Airflow Website

Airflow On Kubernetes ⭐ 119

A guide to running Airflow on Kubernetes

Movalytics Data Warehouse ⭐ 116

Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow

Aws Ecs Airflow ⭐ 110

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

Airflow Extended Api Plugin ⭐ 108

Yet another Airflow plugin using CLI command as RESTful api, supports Airflow v2.X.

Airflow Clickhouse Plugin ⭐ 105

The most popular ClickHouse plugin for Airflow. 🔝 Top-1% downloads on PyPI: https://pypi.org/project/airflow-clickhouse-plugin Based on mymarilyn/clickhouse-driver.

Hark Lang ⭐ 104

A serverless virtual machine

Sagemaker Ml Workflow With Apache Airflow ⭐ 104

This repository shows a sample example to build, manage and orchestrate Machine Learning workflows using Amazon Sagemaker and Apache Airflow.

Airflow Client Go ⭐ 104

Apache Airflow - OpenApi Client for Go

afctl helps to manage and deploy Apache Airflow projects faster and smoother.

Airflow Toolkit ⭐ 101

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥️ >> [ 🚀, 🚢 ]

1-100 of 512 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.