Awesome Open Source

Programming Languages

Search results for python dag

278 search results found

Airflow ⭐ 31,852

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Orchest ⭐ 3,876

Build data pipelines, the easy way 🛠️

DDSP: Differentiable Digital Signal Processing

Pgmpy ⭐ 2,468

Python Library for learning (Structure and Parameter), inference (Probabilistic and Causal), and simulations in Bayesian Networks.

Airflow Maintenance Dags ⭐ 1,484

A series of DAGs/Workflows to help maintain the operation of Airflow

Dag Factory ⭐ 968

Dynamically generate Apache Airflow DAGs from YAML configuration files

Hamilton ⭐ 950

Your single tool to express data, ML, and LLM pipelines with simple python functions. Runs anywhere that python runs, E.G. spark, airflow, jupyter, fastapi, etc. Incrementally adoptable. Use Hamilton to build testable, reusable, and self-documenting dataflows with lineage and metadata out of the box.

Hamilton ⭐ 877

A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

Serving ⭐ 834

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

Goodreads_etl_pipeline ⭐ 593

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Kube Airflow ⭐ 556

A docker image and kubernetes config files to run Airflow on Kubernetes

Rustworkx ⭐ 548

A high performance Python graph library implemented in Rust.

Jieba_fast ⭐ 512

Use C Api and Swig to Speed up jieba 高效的中文分词库

Notears ⭐ 442

DAGs with NO TEARS: Continuous Optimization for Structure Learning

Bnlearn ⭐ 375

Python library for learning the graphical structure of Bayesian networks, parameter learning, inference and sampling methods.

Aws Airflow Stack ⭐ 370

Turbine: the bare metals that gets you Airflow

Ethereum Etl Airflow ⭐ 369

Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethe

Airflow Tutorial ⭐ 345

Airflow basics tutorial

Airflow Code Editor ⭐ 323

A plugin for Apache Airflow that allows you to edit DAGs in browser

Astro Sdk ⭐ 285

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Microservices Based Algorithmic Trading System ⭐ 267

MBATS is a docker based platform for developing, testing and deploying Algorthmic Trading strategies with a focus on Machine Learning based algorithms.

Airflow Diagrams ⭐ 263

Auto-generated Diagrams from Airflow DAGs. 🔮 🪄

Boundary Layer ⭐ 255

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform

Airflow Testing ⭐ 227

Airflow Unit Tests and Integration Tests

The easiest way to use Machine Learning. Mix and match underlying ML libraries and data set sources. Generate new datasets or modify existing ones with ease.

Example Airflow Dags ⭐ 204

Example DAGs using hooks and operators from Airflow Plugins

Airflow Exporter ⭐ 190

Airflow plugin to export dag and task based metrics to Prometheus.

Data Testing With Airflow ⭐ 188

Py Bitcoin ⭐ 187

比特币的python简易实现，主要介绍比特币底层技术。如Base58编码、椭圆加密算法、Merkle

Dag Gnn ⭐ 165

Cwl Airflow ⭐ 161

Python package to extend Airflow functionality with CWL1.1 support

Telemetry Airflow ⭐ 159

Airflow configuration for Telemetry

Airflow Gcp Examples ⭐ 129

Repository with examples and smoke tests for the GCP Airflow operators and hooks

Airflow Dbt Demo ⭐ 120

A repository of sample code to accompany our blog post on Airflow and dbt.

Kedro Airflow ⭐ 120

Kedro-Airflow makes it easy to deploy Kedro projects to Airflow.

Microtokenizer ⭐ 119

一个微型&算法全面的中文分词引擎 | A micro tokenizer for Chinese

Airflowctl ⭐ 114

A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

Bigflow ⭐ 109

A Python framework for data processing on GCP.

afctl helps to manage and deploy Apache Airflow projects faster and smoother.

Airflow Toolkit ⭐ 101

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥 >> [ 🚀, 🚢 ]

Airflow Prometheus Exporter ⭐ 100

Prometheus Exporter for Airflow

ForML - A development framework and MLOps platform for the lifecycle management of data science projects

Airflow Repo Template ⭐ 98

The easiest way to run Airflow locally, with linting & tests for valid DAGs and Plugins.

Causaldag ⭐ 95

Python package for the creation, manipulation, and learning of Causal DAGs

Celery Dyrygent ⭐ 85

Celery extension which allows to orchestrate 100/1000/10000 tasks combined into a complex workflow

Oozie To Airflow ⭐ 80

Oozie Workflow to Airflow DAGs migration tool

Algorithms ⭐ 77

Algorithms in python and C

Airflow Declarative ⭐ 77

Airflow declarative DAGs via YAML

Airflow_multi_dagrun ⭐ 76

triggering a DAG run multiple times

Nary Grn ⭐ 76

Code regarding to our paper "N-ary Relation Extraction using Graph State LSTM"

PyTorch code for ICPR 2020 paper "DAG-Net: Double Attentive Graph Neural Network for Trajectory Forecasting"

Hathor Core ⭐ 71

HathorNetwork's fullnode core

Discreetly ⭐ 70

ETLy is an add-on dashboard service on top of Apache Airflow.

Visualize Airflow's schedule by exporting future DAG runs as events to Google Calendar.

Airflint ⭐ 68

Enforce Best Practices for all your Airflow DAGs. ⭐

AutoDiff DAG constructor, built on numpy and Cython. A Neural Turing Machine and DeepQ agent run on it. Clean code for educational purpose.

Aws Concurrent Data Orchestration Pipeline Emr Livy ⭐ 66

This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.

D-VAE: A Variational Autoencoder for Directed Acyclic Graphs, NeurIPS 2019

Airflow Spark ⭐ 64

Docker with Airflow and Spark standalone cluster

Dbt Airflow Docker Compose ⭐ 64

Execution of DBT models using Apache Airflow through Docker Compose

Breaking_cycles_in_noisy_hierarchies ⭐ 58

breaking cycles in noisy hierarchies

Mlinspect ⭐ 57

Inspect ML Pipelines in Python in the form of a DAG

Production Tools ⭐ 56

A bare-bones repository demonstrating how to set up tools for data science projects that will help you write higher quality code.

Airflow Gcp K8s ⭐ 56

Django Dag ⭐ 55

Basic portable Directed Acyclic Graph application for Django

Airflow Tutorial ⭐ 51

🐍💨 Airflow tutorial for PyCon 2019

Causing: CAUsal INterpretation using Graphs

Afl Mutation Graph ⭐ 46

Recover mutation graph from an AFL seed

Graph Optimiser for Learning and Evolution of Models

Airflow Plugin Demo ⭐ 43

Example for an airflow plugin

A DAG Scheduler library written in pure python

Airflow_api_plugin ⭐ 41

REST-like API exposing Airflow data and operations

Loginputengine ⭐ 41

落格输入法中文拼音整句输入引擎

Tensorflow Tracer ⭐ 39

Runtime Tracing Library for TensorFlow

Spark With Python My Learning Notes ⭐ 39

ETL pipeline using pyspark (Spark - Python)

Pylint Airflow ⭐ 34

Pylint plugin for static code analysis on Airflow code

Primrose ⭐ 33

Primrose modeling framework for simple production models

(deprecated) A fast and memory-efficient Python data engineering framework for machine learning.

Graph Llvm Ir ⭐ 30

Visualization of LLVM IR

Polyaxon Quick Start ⭐ 29

A quick start project for polyaxon

Fairflow ⭐ 29

Functional Airflow DAG definitions.

Apache Mesos backend for Dask scheduling library

Stellar Etl Airflow ⭐ 29

Airflow DAGs for the Stellar ETL project

Dea Airflow ⭐ 28

Docker Airflow ⭐ 28

Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

Function dependencies resolution and execution

Yet Another Workflow Engine: a subprocess-based DAG execution system

Triggerflow ⭐ 26

Event-based Orchestration of Serverless Workflows

💨🥫 A Data Factory system for running data processing pipelines built on AirFlow and tailored to CKAN. Includes evolution of DataPusher and Xloader for loading data to DataStore.

Spot Price Reporter ⭐ 24

Fetch and plot AWS spot pricing history

Django Postgresql Dag ⭐ 23

Directed Acyclic Graphs with a variety of methods for both Nodes and Edges, and multiple exports (NetworkX, Pandas, etc). This project is the foundation for a commercial product, so expect regular improvements. PR's and other contributions are welcomed.

Data Pipeline ⭐ 23

An efficient implementation of the PHANTOM (GhostDAG) block-DAG protocol, complete with a block-DAG network simulation framework and other niceties.

Oliver Twist ⭐ 22

dbt DAG Auditor

Heroku Airflow ⭐ 21

Demo of Author data workflows with Airflow on Heroku (not maintained)

Jupiter is an orchestrator for Dispersed (Networked) Computing that uses Docker containers and Kubernetes.

Sf Evictiontracker ⭐ 20

Tracking and measuring neighborhood and district-level eviction rates in the city of San Francisco.

Superset Trading Dashboard ⭐ 19

Superset Trading Dashboard

Databazel ⭐ 18

Bazel build rules for machine learning workflows

Related Searches

Python Python3 (857,414)

Python Ml (20,195)

Python Pytorch (17,959)

Python Jupyter (17,496)

Python Dataset (14,792)

Python Docker (14,113)

Python Machine Learning (14,099)

Python Tensorflow (13,736)

Python Deep Learning (13,092)

Python Jupyter Notebook (12,976)

1-100 of 278 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2023 Awesome Open Source. All rights reserved.