Awesome Open Source

Programming Languages

Search results for data engineering airflow

data-engineering x

56 search results found

Airflow ⭐ 34,299

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Argo Workflows ⭐ 14,227

Workflow Engine for Kubernetes

Ploomber ⭐ 3,318

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

Data Engineering Howto ⭐ 2,949

A list of useful resources to learn Data Engineering from scratch

Udacity Data Engineering Projects ⭐ 1,335

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Around Dataengineering ⭐ 926

A Data Engineering & Machine Learning Knowledge Hub

Dataengineeringproject ⭐ 644

Example end to end data engineering project.

Goodreads_etl_pipeline ⭐ 593

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Data Engineering Interview Questions ⭐ 554

More than 2000+ Data engineer interview questions.

Ethereum Etl Airflow ⭐ 378

Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethe

Data Engineering Projects ⭐ 322

Personal Data Engineering Projects

Dataplane ⭐ 171

Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

Airflow Dbt Python ⭐ 139

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Public Datasets Pipelines ⭐ 131

Cloud-native, data onboarding architecture for Google Cloud Datasets

Movalytics Data Warehouse ⭐ 114

Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow

Streamify ⭐ 97

A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Polygon Etl ⭐ 93

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Airflow Autoscaling Ecs ⭐ 87

Airflow Deployment on AWS ECS Fargate Using Cloudformation

Viewflow ⭐ 84

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Data Engineering Nanodegree ⭐ 76

Projects done in the Data Engineering Nanodegree by Udacity.com

Magniv Core ⭐ 67

Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.

Data Engineering made simple - An opinionated Data Engineering framework

Ml In Production ⭐ 39

The practical use-cases of how to make your Machine Learning Pipelines robust and reliable using Apache Airflow.

PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows

Airflow Pentaho Plugin ⭐ 32

Pentaho plugin for Apache Airflow - Orquestate pentaho transformations and jobs from Airflow

Debussy_concert ⭐ 29

Debussy is an opinionated Data Architecture and Engineering framework, enabling data analysts and engineers to build better platforms and pipelines.

Data Engineering Nanodegree ⭐ 27

Solution to all projects of Udacity's Data Engineering Nanodegree: Data Modeling with Postgres & Cassandra, Data Warehouse with Redshift, Data Lake with Spark and Data Pipeline with Airflow.

Audiophile E2e Pipeline ⭐ 24

Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard.

Jobanalytics_and_search ⭐ 22

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Ndexr Platform ⭐ 20

The NDEXR platform code

Spotify Api ⭐ 19

Pipeline that extracts data from the Spotify API to build a more detailed version of Spotify Wrapped

Airflow Docker ⭐ 19

This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.

Airflowetl ⭐ 16

Blog post on ETL pipelines with Airflow

Airflow Valohai Plugin ⭐ 15

🦈 Airflow plugin to scale machine learning tasks with Valohai and get automatic version control

Airflowdatapipeline ⭐ 15

Example of an ETL Pipeline using Airflow

Dbt Airflow ⭐ 14

A Python package that creates fine-grained dbt tasks on Apache Airflow

Data Pipeline from the Global Historical Climatology Network DataSet

Bootcamp_data Engineering ⭐ 13

Bootcamp to learn basics in Data Engineering

Thepipelinetool ⭐ 13

A pipeline orchestration tool

Airflow Rbac Roles Cli ⭐ 12

A tool to create Airflow RBAC roles with dag-level permissions from cli.

Spotify Etl ⭐ 12

Spotify ETL Pipeline

Airflowjob ⭐ 11

Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE

Data Pipeline With Dbt Using Airflow On Gcp ⭐ 10

This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. There are different tools that have been used in this project such as Astro, DBT, GCP, Airflow, Metabase.

Airflow Docker Metrics ⭐ 10

Awesome Dataops ⭐ 10

Awesome list of dataops products, open source and resources

A project for exploring how Great Expectations can be used to ensure data quality and validate batches within a data pipeline defined in Airflow.

Apache Airflow Providers Transfers ⭐ 10

De Devtools ⭐ 8

Data Engineering Development Tools Installation and Configuration

Devops Mlops ⭐ 8

Tools for DevOps and MLOps. Materials and projects. New technologies and infrastructure review.

This set of code and instructions has the porpouse to instanciate a compiled environment with set of docker images like airflow webserver, airflow scheduler, postgresql, pyspark, Data Pipeline consuming data from weather api , processing with pyspark and storing in postgresql

Reddit Data Engineering ⭐ 7

An end-to-end data engineering pipeline to create a dashboard for the latest content on the r/Stocks subreddit

Data Engineer Portfolio ⭐ 6

This is a repository to demonstrate my details, skills, projects and to keep track of my progression in Data Analytics and Data Science topics.

Dockerized_data_science_playground ⭐ 6

Multi-docker container data science / engineering playground (w/ Kafka, Airflow, MLFlow, Tensorflow-Keras / SKLearn) for simulating a microservices-oriented architecture

Udacity Data Engineering Nanodegree ⭐ 5

This is a repository to hold the files and notebooks produced throughout my Udacity's Nanodegree Data Engineering program.

Airflow Terraform ⭐ 5

Easily deploy airflow infrastructure on an AWS VPC using terraform.

Gcp Airflow Foundations ⭐ 5

Opinionated framework based on Airflow 2.0 for building pipelines to ingest data into a BigQuery data warehouse

Aiscalator ⭐ 5

Tools to streamline Jupyter Notebook Prototypes into robust Data Products

Data_infra_repo ⭐ 5

Collections of POC/dev data infrastructure. | #SE

Analytics_data_where_house ⭐ 5

An analytics engineering sandbox focusing on real estates prices in Cook County, IL

Providence ⭐ 5

Apply Data Engineering to Personal Finance

Related Searches

Python Airflow (658)

1-56 of 56 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.