Data Pipeline With Dbt Using Airflow On Gcp
This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. There are different tools that have been used in this project such as Astro, DBT, GCP, Airflow, Metabase.
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Optimus | 707 | 3 | 9 months ago | 120 | July 25, 2023 | 53 | apache-2.0 | Go | ||
| Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management. | ||||||||||
| Ethereum Etl Airflow | 378 | 3 months ago | 27 | mit | Python | |||||
| Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethereum-smart-contract-into-bigquery-in-8-mins-bab5db1fdeee | ||||||||||
| Astro Sdk | 303 |  | 2 | 3 months ago | 49 | August 30, 2023 | 153 | apache-2.0 | Python | |
| Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow. | ||||||||||
| Starthinker | 156 | 9 months ago | 15 | April 07, 2022 | 18 | apache-2.0 | Python | |||
| Reference framework for building data workflows provided by Google. Accelerates authentication, logging, scheduling, and deployment of solutions using GCP. To borrow a tagline.. "The framework for professionals with deadlines." | ||||||||||
| Tensorflow Lifetime Value | 140 | 2 years ago | 9 | apache-2.0 | Jupyter Notebook | |||||
| Predict customer lifetime value using AutoML Tables, or ML Engine with a TensorFlow neural network and the Lifetimes Python library. | ||||||||||
| Public Datasets Pipelines | 131 | 3 months ago | 56 | apache-2.0 | Python | |||||
| Cloud-native, data onboarding architecture for Google Cloud Datasets | ||||||||||
| Cortex Data Foundation | 129 | 3 months ago | 3 | apache-2.0 | Python | |||||
| Data Foundation - Google Cloud Cortex Framework | ||||||||||
| Tensorflow Recommendation Wals | 126 | 4 years ago | 8 | apache-2.0 | Python | |||||
| An end-to-end solution for website article recommendations based on Google Analytics data. Uses WALS matrix-factorization in TensorFlow, trained on Cloud ML Engine. Recommendations served with App Engine Flex and Cloud Endpoints. Orchestration is performed using Airflow on Cloud Composer. See the solution tutorials at: | ||||||||||
| Polygon Etl | 93 | 5 months ago | 39 | November 21, 2023 | 10 | mit | Python | |||
| ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub | ||||||||||
| Airflow Tutorial | 86 | 4 years ago | 1 | apache-2.0 | PLpgSQL | |||||
| Use Airflow to move data from multiple MySQL databases to BigQuery | ||||||||||
Alternatives To Data Pipeline With Dbt Using Airflow On GcpSelect To Compare
Alternative Project Comparisons
Popular Bigquery Projects
Popular Airflow Projects
Popular Data Processing Categories
Related Searches
Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Database
Mysql
Sql
Data Science
Etl
Bigquery
Data Engineering
Airflow