Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Python Etl Data Engineering Open Source Projects

Open source projects categorized as Python Etl Data Engineering
Categories > Data Processing > Data Engineering
Airflow ⭐ 34,299
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169most recent commit 15 days agopypi apache-airflow} Downloads
Airbyte ⭐ 12,918
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
dependent packages 11total releases 311most recent commit 3 months agopypi airbyte-cdk} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585most recent commit 3 months agopypi dagster} Downloads
Mage Ai ⭐ 6,324
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
total releases 314most recent commit 3 months ago
Aws Sdk Pandas ⭐ 3,779
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
dependent packages 65total releases 143most recent commit a month agopypi awswrangler} Downloads
Quadratic ⭐ 2,485
Quadratic | Data Science Spreadsheet with Python & SQL
most recent commit 3 months ago
Hamilton ⭐ 1,272
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
dependent packages 2total releases 116most recent commit a month agopypi sf-hamilton} Downloads
Pyspark Example Project ⭐ 1,034
Example project implementing best practices for PySpark ETL jobs and applications.
most recent commit a year ago
Sqlmesh ⭐ 931
SQLMesh is a data transformation framework that brings the benefits of DevOps to data teams. It enables data scientists, analysts, and engineers to efficiently run and deploy data transformations written in SQL or Python.
dependent packages 1total releases 254most recent commit 3 months agopypi sqlmesh} Downloads
Hamilton ⭐ 877
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
total releases 10most recent commit 10 months ago
Neumai ⭐ 693
Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.
most recent commit 3 months ago
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 4 years ago
Redun ⭐ 464
Yet another redundant workflow engine
dependent packages 1total releases 18most recent commit 5 months agopypi redun} Downloads
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181most recent commit 3 months agopypi vdk-core} Downloads
Aws Serverless Data Lake Framework ⭐ 379
Enterprise-grade, production-hardened, serverless data lake on AWS
most recent commit 3 months ago
Ethereum Etl Airflow ⭐ 378
Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethe
most recent commit 3 months ago
Bitcoin Etl ⭐ 350
ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ
dependent packages 1total releases 2most recent commit 8 months agopypi graphsense-bitcoin-etl} Downloads
Recap ⭐ 292
Work with your web service, database, and streaming schemas in a single format.
total releases 29most recent commit 3 months ago
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35most recent commit 5 months agopypi butterfree} Downloads
Morph Kgc ⭐ 151
Powerful RDF Knowledge Graph Generation with RML Mappings
dependent packages 2total releases 35most recent commit 5 days agopypi morph-kgc} Downloads
Datachecks ⭐ 117
Open Source Data Quality Monitoring.
total releases 14most recent commit 3 months ago
Sayn ⭐ 117
Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).
total releases 32most recent commit 3 months ago
Pangeo Forge Recipes ⭐ 108
Python library for building Pangeo Forge recipes.
dependent packages 4total releases 22most recent commit 5 months agopypi pangeo-forge-recipes} Downloads
The 10 Most Depended On Python Etl Data Engineering Open Source Projects
Airflow ⭐ 34,299
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169latest release November 27, 2023most recent commit 15 days agopypi apache-airflow} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181latest release November 28, 2023most recent commit 3 months agopypi vdk-core} Downloads
Pangeo Forge Recipes ⭐ 108
Python library for building Pangeo Forge recipes.
dependent packages 4total releases 22latest release November 15, 2023most recent commit 5 months agopypi pangeo-forge-recipes} Downloads
Morph Kgc ⭐ 151
Powerful RDF Knowledge Graph Generation with RML Mappings
dependent packages 2total releases 35latest release November 21, 2023most recent commit 5 days agopypi morph-kgc} Downloads
Redun ⭐ 464
Yet another redundant workflow engine
dependent packages 1total releases 18latest release November 12, 2023most recent commit 5 months agopypi redun} Downloads
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35latest release November 14, 2023most recent commit 5 months agopypi butterfree} Downloads
The 10 Latest Releases In Python Etl Data Engineering Open Source Projects
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Datachecks ⭐ 117
Open Source Data Quality Monitoring.
total releases 14latest release December 07, 2023most recent commit 3 months ago
Drivers ⭐ 53
Low-code Python library enabling access to APIs, tools, data sources in seconds.
total releases 403latest release November 29, 2023most recent commit 3 months ago
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181latest release November 28, 2023most recent commit 3 months agopypi vdk-core} Downloads
Airflow ⭐ 34,299
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169latest release November 27, 2023most recent commit 15 days agopypi apache-airflow} Downloads
Yaetos ⭐ 32
Write data & AI pipelines in (SQL, Spark, Pandas) and deploy to the cloud, simplified
total releases 32latest release November 27, 2023most recent commit 3 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.