Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Python Etl Open Source Projects

Open source projects categorized as Python Etl
Airflow ⭐ 34,299
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169most recent commit 16 days agopypi apache-airflow} Downloads
Airbyte ⭐ 12,918
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
dependent packages 11total releases 311most recent commit 3 months agopypi airbyte-cdk} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585most recent commit 3 months agopypi dagster} Downloads
Mage Ai ⭐ 6,324
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
total releases 314most recent commit 3 months ago
Orchest ⭐ 3,876
Build data pipelines, the easy way 🛠️
total releases 19most recent commit a year ago
Aws Sdk Pandas ⭐ 3,779
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
dependent packages 65total releases 143most recent commit a month agopypi awswrangler} Downloads
Ethereum Etl ⭐ 2,760
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
dependent packages 2total releases 44most recent commit 3 months agopypi ethereum-etl} Downloads
Quadratic ⭐ 2,485
Quadratic | Data Science Spreadsheet with Python & SQL
most recent commit 3 months ago
Mara Pipelines ⭐ 2,053
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
dependent packages 1total releases 8most recent commit 4 months agopypi mara-pipelines} Downloads
Deepie ⭐ 1,737
DeepIE: Deep Learning for Information Extraction
most recent commit a year ago
Riko ⭐ 1,573
A Python stream processing engine modeled after Yahoo! Pipes
total releases 59most recent commit 2 years ago
Vdp ⭐ 1,556
💧 Instill VDP (Versatile Data Pipeline) is an open-source tool to seamlessly integrate AI to process unstructured data in the modern data stack
dependent packages 1total releases 42most recent commit 3 months ago
Aws Glue Samples ⭐ 1,334
AWS Glue code samples
most recent commit 6 months ago
Hamilton ⭐ 1,272
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
dependent packages 2total releases 116most recent commit a month agopypi sf-hamilton} Downloads
Getting Started ⭐ 1,098
This repository is a getting started guide to Singer.
most recent commit a year ago
Pyspark Example Project ⭐ 1,034
Example project implementing best practices for PySpark ETL jobs and applications.
most recent commit a year ago
Pgsync ⭐ 1,003
Postgres to Elasticsearch/OpenSearch sync
total releases 54most recent commit 4 months ago
Sqlmesh ⭐ 931
SQLMesh is a data transformation framework that brings the benefits of DevOps to data teams. It enables data scientists, analysts, and engineers to efficiently run and deploy data transformations written in SQL or Python.
dependent packages 1total releases 254most recent commit 3 months agopypi sqlmesh} Downloads
Hamilton ⭐ 877
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
total releases 10most recent commit 10 months ago
Open Semantic Search ⭐ 741
Open Source research tool to search, browse, analyze and explore large document collections by Semantic Search Engine and Open Source Text Mining & Text Analytics platform (Integrates ETL for document processing, OCR for images & PDF, named entity recognition for persons, organizations & locations, metadata management by thesaurus & ontologies, search user interface & search apps for fulltext search, faceted search & knowledge graph)
most recent commit a year ago
Neumai ⭐ 693
Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.
most recent commit 3 months ago
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 4 years ago
Eland ⭐ 588
Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
dependent packages 3total releases 30most recent commit 3 months agopypi eland} Downloads
The 10 Most Depended On Python Etl Open Source Projects
Airflow ⭐ 34,299
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169latest release November 27, 2023most recent commit 16 days agopypi apache-airflow} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181latest release November 28, 2023most recent commit 3 months agopypi vdk-core} Downloads
Diffsync ⭐ 121
A utility library for comparing and synchronizing different datasets.
dependent packages 13total releases 15latest release November 16, 2023most recent commit 3 months agopypi diffsync} Downloads
Aliyun Log Python Sdk ⭐ 159
Use python to manage, produce and consume data with Aliyun Log Service.
dependent packages 12total releases 122latest release November 15, 2023most recent commit 4 months agopypi aliyun-log-python-sdk} Downloads
Dbt Databricks ⭐ 165
A dbt adapter for Databricks.
dependent packages 6total releases 91latest release November 30, 2023most recent commit 3 months agopypi dbt-databricks} Downloads
Pangeo Forge Recipes ⭐ 108
Python library for building Pangeo Forge recipes.
dependent packages 4total releases 22latest release November 15, 2023most recent commit 5 months agopypi pangeo-forge-recipes} Downloads
Target Postgres ⭐ 93
A Singer.io Target for Postgres
dependent packages 3total releases 17latest release April 18, 2020most recent commit a year agopypi singer-target-postgres} Downloads
The 10 Latest Releases In Python Etl Open Source Projects
Ayeaye ⭐ 5
An ETL (Extract, Transform, Load) framework.
total releases 51latest release December 12, 2023most recent commit 3 months ago
Locopy ⭐ 99
locopy: Loading/Unloading to Redshift and Snowflake using Python.
total releases 21latest release December 08, 2023most recent commit 3 months ago
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Datachecks ⭐ 117
Open Source Data Quality Monitoring.
total releases 14latest release December 07, 2023most recent commit 3 months ago
Mara Pipelines ⭐ 2,053
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
dependent packages 1total releases 8latest release December 06, 2023most recent commit 4 months agopypi mara-pipelines} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.