Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Etl Data Engineering Open Source Projects

Open source projects categorized as Etl Data Engineering
Categories > Data Processing > Data Engineering
Airflow ⭐ 34,468
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169most recent commit 15 days agopypi apache-airflow} Downloads
Airbyte ⭐ 12,918
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
dependent packages 11total releases 311most recent commit 3 months agopypi airbyte-cdk} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585most recent commit 3 months agopypi dagster} Downloads
Benthos ⭐ 7,564
Fancy stream processing made operationally mundane
dependent packages 12total releases 51most recent commit 18 days ago
Mage Ai ⭐ 6,324
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
total releases 314most recent commit 3 months ago
Cloudquery ⭐ 5,380
The open source high performance data integration platform built for developers.
dependent packages 7total releases 345most recent commit 3 months ago
Kestra ⭐ 5,257
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
dependent packages 4total releases 58most recent commit 3 months ago
Aws Sdk Pandas ⭐ 3,779
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
dependent packages 65total releases 143most recent commit a month agopypi awswrangler} Downloads
Quadratic ⭐ 2,485
Quadratic | Data Science Spreadsheet with Python & SQL
most recent commit 3 months ago
Incubator Devlake ⭐ 2,322
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
total releases 207most recent commit 3 months ago
Hamilton ⭐ 1,272
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
dependent packages 2total releases 116most recent commit a month agopypi sf-hamilton} Downloads
Pyspark Example Project ⭐ 1,034
Example project implementing best practices for PySpark ETL jobs and applications.
most recent commit a year ago
Data Engineering Wiki ⭐ 934
The best place to learn data engineering. Built and maintained by the data engineering community.
most recent commit 4 months ago
Sqlmesh ⭐ 931
SQLMesh is a data transformation framework that brings the benefits of DevOps to data teams. It enables data scientists, analysts, and engineers to efficiently run and deploy data transformations written in SQL or Python.
dependent packages 1total releases 254most recent commit 3 months agopypi sqlmesh} Downloads
Hamilton ⭐ 877
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
total releases 10most recent commit 10 months ago
Zingg ⭐ 828
Scalable identity resolution, entity resolution, data mastering and deduplication using ML
total releases 1most recent commit 4 months ago
Dataform ⭐ 757
Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
dependent packages 15total releases 222most recent commit 4 months agonpm @dataform/core} Downloads
Neumai ⭐ 693
Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.
most recent commit 4 months ago
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 4 years ago
Redun ⭐ 464
Yet another redundant workflow engine
dependent packages 1total releases 18most recent commit 6 months agopypi redun} Downloads
Automate Dv ⭐ 456
A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
most recent commit a month ago
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181most recent commit 3 months agopypi vdk-core} Downloads
Aws Serverless Data Lake Framework ⭐ 379
Enterprise-grade, production-hardened, serverless data lake on AWS
most recent commit 3 months ago
The 10 Most Depended On Etl Data Engineering Open Source Projects
Airflow ⭐ 34,468
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169latest release November 27, 2023most recent commit 15 days agopypi apache-airflow} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181latest release November 28, 2023most recent commit 3 months agopypi vdk-core} Downloads
Dataform ⭐ 757
Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
dependent packages 15total releases 222latest release December 07, 2023most recent commit 4 months agonpm @dataform/core} Downloads
Etl ⭐ 327
PHP - ETL (Extract Transform Load) data processing library
dependent packages 15total releases 16latest release November 30, 2023most recent commit 4 months agopackagist flow-php/etl} Downloads
Benthos ⭐ 7,564
Fancy stream processing made operationally mundane
dependent packages 12total releases 51latest release November 24, 2023most recent commit 18 days ago
Cloudquery ⭐ 5,380
The open source high performance data integration platform built for developers.
dependent packages 7total releases 345latest release May 22, 2023most recent commit 3 months ago
The 10 Latest Releases In Etl Data Engineering Open Source Projects
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Conduit ⭐ 321
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
total releases 446latest release December 07, 2023most recent commit 3 months ago
Dataform ⭐ 757
Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
dependent packages 15total releases 222latest release December 07, 2023most recent commit 4 months agonpm @dataform/core} Downloads
Datachecks ⭐ 117
Open Source Data Quality Monitoring.
total releases 14latest release December 07, 2023most recent commit 4 months ago
Etl ⭐ 327
PHP - ETL (Extract Transform Load) data processing library
dependent packages 15total releases 16latest release November 30, 2023most recent commit 4 months agopackagist flow-php/etl} Downloads
Drivers ⭐ 53
Low-code Python library enabling access to APIs, tools, data sources in seconds.
total releases 403latest release November 29, 2023most recent commit 4 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.