Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Pipeline Etl Open Source Projects

Open source projects categorized as Pipeline Etl
Airbyte ⭐ 12,918
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
dependent packages 11total releases 311most recent commit 3 months agopypi airbyte-cdk} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585most recent commit 3 months agopypi dagster} Downloads
Mage Ai ⭐ 6,324
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
total releases 314most recent commit 3 months ago
Kestra ⭐ 5,257
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
dependent packages 4total releases 58most recent commit 3 months ago
Orchest ⭐ 3,876
Build data pipelines, the easy way 🛠️
total releases 19most recent commit a year ago
Awesome Etl ⭐ 3,041
A curated list of awesome ETL frameworks, libraries, and software.
most recent commit 3 months ago
Mara Pipelines ⭐ 2,053
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
dependent packages 1total releases 8most recent commit 4 months agopypi mara-pipelines} Downloads
Go Streams ⭐ 1,656
A lightweight stream processing library for Go
dependent packages 8total releases 19most recent commit 3 months ago
Vdp ⭐ 1,556
💧 Instill VDP (Versatile Data Pipeline) is an open-source tool to seamlessly integrate AI to process unstructured data in the modern data stack
dependent packages 1total releases 42most recent commit 3 months ago
Onepanel ⭐ 704
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.
dependent packages 1total releases 64most recent commit a year ago
Neumai ⭐ 693
Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.
most recent commit 3 months ago
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 4 years ago
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181most recent commit 3 months agopypi vdk-core} Downloads
Smooks ⭐ 377
Extensible data integration Java framework for building XML and non-XML fragment-based applications
dependent packages 14total releases 5most recent commit 3 months ago
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35most recent commit 5 months agopypi butterfree} Downloads
Cuelake ⭐ 266
Use SQL to build ELT pipelines on a data lakehouse.
most recent commit 2 years ago
Naas ⭐ 266
Low-code Python library to safely use notebooks in production: schedule workflows, generate assets, trigger webhooks, send notifications, build pipelines, manage secrets (Cloud-only)
most recent commit 3 months ago
Bulk Writer ⭐ 218
Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.
most recent commit 2 years ago
Setl ⭐ 173
A simple Spark-powered ETL framework that just works 🍺
total releases 4most recent commit 5 months ago
Dataplane ⭐ 171
Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
total releases 30most recent commit 4 months ago
Airflow_for_beginners ⭐ 166
most recent commit 2 years ago
Analytics Readings ⭐ 155
Readings for Analytics Engineers
most recent commit a year ago
Metl ⭐ 154
mito ETL tool
most recent commit 3 years ago
The 10 Most Depended On Pipeline Etl Open Source Projects
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181latest release November 28, 2023most recent commit 3 months agopypi vdk-core} Downloads
Smooks ⭐ 377
Extensible data integration Java framework for building XML and non-XML fragment-based applications
dependent packages 14total releases 5latest release June 19, 2023most recent commit 3 months ago
Go Streams ⭐ 1,656
A lightweight stream processing library for Go
dependent packages 8total releases 19latest release October 17, 2022most recent commit 3 months ago
Barnard59 ⭐ 20
An intuitive and flexible RDF pipeline solution designed to simplify and automate ETL processes for efficient data management.
dependent packages 5total releases 16latest release June 08, 2022most recent commit 3 months agonpm barnard59} Downloads
Datayoga ⭐ 27
streaming data pipeline platform
dependent packages 3total releases 34latest release November 26, 2022most recent commit 5 months agonpm @datayoga-io/shared} Downloads
Diem ⭐ 10
DIEM Data Integration Engine Multipurpose
dependent packages 3total releases 2latest release February 24, 2021most recent commit 4 months agonpm @mydiem/diem-util} Downloads
Lineage ⭐ 14
Generate beautiful documentation for your data pipelines in markdown format
dependent packages 2total releases 11latest release January 26, 2022most recent commit 2 years agonpm @datayoga-io/lineage} Downloads
The 10 Latest Releases In Pipeline Etl Open Source Projects
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Mara Pipelines ⭐ 2,053
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
dependent packages 1total releases 8latest release December 06, 2023most recent commit 4 months agopypi mara-pipelines} Downloads
Drivers ⭐ 53
Low-code Python library enabling access to APIs, tools, data sources in seconds.
total releases 403latest release November 29, 2023most recent commit 3 months ago
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181latest release November 28, 2023most recent commit 3 months agopypi vdk-core} Downloads
Covalent Ecs Plugin ⭐ 7
Executor plugin interfacing Covalent with Amazon ECS Fargate
total releases 35latest release November 21, 2023most recent commit 5 months ago
Butterfree ⭐ 269
A tool for building feature stores.
dependent packages 1total releases 35latest release November 14, 2023most recent commit 5 months agopypi butterfree} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.