Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Data Pipeline Open Source Projects

Open source projects categorized as Data Pipeline
Categories > Data Pipeline
Airflow ⭐ 34,468
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169most recent commit 2 days agopypi apache-airflow} Downloads
Airbyte ⭐ 12,918
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
dependent packages 11total releases 311most recent commit 3 months agopypi airbyte-cdk} Downloads
Dolphinscheduler ⭐ 11,613
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
dependent packages 28total releases 24most recent commit 3 months ago
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585most recent commit 3 months agopypi dagster} Downloads
Snowplow ⭐ 6,677
The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP
total releases 42most recent commit 3 months ago
Mage Ai ⭐ 6,324
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
total releases 314most recent commit 3 months ago
Kestra ⭐ 5,257
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
dependent packages 4total releases 58most recent commit 3 months ago
Unstructured ⭐ 4,404
Open source libraries and APIs to build custom preprocessing pipelines for labeling, training, or production machine learning pipelines.
dependent packages 65total releases 114most recent commit 3 months agopypi unstructured} Downloads
Orchest ⭐ 3,876
Build data pipelines, the easy way 🛠️
total releases 19most recent commit a year ago
Rudder Server ⭐ 3,841
Privacy and Security focused Segment-alternative, in Golang and React
dependent packages 1total releases 298most recent commit 3 months ago
Memphis ⭐ 3,078
Memphis.dev is a highly scalable and effortless data streaming platform
total releases 3most recent commit 3 months ago
Data Engineering Howto ⭐ 2,949
A list of useful resources to learn Data Engineering from scratch
most recent commit 4 months ago
Whylogs ⭐ 2,533
An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈
total releases 20most recent commit 21 days ago
Fluvio ⭐ 2,373
Lean and mean distributed stream processing system written in rust and web assembly.
dependent packages 17total releases 22most recent commit 3 months agocargo fluvio-types} Downloads
Elementary ⭐ 1,721
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
dependent packages 1total releases 68most recent commit 20 days agopypi elementary-data} Downloads
Go Streams ⭐ 1,656
A lightweight stream processing library for Go
dependent packages 8total releases 19most recent commit 3 months ago
Doit ⭐ 1,590
task management & automation tool
dependent packages 62total releases 48most recent commit a year agopypi doit} Downloads
Bitsail ⭐ 1,514
BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
most recent commit 4 months ago
Mleap ⭐ 1,479
MLeap: Deploy ML Pipelines to Production
dependent packages 12total releases 26most recent commit 5 months ago
Meltano ⭐ 1,460
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
dependent packages 1total releases 263most recent commit 3 months agopypi meltano} Downloads
Data Science On Gcp ⭐ 1,249
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
most recent commit 9 months ago
Odd Platform ⭐ 1,047
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
most recent commit 3 months ago
Data Engineering Wiki ⭐ 934
The best place to learn data engineering. Built and maintained by the data engineering community.
most recent commit 3 months ago
The 10 Most Depended On Data Pipeline Open Source Projects
Airflow ⭐ 34,468
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169latest release November 27, 2023most recent commit 2 days agopypi apache-airflow} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Doit ⭐ 1,590
task management & automation tool
dependent packages 62total releases 48latest release April 22, 2022most recent commit a year agopypi doit} Downloads
Dolphinscheduler ⭐ 11,613
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
dependent packages 28total releases 24latest release October 11, 2023most recent commit 3 months ago
Datajoint Python ⭐ 158
Relational data pipelines for the science lab
dependent packages 26total releases 66latest release September 21, 2022most recent commit 4 months agopypi datajoint} Downloads
Versatile Data Kit ⭐ 389
One framework to develop, deploy and operate data workflows with Python and SQL.
dependent packages 25total releases 181latest release November 28, 2023most recent commit 3 months agopypi vdk-core} Downloads
Fluvio ⭐ 2,373
Lean and mean distributed stream processing system written in rust and web assembly.
dependent packages 17total releases 22latest release September 28, 2023most recent commit 3 months agocargo fluvio-types} Downloads
Dataform ⭐ 757
Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
dependent packages 15total releases 222latest release December 07, 2023most recent commit 3 months agonpm @dataform/core} Downloads
Mleap ⭐ 1,479
MLeap: Deploy ML Pipelines to Production
dependent packages 12total releases 26latest release May 07, 2021most recent commit 5 months ago
The 10 Latest Releases In Data Pipeline Open Source Projects
Piperider ⭐ 443
Code review for data in dbt
total releases 427latest release December 11, 2023most recent commit 4 months ago
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Conduit ⭐ 321
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
total releases 446latest release December 07, 2023most recent commit 3 months ago
Dataform ⭐ 757
Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
dependent packages 15total releases 222latest release December 07, 2023most recent commit 3 months agonpm @dataform/core} Downloads
Rudder Server ⭐ 3,841
Privacy and Security focused Segment-alternative, in Golang and React
dependent packages 1total releases 298latest release December 07, 2023most recent commit 3 months ago
Covalent ⭐ 608
Pythonic tool for running machine-learning/high performance/quantum-computing workflows in heterogeneous environments.
dependent packages 2total releases 118latest release December 01, 2023most recent commit 3 months agopypi covalent} Downloads
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.