Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Data Integration Open Source Projects

Open source projects categorized as Data Integration
Categories > Data Integration
Airflow ⭐ 34,468
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169most recent commit an hour agopypi apache-airflow} Downloads
Airbyte ⭐ 12,918
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
dependent packages 11total releases 311most recent commit 3 months agopypi airbyte-cdk} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585most recent commit 3 months agopypi dagster} Downloads
Seatunnel ⭐ 7,139
SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
dependent packages 28total releases 4most recent commit a month ago
Mage Ai ⭐ 6,324
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
total releases 314most recent commit 3 months ago
Cloudquery ⭐ 5,380
The open source high performance data integration platform built for developers.
dependent packages 7total releases 345most recent commit 3 months ago
Kestra ⭐ 5,257
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
dependent packages 4total releases 58most recent commit 3 months ago
Hudi ⭐ 4,901
Upserts, Deletes And Incremental Processing on Big Data.
dependent packages 58total releases 21most recent commit 3 months ago
Chunjun ⭐ 3,893
A data integration framework
most recent commit 13 days ago
Rudder Server ⭐ 3,841
Privacy and Security focused Segment-alternative, in Golang and React
dependent packages 1total releases 298most recent commit 3 months ago
Jitsu ⭐ 3,723
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
dependent packages 2total releases 38most recent commit 3 months agonpm @jitsu/catalog} Downloads
Awesome Single Cell ⭐ 2,784
Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.
most recent commit 3 months ago
Fluvio ⭐ 2,373
Lean and mean distributed stream processing system written in rust and web assembly.
dependent packages 17total releases 22most recent commit 3 months agocargo fluvio-types} Downloads
Incubator Devlake ⭐ 2,322
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
total releases 207most recent commit 3 months ago
Mara Pipelines ⭐ 2,053
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
dependent packages 1total releases 8most recent commit 4 months agopypi mara-pipelines} Downloads
Bitsail ⭐ 1,514
BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
most recent commit 4 months ago
Kuwala ⭐ 610
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demograp
most recent commit 2 years ago
Hudi Resources ⭐ 509
汇总Apache Hudi相关资料
most recent commit 3 months ago
Transfer ⭐ 495
Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.
total releases 2most recent commit 3 months ago
Harmony ⭐ 460
Fast, sensitive and accurate integration of single-cell data with Harmony
most recent commit 4 months ago
Nichenetr ⭐ 416
NicheNet: predict active ligand-target links between interacting cells
most recent commit 3 months ago
Seatunnel Web ⭐ 365
SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).
dependent packages 2total releases 1most recent commit 3 months ago
Conduit ⭐ 321
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
total releases 446most recent commit 3 months ago
The 10 Most Depended On Data Integration Open Source Projects
Airflow ⭐ 34,468
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dependent packages 320total releases 169latest release November 27, 2023most recent commit an hour agopypi apache-airflow} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Hudi ⭐ 4,901
Upserts, Deletes And Incremental Processing on Big Data.
dependent packages 58total releases 21latest release November 11, 2023most recent commit 3 months ago
Seatunnel ⭐ 7,139
SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
dependent packages 28total releases 4latest release August 18, 2023most recent commit a month ago
Bio2bel ⭐ 14
A Python framework for integrating biological databases and structured data sources in Biological Expression Language (BEL)
dependent packages 21total releases 40latest release November 03, 2021most recent commit 2 years agopypi bio2bel} Downloads
Fluvio ⭐ 2,373
Lean and mean distributed stream processing system written in rust and web assembly.
dependent packages 17total releases 22latest release September 28, 2023most recent commit 3 months agocargo fluvio-types} Downloads
Harmonypy ⭐ 116
🎼 Integrate multiple high-dimensional datasets with fuzzy k-means and locally linear adjustments.
dependent packages 14total releases 8latest release November 23, 2022most recent commit a year agopypi harmonypy} Downloads
Cloudquery ⭐ 5,380
The open source high performance data integration platform built for developers.
dependent packages 7total releases 345latest release May 22, 2023most recent commit 3 months ago
Barnard59 ⭐ 20
An intuitive and flexible RDF pipeline solution designed to simplify and automate ETL processes for efficient data management.
dependent packages 5total releases 16latest release June 08, 2022most recent commit 3 months agonpm barnard59} Downloads
The 10 Latest Releases In Data Integration Open Source Projects
Scarches ⭐ 294
Reference mapping for single-cell genomics
dependent packages 2total releases 28latest release December 08, 2023most recent commit 3 months agopypi scArches} Downloads
Dagster ⭐ 9,467
An orchestration platform for the development, production, and observation of data assets.
dependent packages 133total releases 585latest release December 07, 2023most recent commit 3 months agopypi dagster} Downloads
Conduit ⭐ 321
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
total releases 446latest release December 07, 2023most recent commit 3 months ago
Rudder Server ⭐ 3,841
Privacy and Security focused Segment-alternative, in Golang and React
dependent packages 1total releases 298latest release December 07, 2023most recent commit 3 months ago
Mara Pipelines ⭐ 2,053
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
dependent packages 1total releases 8latest release December 06, 2023most recent commit 4 months agopypi mara-pipelines} Downloads
Nomenklatura ⭐ 171
Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources
dependent packages 5total releases 122latest release December 01, 2023most recent commit 3 months agopypi nomenklatura} Downloads
Drivers ⭐ 53
Low-code Python library enabling access to APIs, tools, data sources in seconds.
total releases 403latest release November 29, 2023most recent commit 3 months ago
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.