Awesome Open Source
Awesome Open Source
Combined Topics
etl x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 787 Etl Open Source Projects on Github

Airbyte ⭐ 4,337
Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.
Dagster ⭐ 3,935
A data orchestrator for machine learning, analytics, and ETL.
Benthos ⭐ 3,475
Fancy stream processing made operationally mundane
Aws Data Wrangler ⭐ 2,240
Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Linq2db ⭐ 2,166
Linq to database provider.
Mara Pipelines ⭐ 1,802
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Dataspherestudio ⭐ 1,611
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Kiba ⭐ 1,604
Data processing & ETL framework for Ruby
Riko ⭐ 1,533
A Python stream processing engine modeled after Yahoo! Pipes
Awesome Business Intelligence ⭐ 1,318
Actively curated list of awesome BI tools. PRs welcome!
Ethereum Etl ⭐ 1,215
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
Transporter ⭐ 1,148
Sync data between persistence engines, like ETL only not stodgy
Panther ⭐ 894
[DEPRECATED] Detect threats with log data and improve cloud security posture
Monstache ⭐ 839
a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
React Csv ⭐ 789
React components to build CSV files on the fly basing on Array/literal object of data
Getting Started ⭐ 763
This repository is a getting started guide to Singer.
Pyspark Example Project ⭐ 727
Example project implementing best practices for PySpark ETL jobs and applications.
Go Streams ⭐ 707
A lightweight stream processing library for Go
Baby Names Analysis ⭐ 554
Data ETL & Analysis on the dataset 'Baby Names from Social Security Card Applications - National Data'.
Ananas Desktop ⭐ 552
A hackable data integration & analysis tool to enable non technical users to edit data processing jobs and visualise data on demand.
Optimus ⭐ 548
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Onepanel ⭐ 533
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.
Koop ⭐ 519
🔮 Transform, query, and download geospatial data on the web.
Pglogical ⭐ 517
Logical Replication extension for PostgreSQL 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.
Aistore ⭐ 496
AIStore: scalable storage for AI applications
Smartcode ⭐ 479
SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!
Bigslice ⭐ 476
A serverless cluster computing system for the Go programming language
Choetl ⭐ 455
ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
Dataform ⭐ 434
Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
Metorikku ⭐ 416
A simplified, lightweight ETL Framework based on Apache Spark
Etlalchemy ⭐ 414
Extract, Transform, Load: Any SQL Database in 4 lines of Code.
Wedatasphere ⭐ 413
WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!
Datacleaner ⭐ 396
The premier open source Data Quality solution
Abc ⭐ 392
Power of appbase.io via CLI, with nifty imports from your favorite data sources
Omniparser ⭐ 372
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
Addax ⭐ 359
Addax is an open source universal ETL tool that supports most of those RDBMS and NoSQLs on the planet, helping you transfer data from any one place to another.
Webkettle ⭐ 350
基于web版kettle开发的一套分布式综合调度,管理,ETL开发的用户专业版B/S架构工具
Cascading ⭐ 321
Cascading is a feature rich API for defining and executing complex and fault tolerant data processing workflows on various cluster computing platforms. Please see https://github.com/cwensel/cascading for access to all WIP branches.
Smooks ⭐ 318
Extensible data integration Java framework for building XML and non-XML fragment-based applications
Eland ⭐ 300
Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
Etl2pcapng ⭐ 281
Utility that converts an .etl file containing a Windows network packet capture into .pcapng format.
Zingg ⭐ 276
Scalable data mastering, deduplication and entity resolution.
Datavec ⭐ 255
ETL Library for Machine Learning - data pipelines, data munging and wrangling
Data Making Guidelines ⭐ 248
📘 Making Data, the DataMade Way
Aws Serverless Data Lake Framework ⭐ 242
Enterprise-grade, production-hardened, serverless data lake on AWS
Cuelake ⭐ 240
Use SQL to build ELT pipelines on a data lakehouse.
Elastic ⭐ 230
R client for the Elasticsearch HTTP API
Etlbox ⭐ 219
A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.
Cql ⭐ 216
Categorical Query Language IDE
Bitcoin Etl ⭐ 212
ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ
Bulk Writer ⭐ 212
Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.
Example Airflow Dags ⭐ 204
Example DAGs using hooks and operators from Airflow Plugins
Storagetapper ⭐ 196
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Extract ⭐ 194
A cross-platform command line tool for parallelised content extraction and analysis.
Mongo Es ⭐ 188
A MongoDB to Elasticsearch connector
Usaspending Api ⭐ 187
Server application to serve U.S. federal spending data via a RESTful API
Aws Etl Orchestrator ⭐ 185
A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.
Metl ⭐ 185
Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at www.jumpmind.com/products/metl/overview
Grafter ⭐ 178
Linked Data & RDF Manufacturing Tools in Clojure
Tis Solr ⭐ 176
an enterprise ETL System base on DataX
Bender ⭐ 173
Bender - Serverless ETL Framework
Butterfree ⭐ 166
A tool for building feature stores.
Kettle Web ⭐ 157
基于spring boot通过java代码调用kette
Mara Example Project 2 ⭐ 157
An example mini data warehouse for python project stats, template for new projects
Etl.net ⭐ 156
Mass processing data with a complete ETL for .net developers
Etl_unicorn ⭐ 156
数据可视化, 数据挖掘, 数据处理 ETL
Metl ⭐ 154
mito ETL tool
Kafka Connect File Pulse ⭐ 151
🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka
Squirtle ⭐ 151
Flock: A Practical Serverless Streaming Query Engine
Eel Sdk ⭐ 140
Big Data Toolkit for the JVM
Hydrograph ⭐ 138
A visual ETL development and debugging tool for big data
Dbtvault ⭐ 133
A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
Transformalize ⭐ 129
Configurable Extract, Transform, and Load
Openkettlewebui ⭐ 129
一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中
Reddit Detective ⭐ 126
Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more
Sentinel Crawler ⭐ 118
Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器，分布式爬虫
Marklogic Data Hub ⭐ 116
The MarkLogic Data Hub: documentation ==>
Od ⭐ 116
Česká otevřená data
Naas ⭐ 113
⚙️ Schedule notebooks, run them like APIs, expose securely your assets: Jupyter as a viable ⚡️ Production environment
Replicadb ⭐ 111
ReplicaDB is open source tool for database replication, designed for efficiently transferring bulk data between relational and non-relational databases
Aws Ecs Airflow ⭐ 110
Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
Csv2db ⭐ 109
The CSV to database command line loader
Zdh_web ⭐ 107
大数据采集,抽取平台
Dataxserver ⭐ 106
为DataX(https://github.com/alibaba/DataX) 提供远程多语言调用（ThriftServer，HttpServer） 分布式运行（DataX on YARN） 功能
Data Story ⭐ 105
A visual process builder
Etl ⭐ 102
LinkedPipes ETL is an RDF based, lightweight ETL tool
Kafka Connect ⭐ 102
equivalent to kafka-connect 🔧 for nodejs ✨🐢🚀✨
Setl ⭐ 101
A simple Spark-powered ETL framework that just works 🍺
Neo4j Jdbc ⭐ 99
Neo4j JDBC driver for Neo4j 3.x and 4.x with BOLT protocol
Hale ⭐ 99
(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)
Grate ⭐ 96
A Go native tabular data extraction package. Currently supports .xls, .xlsx, .csv, .tsv formats.
Sayn ⭐ 94
Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).
Carry ⭐ 93
Python ETL(Extract-Transform-Load) tool / Data migration tool
Cobrix ⭐ 91
A COBOL parser and Mainframe/EBCDIC data source for Apache Spark
Nbi ⭐ 91
NBi is a testing framework (add-on to NUnit) for Business Intelligence and Data Access. The main goal of this framework is to let users create tests with a declarative approach based on an Xml syntax. By the means of NBi, you don't need to develop C# or Java code to specify your tests! Either, you don't need Visual Studio or Eclipse to compile your test suite. Just create an Xml file and let the framework interpret it and play your tests. The framework is designed as an add-on of NUnit but with the possibility to port it easily to other testing frameworks.
Open Data Etl Utility Kit ⭐ 87
Use Pentaho's open source data integration tool (Kettle) to create Extract-Transform-Load (ETL) processes to update a Socrata open data portal. Documentation is available at http://open-data-etl-utility-kit.readthedocs.io/en/stable
Paperetl ⭐ 82
ETL processes for medical and scientific papers
Locopy ⭐ 78
locopy: Loading/Unloading to Redshift and Snowflake using Python.
Thain ⭐ 77
Thain is a distributed flow schedule platform.
Luigi Warehouse ⭐ 73
A luigi powered analytics / warehouse stack
1-100 of 787 projects
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2021