Awesome Open Source
Awesome Open Source
Combined Topics
etl x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 97 Etl Open Source Projects

Dagster ⭐ 3,088
A data orchestrator for machine learning, analytics, and ETL.
Benthos ⭐ 2,979
Declarative stream processing for mundane tasks and data engineering
Airbyte ⭐ 2,186
Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.
Linq2db ⭐ 2,014
Linq to database provider.
Mara Pipelines ⭐ 1,661
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Aws Data Wrangler ⭐ 1,628
Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Kiba ⭐ 1,580
Data processing & ETL framework for Ruby
Riko ⭐ 1,563
A Python stream processing engine modeled after Yahoo! Pipes
Dataspherestudio ⭐ 1,235
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Transporter ⭐ 1,179
Sync data between persistence engines, like ETL only not stodgy
Awesome Business Intelligence ⭐ 1,173
Actively curated list of awesome BI tools. PRs welcome!
Ethereum Etl ⭐ 976
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
Panther ⭐ 893
[DEPRECATED] Detect threats with log data and improve cloud security posture
Monstache ⭐ 752
a go daemon that syncs MongoDB to Elasticsearch in realtime
Getting Started ⭐ 745
This repository is a getting started guide to Singer.
React Csv ⭐ 736
React components to build CSV files on the fly basing on Array/literal object of data
Pyspark Example Project ⭐ 646
Example project implementing best practices for PySpark ETL jobs and applications.
Go Streams ⭐ 619
A lightweight stream processing library for Go
Baby Names Analysis ⭐ 557
Data ETL & Analysis on the dataset 'Baby Names from Social Security Card Applications - National Data'.
Ananas Desktop ⭐ 551
A hackable data integration & analysis tool to enable non technical users to edit data processing jobs and visualise data on demand.
Koop ⭐ 507
🔮 Transform, query, and download geospatial data on the web.
Bigslice ⭐ 472
A serverless cluster computing system for the Go programming language
Smartcode ⭐ 467
SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!
Etlalchemy ⭐ 461
Extract, Transform, Load: Any SQL Database in 4 lines of Code.
Pglogical ⭐ 458
Logical Replication extension for PostgreSQL 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.
Datacleaner ⭐ 393
The premier open source Data Quality solution
Aistore ⭐ 381
AIStore: scalable storage for AI applications
Choetl ⭐ 380
ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
Wedatasphere ⭐ 379
WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!
Abc ⭐ 376
Power of appbase.io via CLI, with nifty imports from your favorite data sources
Metorikku ⭐ 367
A simplified, lightweight ETL Framework based on Apache Spark
Dataform ⭐ 356
Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
Webkettle ⭐ 342
基于web版kettle开发的一套分布式综合调度,管理,ETL开发的用户专业版B/S架构工具
Omniparser ⭐ 313
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
Smooks ⭐ 296
Extensible Java framework for building XML and non-XML fragment-based applications
Datavec ⭐ 272
ETL Library for Machine Learning - data pipelines, data munging and wrangling
Data Making Guidelines ⭐ 249
📘 Making Data, the DataMade Way
Example Airflow Dags ⭐ 247
Example DAGs using hooks and operators from Airflow Plugins
Aws Etl Orchestrator ⭐ 247
A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.
Eland ⭐ 242
Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
Storagetapper ⭐ 236
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Etl2pcapng ⭐ 234
Utility that converts an .etl file containing a Windows network packet capture into .pcapng format.
Elastic ⭐ 229
R client for the Elasticsearch HTTP API
Bulk Writer ⭐ 211
Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.
Etlbox ⭐ 206
A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.
Cql ⭐ 197
Categorical Query Language IDE
Extract ⭐ 189
A cross-platform command line tool for parallelised content extraction and analysis.
Mongo Es ⭐ 186
A MongoDB to Elasticsearch connector
Metl ⭐ 185
Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at www.jumpmind.com/products/metl/overview
Aws Serverless Data Lake Framework ⭐ 185
Enterprise-grade, production-hardened, serverless data lake on AWS
Bitcoin Etl ⭐ 180
ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ
Grafter ⭐ 174
Linked Data & RDF Manufacturing Tools in Clojure
Bender ⭐ 171
Bender - Serverless ETL Framework
Usaspending Api ⭐ 167
Server application to serve U.S. federal spending data via a RESTful API
Open Semantic Etl ⭐ 165
Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database
Etl_unicorn ⭐ 156
数据可视化, 数据挖掘, 数据处理 ETL
Mara Example Project 2 ⭐ 155
An example mini data warehouse for python project stats, template for new projects
Metl ⭐ 153
mito ETL tool
Hydrograph ⭐ 144
A visual ETL development and debugging tool for big data
Eel Sdk ⭐ 140
Big Data Toolkit for the JVM
Datax ⭐ 135
DataX is an open source universal ETL tool that support Cassandra, ClickHouse, DBF, Hive, InfluxDB, Kudu, MySQL, Oracle, Presto(Trino), PostgreSQL, SQL Server
Butterfree ⭐ 135
A tool for building feature stores.
Reddit Detective ⭐ 132
Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more
Etl.net ⭐ 130
Mass processing data with a complete ETL for .net developers
Kettle Web ⭐ 129
基于spring boot通过java代码调用kette
Openkettlewebui ⭐ 128
一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中
Transformalize ⭐ 126
Configurable Extract, Transform, and Load
Sentinel Crawler ⭐ 118
Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器，分布式爬虫
Marklogic Data Hub ⭐ 113
The MarkLogic Data Hub: documentation ==>
Aws Ecs Airflow ⭐ 108
Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
Kafka Connect ⭐ 106
equivalent to kafka-connect 🔧 for nodejs ✨🐢🚀✨
Csv2db ⭐ 103
The CSV to database command line loader
Od ⭐ 99
Česká otevřená data
Open Data Etl Utility Kit ⭐ 93
Use Pentaho's open source data integration tool (Kettle) to create Extract-Transform-Load (ETL) processes to update a Socrata open data portal. Documentation is available at http://open-data-etl-utility-kit.readthedocs.io/en/stable
Etl ⭐ 91
LinkedPipes ETL is an RDF based, lightweight ETL tool
Udacity Data Engineering ⭐ 90
Udacity Data Engineering Nano Degree (DEND)
Hale ⭐ 85
(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)
Data Story ⭐ 82
A visual process builder
Setl ⭐ 80
A simple Spark-powered ETL framework that just works 🍺
Sayn ⭐ 79
Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).
Target Postgres ⭐ 73
A Singer.io Target for Postgres
Locopy ⭐ 73
locopy: Loading/Unloading to Redshift and Snowflake using Python.
Luigi Warehouse ⭐ 72
A luigi powered analytics / warehouse stack
Etl_with_python ⭐ 70
ETL with Python - Taught at DWH course 2017 (TAU)
Stetl ⭐ 64
Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.
Discreetly ⭐ 59
ETLy is an add-on dashboard service on top of Apache Airflow.
Kiba Plus ⭐ 47
Kiba enhancement for Ruby ETL.
Bentools Etl ⭐ 46
PHP ETL (Extract / Transform / Load) library with SOLID principles + almost no dependency.
Ether_sql ⭐ 41
A python library to push ethereum blockchain data into an sql database.
Configs ⭐ 37
Public, free to use, repository with diggers configs for scraping / extracting data from various e-commerce websites and online stores
Pyetl ⭐ 35
python ETL framework
Yunmai Data Extract ⭐ 21
Extract your data from the Yunmai weighing scales cloud API so you can use it elsewhere
Aws Auto Terminate Idle Emr ⭐ 21
AWS Auto Terminate Idle AWS EMR Clusters Framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.
Bandar Log ⭐ 19
Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.
Phila Airflow ⭐ 16
Dswarm Backoffice Web ⭐ 11
The backoffice web application of d:swarm (https://github.com/dswarm/dswarm-documentation/wiki)
Tuna ⭐ 11
🐟 A streaming ETL for fish
1-97 of 97 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210