The Top 106 Etl Open Source Projects

Awesome Open Source

Awesome Open Source

Combined Topics

Advertising 📦 10

Application Programming Interfaces 📦 124

Applications 📦 192

Artificial Intelligence 📦 78

Blockchain 📦 73

Build Tools 📦 113

Cloud Computing 📦 80

Code Quality 📦 28

Collaboration 📦 32

Command Line Interface 📦 49

Community 📦 83

Companies 📦 60

Compilers 📦 63

Computer Science 📦 80

Configuration Management 📦 42

Content Management 📦 175

Control Flow 📦 213

Data Formats 📦 78

Data Processing 📦 276

Data Storage 📦 135

Economics 📦 64

Frameworks 📦 215

Graphics 📦 110

Hardware 📦 152

Integrated Development Environments 📦 49

Learning Resources 📦 166

Libraries 📦 129

Lists Of Projects 📦 22

Machine Learning 📦 347

Mapping 📦 64

Marketing 📦 15

Mathematics 📦 55

Messaging 📦 98

Networking 📦 315

Operating Systems 📦 89

Operations 📦 121

Package Managers 📦 55

Programming Languages 📦 245

Runtime Environments 📦 100

Science 📦 42

Security 📦 396

Social Media 📦 27

Software Architecture 📦 72

Software Development 📦 72

Software Performance 📦 58

Software Quality 📦 133

Text Editors 📦 49

Text Processing 📦 136

User Interface 📦 330

User Interface Components 📦 514

Version Control 📦 30

Virtualization 📦 71

Web Browsers 📦 42

Web Servers 📦 26

Web User Interface 📦 210

The Top 106 Etl Open Source Projects

Categories > Data Processing > Etl

Dagster ⭐ 3,477

A data orchestrator for machine learning, analytics, and ETL.

Airbyte ⭐ 3,200

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

Benthos ⭐ 3,174

Declarative stream processing for mundane tasks and data engineering

Linq2db ⭐ 2,073

Linq to database provider.

Aws Data Wrangler ⭐ 1,879

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Mara Pipelines ⭐ 1,732

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Data processing & ETL framework for Ruby

A Python stream processing engine modeled after Yahoo! Pipes

Dataspherestudio ⭐ 1,385

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Awesome Business Intelligence ⭐ 1,238

Actively curated list of awesome BI tools. PRs welcome!

Transporter ⭐ 1,190

Sync data between persistence engines, like ETL only not stodgy

Ethereum Etl ⭐ 1,063

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

Panther ⭐ 898

[DEPRECATED] Detect threats with log data and improve cloud security posture

Getting Started ⭐ 798

This repository is a getting started guide to Singer.

Monstache ⭐ 787

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.

React Csv ⭐ 781

React components to build CSV files on the fly basing on Array/literal object of data

Pyspark Example Project ⭐ 688

Example project implementing best practices for PySpark ETL jobs and applications.

Go Streams ⭐ 662

A lightweight stream processing library for Go

Baby Names Analysis ⭐ 555

Data ETL & Analysis on the dataset 'Baby Names from Social Security Card Applications - National Data'.

Ananas Desktop ⭐ 552

A hackable data integration & analysis tool to enable non technical users to edit data processing jobs and visualise data on demand.

🔮 Transform, query, and download geospatial data on the web.

Pglogical ⭐ 491

Logical Replication extension for PostgreSQL 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

Bigslice ⭐ 477

A serverless cluster computing system for the Go programming language

Smartcode ⭐ 473

SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!

Etlalchemy ⭐ 470

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

Aistore ⭐ 416

AIStore: scalable storage for AI applications

Datacleaner ⭐ 410

The premier open source Data Quality solution

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

Dataform ⭐ 398

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

Wedatasphere ⭐ 395

WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!

Metorikku ⭐ 388

A simplified, lightweight ETL Framework based on Apache Spark

Power of appbase.io via CLI, with nifty imports from your favorite data sources

Webkettle ⭐ 362

基于web版kettle开发的一套分布式综合调度,管理,ETL开发的用户专业版B/S架构工具

Omniparser ⭐ 352

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

Extensible data integration Java framework for building XML and non-XML fragment-based applications

Datavec ⭐ 276

ETL Library for Machine Learning - data pipelines, data munging and wrangling

Example Airflow Dags ⭐ 264

Example DAGs using hooks and operators from Airflow Plugins

Aws Etl Orchestrator ⭐ 263

A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

Etl2pcapng ⭐ 257

Utility that converts an .etl file containing a Windows network packet capture into .pcapng format.

Data Making Guidelines ⭐ 254

📘 Making Data, the DataMade Way

Storagetapper ⭐ 248

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

Elastic ⭐ 230

R client for the Elasticsearch HTTP API

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

Bulk Writer ⭐ 212

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

DataX is an open source universal ETL tool that support Cassandra, ClickHouse, DBF, Hive, InfluxDB, Kudu, MySQL, Oracle, Presto(Trino), PostgreSQL, SQL Server

Categorical Query Language IDE

Cuelake ⭐ 205

Use SQL to build ELT pipelines on a data lakehouse.

Aws Serverless Data Lake Framework ⭐ 204

Enterprise-grade, production-hardened, serverless data lake on AWS

Bitcoin Etl ⭐ 202

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Extract ⭐ 191

A cross-platform command line tool for parallelised content extraction and analysis.

Mongo Es ⭐ 188

A MongoDB to Elasticsearch connector

Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at www.jumpmind.com/products/metl/overview

Grafter ⭐ 178

Linked Data & RDF Manufacturing Tools in Clojure

Usaspending Api ⭐ 175

Server application to serve U.S. federal spending data via a RESTful API

Bender - Serverless ETL Framework

Open Semantic Etl ⭐ 169

Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database

Mara Example Project 2 ⭐ 157

An example mini data warehouse for python project stats, template for new projects

Butterfree ⭐ 157

A tool for building feature stores.

Etl_unicorn ⭐ 155

数据可视化, 数据挖掘, 数据处理 ETL

Kettle Web ⭐ 146

基于spring boot通过java代码调用kette

Hydrograph ⭐ 144

A visual ETL development and debugging tool for big data

Reddit Detective ⭐ 143

Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more

Etl.net ⭐ 142

Mass processing data with a complete ETL for .net developers

Eel Sdk ⭐ 140

Big Data Toolkit for the JVM

Transformalize ⭐ 129

Configurable Extract, Transform, and Load

Openkettlewebui ⭐ 129

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

Sentinel Crawler ⭐ 119

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器，分布式爬虫

Marklogic Data Hub ⭐ 116

The MarkLogic Data Hub: documentation ==>

Aws Ecs Airflow ⭐ 114

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

The CSV to database command line loader

Kafka Connect File Pulse ⭐ 111

🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

Česká otevřená data

Kafka Connect ⭐ 108

equivalent to kafka-connect 🔧 for nodejs ✨🐢🚀✨

Zdh_web ⭐ 106

大数据采集,抽取平台

Data Story ⭐ 100

A visual process builder

LinkedPipes ETL is an RDF based, lightweight ETL tool

Udacity Data Engineering ⭐ 96

Udacity Data Engineering Nano Degree (DEND)

A simple Spark-powered ETL framework that just works 🍺

Beginner_de_project ⭐ 95

Beginner data engineering project - batch edition

A Go native tabular data extraction package. Currently supports .xls, .xlsx, .csv, .tsv formats.

Open Data Etl Utility Kit ⭐ 94

Use Pentaho's open source data integration tool (Kettle) to create Extract-Transform-Load (ETL) processes to update a Socrata open data portal. Documentation is available at http://open-data-etl-utility-kit.readthedocs.io/en/stable

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Target Postgres ⭐ 80

A Singer.io Target for Postgres

⚙️ Schedule notebooks, run them like APIs, expose securely your assets: Jupyter as a viable production environment.

locopy: Loading/Unloading to Redshift and Snowflake using Python.

Etl_with_python ⭐ 73

ETL with Python - Taught at DWH course 2017 (TAU)

Luigi Warehouse ⭐ 72

A luigi powered analytics / warehouse stack

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

Discreetly ⭐ 62

ETLy is an add-on dashboard service on top of Apache Airflow.

Snapflow ⭐ 60

Functional reactive data pipelines

Bentools Etl ⭐ 47

PHP ETL (Extract / Transform / Load) library with SOLID principles + almost no dependency.

Kiba Plus ⭐ 47

Kiba enhancement for Ruby ETL.

Ether_sql ⭐ 45

A python library to push ethereum blockchain data into an sql database.

python ETL framework

Public, free to use, repository with diggers configs for scraping / extracting data from various e-commerce websites and online stores

Pangeo Forge Recipes ⭐ 37

Python library for building Pangeo Forge recipes.

Mongodb Rdbms Sync ⭐ 34

Bi-directional data sync between Relational Databases (Oracle, MySql) & MongoDB.

1-100 of 106 projects

Advertising 📦 10

Application Programming Interfaces 📦 124

Applications 📦 192

Artificial Intelligence 📦 78

Blockchain 📦 73

Build Tools 📦 113

Cloud Computing 📦 80

Code Quality 📦 28

Collaboration 📦 32

Command Line Interface 📦 49

Community 📦 83

Companies 📦 60

Compilers 📦 63

Computer Science 📦 80

Configuration Management 📦 42

Content Management 📦 175

Control Flow 📦 213

Data Formats 📦 78

Data Processing 📦 276

Data Storage 📦 135

Economics 📦 64

Frameworks 📦 215

Graphics 📦 110

Hardware 📦 152

Integrated Development Environments 📦 49

Learning Resources 📦 166

Libraries 📦 129

Lists Of Projects 📦 22

Machine Learning 📦 347

Mapping 📦 64

Marketing 📦 15

Mathematics 📦 55

Messaging 📦 98

Networking 📦 315

Operating Systems 📦 89

Operations 📦 121

Package Managers 📦 55

Programming Languages 📦 245

Runtime Environments 📦 100

Science 📦 42

Security 📦 396

Social Media 📦 27

Software Architecture 📦 72

Software Development 📦 72

Software Performance 📦 58

Software Quality 📦 133

Text Editors 📦 49

Text Processing 📦 136

User Interface 📦 330

User Interface Components 📦 514

Version Control 📦 30

Virtualization 📦 71

Web Browsers 📦 42

Web Servers 📦 26

Web User Interface 📦 210