Awesome Open Source
Awesome Open Source
Combined Topics
etl x

The Top 2,372 Etl Open Source Projects

Tidb ⭐ 31,397
TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try free: https://tidbcloud.com/signup
dependent packages 91total releases 1,289most recent commit 7 hours ago
Airbyte ⭐ 6,860
Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.
total releases 62most recent commit 5 hours ago
Pentaho Kettle ⭐ 5,806
Pentaho Data Integration ( ETL ) a.k.a Kettle
most recent commit 18 hours ago
Dagster ⭐ 4,769
An orchestration platform for the development, production, and observation of data assets.
dependent packages 86total releases 460most recent commit 6 hours agopypi dagster} Downloads
Benthos ⭐ 4,391
Fancy stream processing made operationally mundane
dependent packages 4total releases 511most recent commit 6 hours ago
Aws Data Wrangler ⭐ 2,826
Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
dependent packages 28total releases 119most recent commit 7 hours agopypi awswrangler} Downloads
Hawk ⭐ 2,638
visualized crawler & ETL IDE written with C#/WPF
most recent commit 2 years ago
Awesome Etl ⭐ 2,514
A curated list of awesome ETL frameworks, libraries, and software.
most recent commit 4 months ago
Kestra ⭐ 2,447
Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines.
total releases 26most recent commit 13 days ago
Mara Pipelines ⭐ 1,889
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
total releases 11most recent commit a month ago
Ethereum Etl ⭐ 1,803
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
total releases 32most recent commit 3 days ago
Kiba ⭐ 1,620
Data processing & ETL framework for Ruby
dependent packages 8total releases 11most recent commit 5 months agorubygems kiba} Downloads
Riko ⭐ 1,573
A Python stream processing engine modeled after Yahoo! Pipes
total releases 59most recent commit 5 months ago
Awesome Business Intelligence ⭐ 1,509
Actively curated list of awesome BI tools. PRs welcome!
most recent commit 2 months ago
Brim ⭐ 1,321
Desktop application to efficiently search and analyze super-structured data. Powered by Zed.
total releases 19most recent commit 5 hours ago
Transporter ⭐ 1,244
Sync data between persistence engines, like ETL only not stodgy
total releases 22most recent commit 16 hours ago
Etl ⭐ 1,196
Embedded Template Library
most recent commit 7 days ago
Aws Glue Samples ⭐ 1,083
AWS Glue code samples
most recent commit 2 days ago
Etl With Airflow ⭐ 1,053
ETL best practices with airflow, with examples
most recent commit 6 months ago
Deepie ⭐ 954
DeepIE: Deep Learning for Information Extraction
most recent commit a year ago
Getting Started ⭐ 941
This repository is a getting started guide to Singer.
most recent commit 2 months ago
Monstache ⭐ 937
a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
dependent packages 1total releases 121most recent commit 2 months ago
React Csv ⭐ 879
React components to build CSV files on the fly basing on Array/literal object of data
dependent packages 77total releases 40most recent commit 3 months agonpm react-csv} Downloads
Go Streams ⭐ 840
A lightweight stream processing library for Go
dependent packages 9total releases 18most recent commit 4 months ago
Pyspark Example Project ⭐ 727
Example project implementing best practices for PySpark ETL jobs and applications.
most recent commit 10 months ago
Addax ⭐ 649
Addax is an open source universal ETL tool that supports most of those RDBMS and NoSQLs on the planet, helping you transfer data from any one place to another.
total releases 3most recent commit 21 days ago
Net Libraries That Make Your Life Easier ⭐ 618
Open Source .NET libraries that make your life easier.
most recent commit a year ago
Optimus ⭐ 610
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
dependent packages 1total releases 63most recent commit a day ago
Aistore ⭐ 600
AIStore: scalable storage for AI applications
total releases 162most recent commit a day ago
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 2 years ago
Baby Names Analysis ⭐ 555
Data ETL & Analysis on the dataset 'Baby Names from Social Security Card Applications - National Data'.
most recent commit 6 months ago
Ananas Desktop ⭐ 552
A hackable data integration & analysis tool to enable non technical users to edit data processing jobs and visualise data on demand.
most recent commit a year ago
Butano ⭐ 549
Modern C++ high level GBA engine
most recent commit 3 days ago
Onepanel ⭐ 547
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.
dependent packages 1total releases 64most recent commit 6 months ago
Pglogical ⭐ 543
Logical Replication extension for PostgreSQL 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.
most recent commit 5 months ago
Koop ⭐ 535
🔮 Transform, query, and download geospatial data on the web.
most recent commit 2 years ago
Choetl ⭐ 533
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
dependent packages 7total releases 150most recent commit 16 days agonuget ChoETL.NETStandard} Downloads
Open Semantic Search ⭐ 531
Open Source research tool to search, browse, analyze and explore large document collections by Semantic Search Engine and Open Source Text Mining & Text Analytics platform (Integrates ETL for document processing, OCR for images & PDF, named entity recognition for persons, organizations & locations, metadata management by thesaurus & ontologies, search user interface & search apps for fulltext search, faceted search & knowledge graph)
most recent commit 3 months ago
Dataform ⭐ 520
Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
dependent packages 6total releases 192most recent commit 9 days agonpm @dataform/core} Downloads
Zingg ⭐ 512
Scalable entity resolution, data mastering and deduplication using ML
most recent commit 9 hours ago
Smartcode ⭐ 504
SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!
dependent packages 5total releases 28most recent commit 20 days agonuget SmartCode} Downloads
Metorikku ⭐ 492
A simplified, lightweight ETL Framework based on Apache Spark
total releases 120most recent commit a month ago
Bigslice ⭐ 476
A serverless cluster computing system for the Go programming language
total releases 13most recent commit a year ago
Aws Glue Libs ⭐ 461
AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
most recent commit a month ago
Hamilton ⭐ 460
A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.
total releases 15most recent commit 15 days ago
Datacleaner ⭐ 455
The premier open source Data Quality solution
dependent packages 7total releases 53most recent commit 3 months ago
Bubbles ⭐ 431
[NOT MAINTAINED] Bubbles – Python ETL framework
most recent commit 5 years ago
Etlalchemy ⭐ 414
Extract, Transform, Load: Any SQL Database in 4 lines of Code.
total releases 15most recent commit 3 years ago
Abc ⭐ 412
Power of appbase.io via CLI, with nifty imports from your favorite data sources
most recent commit 2 months ago
Etlpy ⭐ 393
a smart stream-like crawler & etl python library
most recent commit 3 years ago
Tis ⭐ 381
Support agile DataOps Based on DataX and Flink-CDC with Web-UI
most recent commit 3 days ago
Omniparser ⭐ 372
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
dependent packages 1total releases 19most recent commit 9 months ago
Eland ⭐ 371
Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
dependent packages 2total releases 23most recent commit 8 days agopypi eland} Downloads
Etl ⭐ 367
Extract, Transform, and Load data with Ruby
total releases 5most recent commit 2 years ago
Hnpickup ⭐ 358
This is an educational example of a data mining web application: when is good time to post on HN
most recent commit 10 years ago
Big_data_architect_skills ⭐ 353
一个大数据架构师应该掌握的技能
most recent commit 3 years ago
Webkettle ⭐ 350
基于web版kettle开发的一套分布式综合调度,管理,ETL开发的用户专业版B/S架构工具
most recent commit a year ago
Smooks ⭐ 341
Extensible data integration Java framework for building XML and non-XML fragment-based applications
dependent packages 12total releases 20most recent commit 2 months ago
Etl2pcapng ⭐ 335
Utility that converts an .etl file containing a Windows network packet capture into .pcapng format.
most recent commit 3 months ago
Exchangis ⭐ 325
Exchangis is a lightweight,highly extensible data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources
most recent commit 2 months ago
Cascading ⭐ 321
Cascading is a feature rich API for defining and executing complex and fault tolerant data processing workflows on various cluster computing platforms. Please see https://github.com/cwensel/cascading for access to all WIP branches.
most recent commit 3 years ago
Pudl ⭐ 299
The Public Utility Data Liberation Project
total releases 13most recent commit 18 hours ago
Bitcoin Etl ⭐ 271
ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ
total releases 10most recent commit a month ago
Storagetapper ⭐ 269
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
total releases 4most recent commit 4 months ago
Synch ⭐ 268
Sync data from the other DB to ClickHouse(cluster)
total releases 18most recent commit 8 months ago
Aws Serverless Data Lake Framework ⭐ 268
Enterprise-grade, production-hardened, serverless data lake on AWS
most recent commit a month ago
Cuelake ⭐ 266
Use SQL to build ELT pipelines on a data lakehouse.
most recent commit 2 months ago
Data Making Guidelines ⭐ 248
📘 Making Data, the DataMade Way
most recent commit a year ago
Node Datapumps ⭐ 247
Node.js ETL (Extract, Transform, Load) toolkit for easy data import, export or transfer between systems.
dependent packages 5total releases 64most recent commit 4 years agonpm datapumps} Downloads
Etl.net ⭐ 242
Mass processing data with a complete ETL for .net developers
dependent packages 16total releases 5most recent commit 5 days agonuget Paillave.EtlNet.Core} Downloads
Pygrametl ⭐ 238
Official repository for pygrametl - ETL programming in Python
most recent commit 4 months ago
Activewarehouse Etl ⭐ 238
Extract-Transform-Load library from ActiveWarehouse
total releases 22most recent commit 4 years ago
Ethereum Etl Airflow ⭐ 235
Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethe
most recent commit 8 hours ago
Etl ⭐ 233
PHP - ETL (Extract Transform Load) data processing library
most recent commit 2 days ago
Elastic ⭐ 230
R client for the Elasticsearch HTTP API
dependent packages 1total releases 13most recent commit a year ago
Etlbox ⭐ 220
A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.
most recent commit 5 months ago
Zdh_web ⭐ 220
大数据采集,抽取平台
most recent commit a month ago
Dbtvault ⭐ 216
A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
most recent commit 16 days ago
Cql ⭐ 216
Categorical Query Language IDE
most recent commit a year ago
Bulk Writer ⭐ 212
Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.
dependent packages 1total releases 7most recent commit a year agonuget BulkWriter} Downloads
Beginner_de_project ⭐ 208
Beginner data engineering project - batch edition
most recent commit 16 days ago
Usaspending Api ⭐ 206
Server application to serve U.S. federal spending data via a RESTful API
most recent commit 2 days ago
Rhino Etl ⭐ 206
Main repository is here ->
most recent commit 2 years ago
Kafka Connect File Pulse ⭐ 205
🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka
dependent packages 3total releases 5most recent commit 6 days ago
Node Etl ⭐ 204
npm install etl
dependent packages 20total releases 103most recent commit 4 months agonpm etl} Downloads
Example Airflow Dags ⭐ 204
Example DAGs using hooks and operators from Airflow Plugins
most recent commit 4 years ago
Extract ⭐ 201
A cross-platform command line tool for parallelised content extraction and analysis.
dependent packages 1total releases 44most recent commit 6 months ago
Activewarehouse ⭐ 199
ActiveWarehouse for Rails - Implement data warehouses with Rails
most recent commit 8 years ago
Kettle Web ⭐ 197
基于spring boot通过java代码调用kette
most recent commit a month ago
Amundsendatabuilder ⭐ 196
Data ingestion library for Amundsen to build graph and search index
most recent commit 3 months ago
Crunch ⭐ 196
A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.
most recent commit 8 years ago
Flock ⭐ 196
Flock: A Low-Cost Streaming Query Engine on FaaS Platforms
most recent commit 2 months ago
Naas ⭐ 191
⚙️ Schedule notebooks, run them like APIs, expose securely your assets: Jupyter as a viable ⚡️ Production environment
most recent commit 8 days ago
Udacity Data Engineering Projects ⭐ 191
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
most recent commit 2 years ago
Dataflowex ⭐ 190
A .NET dataflow and etl framework built upon Microsoft TPL Dataflow library
total releases 36most recent commit 2 years ago
Mongo Es ⭐ 188
A MongoDB to Elasticsearch connector
total releases 74most recent commit 10 months ago
Etl Language Comparison ⭐ 185
Count the number of times certain words were said in a particular neighborhood. Performed as a basic MapReduce job against 25M tweets. Implemented with different programming languages as a educational exercise.
most recent commit 4 years ago
Aws Etl Orchestrator ⭐ 185
A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.
most recent commit 2 years ago
Metl ⭐ 185
Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at www.jumpmind.com/products/metl/overview
most recent commit a year ago
Conduit ⭐ 182
Data Integration for Production Data Stores.
total releases 32most recent commit 15 hours ago
1-100 of 2,372 projects
Related Awesome Lists
Python Etl Projects (792)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,048,968)
Python  (796,390)
Java  (385,473)
Php  (284,598)
Typescript  (239,890)
C Plus Plus  (236,265)
Ruby  (221,659)
C  (180,422)
Shell  (169,625)
C Sharp  (162,065)
Golang  (156,150)
Swift  (64,706)
R  (57,282)
Objective C  (57,157)
Rust  (56,147)
Dart  (51,498)
Kotlin  (45,526)
Lua  (33,305)
Matlab  (30,902)
Perl  (30,481)
Language  (30,082)
Scala  (28,602)
Haskell  (20,263)
Clojure  (19,076)
Powershell  (18,799)
Bash  (18,607)
Coffeescript  (17,328)
Elixir  (16,572)
Assembly  (14,534)
Processing  (13,301)
Julia  (12,544)
Basic  (10,763)
Flow  (10,527)
Emacs Lisp  (10,371)
Erlang  (9,056)
Scheme  (8,437)
Groovy  (8,431)
Solidity  (7,618)
Ocaml  (6,669)
Pascal  (6,384)
Hack  (6,365)
Lisp  (6,341)
Fortran  (5,485)
Elm  (5,478)
Common Lisp  (5,277)
Dlang  (4,746)
Shell Script  (4,633)
Python Library  (4,447)
Crystal  (4,375)
Scripting  (4,204)
Top Projects
Freecodecamp ⭐ 346,452
996.icu ⭐ 262,320
Free Programming Books ⭐ 233,791
Coding Interview University ⭐ 220,499
Awesome ⭐ 202,651
Vue ⭐ 196,199
Developer Roadmap ⭐ 195,459
Public Apis ⭐ 194,093
React ⭐ 188,476
System Design Primer ⭐ 180,789
Tensorflow ⭐ 165,237
Bootstrap ⭐ 157,508
You Dont Know Js ⭐ 153,649
Cs Notes ⭐ 150,780
Ohmyzsh ⭐ 145,536
Build Your Own X ⭐ 142,246
Flutter ⭐ 140,962
Javascript Algorithms ⭐ 140,055
Python ⭐ 136,329
Gitignore ⭐ 133,487
Linux ⭐ 132,226
Vscode ⭐ 132,097
Awesome Python ⭐ 127,528
Javascript ⭐ 123,347
Javaguide ⭐ 121,691
Python 100 Days ⭐ 117,585
Computer Science ⭐ 113,544
Youtube Dl ⭐ 110,239
Fucking Algorithm ⭐ 106,048
The Art Of Command Line ⭐ 105,874
React Native ⭐ 102,883
Electron ⭐ 101,825
D3 ⭐ 100,901
Go ⭐ 99,518
30 Seconds Of Code ⭐ 95,866
Create React App ⭐ 95,174
Axios ⭐ 93,587
Awesome Selfhosted ⭐ 89,230
Kubernetes ⭐ 88,712
Free Programming Books Zh_cn ⭐ 88,386
Node ⭐ 87,846
Next.js ⭐ 87,168
Terminal ⭐ 83,273
Deno ⭐ 82,611
Three.js ⭐ 82,243
Angular ⭐ 81,542
Awesome Go ⭐ 81,089
Typescript ⭐ 80,904
Ant Design ⭐ 80,428
Material Ui ⭐ 78,650
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.