Awesome Open Source

Programming Languages

Search results for database etl

74 search results found

Tidb ⭐ 35,604

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://tidbcloud.com/free-trial

Doris ⭐ 13,039

Apache Doris is an easy-to-use, high performance and unified analytics database.

Airbyte ⭐ 12,918

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

Risingwave ⭐ 6,701

Continuous SQL for event streams, database CDC, and time series. Perform streaming analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch processing. PostgreSQL compatible.

Awesome Business Intelligence ⭐ 1,862

Actively curated list of awesome BI tools. PRs welcome!

Addax ⭐ 1,034

Addax is a versatile open-source ETL tool that can seamlessly transfer data between various RDBMS and NoSQL databases, making it an ideal solution for data migration.

Pgsync ⭐ 1,003

Postgres to Elasticsearch/OpenSearch sync

Data Engineering Wiki ⭐ 934

The best place to learn data engineering. Built and maintained by the data engineering community.

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

Datacleaner ⭐ 557

The premier open source Data Quality solution

Etlalchemy ⭐ 414

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

Versatile Data Kit ⭐ 389

One framework to develop, deploy and operate data workflows with Python and SQL.

Replicadb ⭐ 304

ReplicaDB is open source tool for database replication, designed for efficiently transferring bulk data between relational and non-relational databases

Beginner_de_project ⭐ 276

Beginner data engineering project - batch edition

Elastic ⭐ 242

R client for the Elasticsearch HTTP API

Feldera ⭐ 199

Feldera Continuous Analytics Platform

Neo4j Etl ⭐ 168

Data import from relational databases to Neo4j.

Reddit Detective ⭐ 160

Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more

Morph Kgc ⭐ 151

Powerful RDF Knowledge Graph Generation with RML Mappings

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

The CSV to database command line loader

Forklift ⭐ 130

Forklift: Moving big databases around. A ruby ETL tool.

Marklogic Data Hub ⭐ 129

The MarkLogic Data Hub: documentation ==>

R package to facilitate ETL operations

Databazel ⭐ 116

The analytical and reporting solution for MongoDB

Nextract ⭐ 114

Nextract is a Extract Transform Load (ETL) platform build on top of Node.js streams

NBi is a testing framework (add-on to NUnit) for Business Intelligence and Data Access. The main goal of this framework is to let users create tests with a declarative approach based on an Xml syntax. By the means of NBi, you don't need to develop C# or Java code to specify your tests! Either, you don't need Visual Studio or Eclipse to compile your test suite. Just create an Xml file and let the framework interpret it and play your tests. The framework is designed as an add-on of NUnit but with

locopy: Loading/Unloading to Redshift and Snowflake using Python.

Python ETL(Extract-Transform-Load) tool / Data migration tool

Etlhelper ⭐ 81

ETL Helper is a Python ETL library to simplify data transfer into and out of databases.

Dataengineeringpilipinas ⭐ 80

Data Engineering Pilipinas is a community for data engineers, data analysts, data scientists, developers, AI / ML engineers, and users of closed and open source data tools and methods / techniques in the Philippines. Data Engineering Pilipinas is a PyData group.

Data Wrangling With Python ⭐ 66

Simplify your ETL processes with these hands-on data sanitation tips, tricks, and best practices

Blockchain Etl ⭐ 63

Blockchain follower that follows and stores the Helium blockchain

Apachespark ⭐ 59

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

Dbt Sqlite ⭐ 59

A SQLite adapter plugin for dbt (data build tool)

python ETL framework

Odoo Etl ⭐ 43

Odoo data manipulation, like an small ELT (Extract, Load, Transform) for odoo databases.

Etl Cdmbuilder ⭐ 35

ETL-CDMBuilder is a repo containing a .NET Core application to perform ETL to OMOP CDM for multiple databases

Pandas To Postgres ⭐ 33

Copy Pandas DataFrames and HDF5 files to PostgreSQL database

Mlbgameday ⭐ 32

Multi-core processing of 'Gameday' data from Major League Baseball Advanced Media. Additional tools to parallelize large data sets and write them to a database.

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

Datayoga ⭐ 27

streaming data pipeline platform

Sql_to_ibis ⭐ 25

A Python package that parses sql and converts it to ibis expressions

Fhir Pipe ⭐ 25

Populate FHIR-compliant objects using SQL databases and processing rules

Postgresql utilities for ETL and data analysis

Mongodb Rdbms Sync ⭐ 23

Bi-directional data sync between Relational Databases (Oracle, MySql) & MongoDB.

Iati Datastore ⭐ 23

An open-source datastore for IATI data with RESTful web API providing XML, JSON, CSV plus ETL tools

De 100 Days ⭐ 22

data engineering 100 days 🤖 🧲 🦾 | #DE

数据质量控制系统

ETL toolkit for 2.5 million electronic nonprofit tax returns released by the IRS.

Dataviva Etl ⭐ 21

Extract / Transform / Load Scripts for databases used in Dataviva Project

Etl_manager ⭐ 21

A python package to create a database on the platform using our moj data warehousing framework

Jira Database Etl ⭐ 20

🚹 💾 Script to import issues from a JIRA instance into a database.

Mysql Mongo Etl ⭐ 19

An ETL Node.js script to migrate MySQL data to MongoDB, mapping MySQL tables to MongoDB collections.

Jun_bigdata ⭐ 18

jun_bigdata大数据平台服务框架。实现了Kafka实时数据过滤、清洗、转换、消费，实现了Sp SQL对Redis、MongoDB等非关系型数据库的数据的读写；集成了规则引擎，可基于规则引擎实现客

Fec Loader ⭐ 18

Loads raw FEC filings into a database

Rivery_cli ⭐ 17

Airflowetl ⭐ 16

Blog post on ETL pipelines with Airflow

Postgresql To Mssql ⭐ 14

Migrate postgresql data to sql server on the fly!

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python

Sheetwork ⭐ 14

A handy package to load Google Sheets to your database right from the CLI and with easy configuration via YAML files.

Dappboard Documentation ⭐ 12

How to install, run and hack DAppBoard.

Oesophagus ⭐ 12

Enterprise Grade Single-Step Streaming Data Infrastructure Setup. (Under Development)

Data Brewery ⭐ 12

Data Brewery is an ETL (Extract-Transform-Load) program that connect to many data sources (cloud services, databases, ...) and manage data warehouse workflow.

Azure Data Factory ⭐ 11

Aprender Gerencimento de Dados ETL/ELT

Fastdata.core ⭐ 11

.net core orm(db first,code frist) for sqlserver mysql etl.

Statcastr ⭐ 10

ETL functionality for Statcast data

Cupcakesinc ⭐ 10

Sample "web store" for my ActiveWarehouse/ActiveWarehouse-ETL test

Transmart Batch ⭐ 10

tranSMART pipeline alternative to ETL, using Spring Batch

Devestates ⭐ 10

Database for visualizing and understanding the impacts of declared disasters on real estate prices

Extract transform load CLI tool for extracting small and middle data volume from sources (databases, csv files, xls files, gspreadsheets) to target (databases, csv files, xls files, gspreadsheets) in free combination.

ETL gem for extracting, transforming and importing data into Greenplum

Data Pipeline With Dbt Using Airflow On Gcp ⭐ 10

This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. There are different tools that have been used in this project such as Astro, DBT, GCP, Airflow, Metabase.

Scribe Data ⭐ 9

Wikidata and Wikipedia data extraction for Scribe applications

Data Engineering ⭐ 9

This is an all-in-one repository for Data Engineers, ideal for beginners & interview preparation, which includes Python as the main Programing language incorporating MySQL, MongoDB and Docker

llamas, but trying with enums for dynamic dispatch

Bigdatawarehouse ⭐ 8

Lobbyfacts ⭐ 7

[REPLACED BY pudo/openinterests.eu]

This is an ETL application on AWS with general open sales and customer data that you can find here: https://github.com/camposvinicius/data/blob/main/A it's a zipped file with some .csvs inside that we will apply transformations.

Wiredflow ⭐ 7

Lightweight library for creating services using just Python

Data Vault data model and ETL generator for Oracle Databases

Source Watcher Core ⭐ 7

This is a PHP project which combines ETL with different strategies to extract data from multiple databases, files, and services, transform it and load it into multiple destinations.

Bigquery Sqlalchemy Tutorial ⭐ 7

📊 ➡️ 💾 ETL script to migrate data from BigQuery to SQL.

Ferc Xbrl Extractor ⭐ 7

A tool for converting FERC filings published in XBRL into SQLite databases

Spark Etl Demo ⭐ 7

Demo of an ETL Spark Job

General ETL Machine, a customizable ETL framework built in Pentaho Data Integration (Kettle)

Teleporter ⭐ 6

Automatically synchronizes any database in RDBMS to OrientDB database. Open Source Project - Apache 2 license.

Reportgenerator ⭐ 6

A small cross-database tool for building excel documents (reports) based on data from database that extacts via View or Stored Procedures with parametres, ordering e.t.c.

An R package for creating a Retrosheet database using the ETL framework

Mongo Transporter ⭐ 5

Sync data between two MongoDB deployments

Urbandev Etl ⭐ 5

Extract, Transform, and Load processes for Urban Development

Doris Sdk ⭐ 5

SDK for Apache Doris

Related Searches

Command Line Database (33,932)

Python Database (10,521)

Javascript Database (9,210)

Php Database (5,990)

Database Mysql (4,382)

Java Database (4,190)

Database Sql (3,816)

Docker Database (3,195)

Golang Database (2,926)

Database Sqlite (2,819)

1-74 of 74 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2025 Awesome Open Source. All rights reserved.