Awesome Open Source

Programming Languages

Search results for sql etl

120 search results found

Tidb ⭐ 35,604

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://tidbcloud.com/free-trial

Doris ⭐ 13,039

Apache Doris is an easy-to-use, high performance and unified analytics database.

Risingwave ⭐ 6,701

Continuous SQL for event streams, database CDC, and time series. Perform streaming analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch processing. PostgreSQL compatible.

Mage Ai ⭐ 6,324

🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.

Steampipe ⭐ 6,061

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

Cloudquery ⭐ 5,380

The open source high performance data integration platform built for developers.

Quadratic ⭐ 2,485

Quadratic | Data Science Spreadsheet with Python & SQL

Awesome Business Intelligence ⭐ 1,862

Actively curated list of awesome BI tools. PRs welcome!

Dozer ⭐ 1,367

Dozer is a real-time data platform for building, deploying and maintaining data products.

Peerdb ⭐ 1,315

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

Pgsync ⭐ 1,003

Postgres to Elasticsearch/OpenSearch sync

Data Engineering Wiki ⭐ 934

The best place to learn data engineering. Built and maintained by the data engineering community.

Sqlmesh ⭐ 931

SQLMesh is a data transformation framework that brings the benefits of DevOps to data teams. It enables data scientists, analysts, and engineers to efficiently run and deploy data transformations written in SQL or Python.

Dataform ⭐ 757

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

Metorikku ⭐ 536

A simplified, lightweight ETL Framework based on Apache Spark

Automate Dv ⭐ 456

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)

Etlalchemy ⭐ 414

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

Versatile Data Kit ⭐ 389

One framework to develop, deploy and operate data workflows with Python and SQL.

Extract, Transform, and Load data with Ruby

Big_data_architect_skills ⭐ 353

一个大数据架构师应该掌握的技能

Webkettle ⭐ 350

基于web版kettle开发的一套分布式综合调度,管理,ETL开发的用户专业版B/S架构工具

Replicadb ⭐ 304

ReplicaDB is open source tool for database replication, designed for efficiently transferring bulk data between relational and non-relational databases

Astro Sdk ⭐ 303

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Cuelake ⭐ 266

Use SQL to build ELT pipelines on a data lakehouse.

Bulk Writer ⭐ 218

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

Feldera ⭐ 199

Feldera Continuous Analytics Platform

Dbt Coves ⭐ 193

CLI tool for dbt users to simplify creation of staging models (yml and sql) files

Mara Example Project 2 ⭐ 174

An example mini data warehouse for python project stats, template for new projects

Neo4j Etl ⭐ 168

Data import from relational databases to Neo4j.

Steampipe Plugin Aws ⭐ 165

Use SQL to instantly query AWS resources across regions and accounts. Open source CLI. No DB required.

Dbt Databricks ⭐ 165

A dbt adapter for Databricks.

Analytics Readings ⭐ 155

Readings for Analytics Engineers

Ethereum Etl Postgres ⭐ 143

ETL for moving Ethereum data to PostgreSQL database

Easy_sql ⭐ 126

A library developed to ease the data ETL development process.

R package to facilitate ETL operations

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Patterns Devkit ⭐ 101

Data pipelines from re-usable components

locopy: Loading/Unloading to Redshift and Snowflake using Python.

Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pipelines.

go-etl是一个集数据源抽取，转化，加载的工具集，提供强大的离线数据同步能力。

Etlhelper ⭐ 81

ETL Helper is a Python ETL library to simplify data transfer into and out of databases.

Dataengineeringpilipinas ⭐ 80

Data Engineering Pilipinas is a community for data engineers, data analysts, data scientists, developers, AI / ML engineers, and users of closed and open source data tools and methods / techniques in the Philippines. Data Engineering Pilipinas is a PyData group.

Projects ⭐ 76

Sample projects using Ploomber.

Sqlbucket ⭐ 67

Lightweight library to write, orchestrate and test your SQL ETL. Writing ETL with data integrity in mind.

Beneath is a serverless real-time data platform ⚡️

Apachespark ⭐ 59

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

A tool for developing and testing ETL and ELT processes for automating the capture, delivery and processing of information in data warehouses on the MicroFocus Vertica platform.

Steampipe Plugin Kubernetes ⭐ 41

Use SQL to instantly query Kubernetes API resources. Open source CLI. No DB required.

Steampipe Sqlite ⭐ 39

Steampipe SQLite is a zero-ETL engine for SQLite. Virtual tables translate queries into live API calls for cloud services and APIs. Hundreds of plugins with thousands of documented examples.

Csv Cruncher ⭐ 38

Treats CSV and JSON files as SQL tables, and exports SQL SELECTs back to CSV or JSON.

Projeto_etl_rfb_ibge_anp ⭐ 38

PYTHON E POSTGRESQL - EXTRACT TRANSFORM LOAD - ETL - DADOS PÚBLICOS DA RECEITA FEDERAL DO BRASIL - RFB, INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA - IBGE E AGÊNCIA NACIONAL DO PETRÓLEO, GÁS NATURAL E BIOCOMBUSTÍVEIS - ANP - PYTHON E POSTGRESQL

Apache Spark ETL Utilities

Steampipe Plugin Gcp ⭐ 36

Use SQL to instantly query GCP resources across regions, projects and organizations. Open source CLI. No DB required.

Sharpetl ⭐ 36

Write ETL using your favorite SQL dialects

multiprocessing enabled out-of-memory data analysis library for tabular data.

Ether_sql ⭐ 35

A python library to push ethereum blockchain data into an sql database.

Etl Cdmbuilder ⭐ 35

ETL-CDMBuilder is a repo containing a .NET Core application to perform ETL to OMOP CDM for multiple databases

Mara Etl Tools ⭐ 33

Utilities for creating ETL pipelines with mara

智能数据探索服务(Intelligent Data Exploration Service)，一站式Data + AI数据解决方案！

Blast is a data orchestration tool that can run SQL and Python against Google BigQuery and Snowflake. It supports templating with Jinja, data quality tests, query validation, environment management and more.

Steampipe Plugin Azure ⭐ 30

Use SQL to instantly query Azure resources across regions and subscriptions. Open source CLI. No DB required.

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

Datayoga ⭐ 27

streaming data pipeline platform

Steampipe Plugin Terraform ⭐ 27

Use SQL to instantly query resources, data sources and more from Terraform code. Open source CLI. No DB required.

Apache AVRO for go

Awesome Vertica ⭐ 26

A curated list of awesome Vertica libraries, tools and resources

Sql_to_ibis ⭐ 25

A Python package that parses sql and converts it to ibis expressions

Fhir Pipe ⭐ 25

Populate FHIR-compliant objects using SQL databases and processing rules

Steampipe Plugin Sdk ⭐ 25

Steampipe Plugin SDK is a simple abstraction layer to write a Steampipe plugin. Plugins automatically work across all engine types including the Steampipe CLI, Postgres FDW, SQLite extension and the export CLI.

Mongodb Rdbms Sync ⭐ 23

Bi-directional data sync between Relational Databases (Oracle, MySql) & MongoDB.

Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.

Sqloogle ⭐ 21

Crawl, Index, and Search Your SQL.

Steampipe Plugin Net ⭐ 20

Use SQL to instantly query DNS records, certificates and other network information. Open source CLI. No DB required.

Steampipe Plugin Oci ⭐ 18

Use SQL to instantly query Oracle Cloud resources across regions and accounts. Open source CLI. No DB required.

Steampipe Plugin Csv ⭐ 18

Use SQL to instantly query data from CSV files. Open source CLI. No DB required.

Sync Engine Example ⭐ 17

Synchronization Algorithm Exploration: Techniques to synchronize a SQL database with external destinations.

Helium Etl Queries ⭐ 17

A collection of SQL views used to enrich data produced by a Helium blockchain-etl

Sql To Redis ⭐ 17

Convert SQL tables to Redis as JSON

Airflowetl ⭐ 16

Blog post on ETL pipelines with Airflow

Spark Etl ⭐ 15

Set of ETL utils for Spark

Automated_etl_google_cloud Social_dashboard ⭐ 15

A dashboard is worth a thousand words => https://datastudio.google.com/reporting/755f3183-d

Postgresql To Mssql ⭐ 14

Migrate postgresql data to sql server on the fly!

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python

Steampipe Plugin Code ⭐ 14

Use SQL to instantly query secrets and more from source code. Open source CLI. No DB required.

Sheetwork ⭐ 14

A handy package to load Google Sheets to your database right from the CLI and with easy configuration via YAML files.

Datalink ⭐ 13

简单易用的ETL工具

Bootcamp Igti Analista De Dados ⭐ 13

Bootcamp online analista de dados disponibilizado pelo IGTI – Instituto de Gestão e Tecnologia da Informação

Steampipe Plugin Twitter ⭐ 13

Use SQL to instantly query tweets, users and followers from Twitter. Open source CLI. No DB required.

Steampipe Plugin Googleworkspace ⭐ 13

Use SQL to instantly query calendar events, drive files, gmail messages, and more from Google Workspace. Open source CLI. No DB required.

Steampipe Plugin Scaleway ⭐ 12

Use SQL to instantly query instances, networks, databases, and more from Scaleway. Open source CLI. No DB required.

Tezos Etl ⭐ 12

Python scripts for ETL (extract, transform and load) jobs for Tezos blocks, balance updates, and operations

Hands On Data Warehousing With Azure Data Factory ⭐ 12

Hands-On Data Warehousing with Azure Data Factory, published by Packt

Data Portfolio ⭐ 11

📊 ⚙️ My professional data analysis portfolio. Check out my works by clicking the link.

Fansisql ⭐ 11

C# library for abstracting the DBMS layer away in ETL applications. Supports table discovery, table creation, bulk insert and type translation for Sql Server, Oracle, PostgresSql and MySql. FAnsiSql is not an ORM its an ETL enabler.

Bigdata Etl Pipeline ⭐ 10

The Data Pipeline and Analytics Stack is a comprehensive solution designed for processing, storing, and visualizing data. Explore a complete data pipeline with all components seamlessly set up and ready to use

A project for exploring how Great Expectations can be used to ensure data quality and validate batches within a data pipeline defined in Airflow.

PHP7.1 Fluent, immutable SQL query builder.

Data Pipeline With Dbt Using Airflow On Gcp ⭐ 10

This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. There are different tools that have been used in this project such as Astro, DBT, GCP, Airflow, Metabase.

Data Engineering ⭐ 9

This is an all-in-one repository for Data Engineers, ideal for beginners & interview preparation, which includes Python as the main Programing language incorporating MySQL, MongoDB and Docker

A declarative, SQL-like DSL for data integration tasks.

Related Searches

Database Sql (5,501)

Python Sql (3,922)

Mysql Sql (2,867)

Java Sql (2,781)

Javascript Sql (2,662)

C Sharp Sql (2,429)

Postgresql Sql (2,411)

Php Sql (2,276)

Golang Sql (1,383)

Sql Table (1,358)

1-100 of 120 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2025 Awesome Open Source. All rights reserved.