Awesome Open Source

Programming Languages

Search results for etl

964 search results found

Usaspending Api ⭐ 273

Server application to serve U.S. federal spending data via a RESTful API

Storagetapper ⭐ 269

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

Butterfree ⭐ 269

A tool for building feature stores.

Sync data from the other DB to ClickHouse(cluster)

Flock: A Low-Cost Streaming Query Engine on FaaS Platforms

Low-code Python library to safely use notebooks in production: schedule workflows, generate assets, trigger webhooks, send notifications, build pipelines, manage secrets (Cloud-only)

Cuelake ⭐ 266

Use SQL to build ELT pipelines on a data lakehouse.

Orbital ⭐ 255

Orbital automates integration between data sources (APIs, Databases, Queues and Functions). BFF's, API Composition and ETL pipelines that adapt as your specs change.

Data Making Guidelines ⭐ 248

📘 Making Data, the DataMade Way

Node Datapumps ⭐ 247

Node.js ETL (Extract, Transform, Load) toolkit for easy data import, export or transfer between systems.

Elastic ⭐ 242

R client for the Elasticsearch HTTP API

Substation ⭐ 242

Substation is a cloud-native, event-driven data pipeline toolkit built for security teams.

Activewarehouse Etl ⭐ 238

Extract-Transform-Load library from ActiveWarehouse

Paperetl ⭐ 235

📄 ⚙️ ETL processes for medical and scientific papers

Extract ⭐ 229

A cross-platform command line tool for parallelised content extraction and analysis.

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

Bulk Writer ⭐ 218

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

Bigquery Etl ⭐ 216

Blazing-fast Expression Templates Library (ETL) with GPU support, in C++

Rhino Etl ⭐ 206

Main repository is here ->

Example Airflow Dags ⭐ 204

Example DAGs using hooks and operators from Airflow Plugins

Node Etl ⭐ 204

npm install etl

Feldera ⭐ 199

Feldera Continuous Analytics Platform

Activewarehouse ⭐ 199

ActiveWarehouse for Rails - Implement data warehouses with Rails

Kettle Web ⭐ 197

基于spring boot通过java代码调用kette

A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.

Amundsendatabuilder ⭐ 196

Data ingestion library for Amundsen to build graph and search index

Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at www.jumpmind.com/products/metl/overview

Dbt Coves ⭐ 193

CLI tool for dbt users to simplify creation of staging models (yml and sql) files

Dataflowex ⭐ 190

A .NET dataflow and etl framework built upon Microsoft TPL Dataflow library

Mongo Es ⭐ 189

A MongoDB to Elasticsearch connector

Bender - Serverless ETL Framework

Etl Language Comparison ⭐ 185

Count the number of times certain words were said in a particular neighborhood. Performed as a basic MapReduce job against 25M tweets. Implemented with different programming languages as a educational exercise.

Aws Etl Orchestrator ⭐ 185

A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

Grafter ⭐ 183

Linked Data & RDF Manufacturing Tools in Clojure

Intelligently transform unstructured to structured data

Mara Example Project 2 ⭐ 175

An example mini data warehouse for python project stats, template for new projects

A simple Spark-powered ETL framework that just works 🍺

Dataplane ⭐ 171

Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

Neo4j Etl ⭐ 168

Data import from relational databases to Neo4j.

Data Story ⭐ 167

A visual process builder

Airflow_for_beginners ⭐ 166

Steampipe Plugin Aws ⭐ 165

Use SQL to instantly query AWS resources across regions and accounts. Open source CLI. No DB required.

Dbt Databricks ⭐ 165

A dbt adapter for Databricks.

Reddit Detective ⭐ 160

Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more

Aliyun Log Python Sdk ⭐ 159

Use python to manage, produce and consume data with Aliyun Log Service.

Whiterabbit ⭐ 158

WhiteRabbit is a small application that can be used to analyse the structure and contents of a database as preparation for designing an ETL. It comes with RabbitInAHat, an application for interactive design of an ETL to the OMOP Common Data Model with the help of the the scan report generated by White Rabbit.

Etl_unicorn ⭐ 156

数据可视化, 数据挖掘, 数据处理 ETL

Analytics Readings ⭐ 155

Readings for Analytics Engineers

Meilisync ⭐ 154

Realtime sync data from MySQL/PostgreSQL/MongoDB to Meilisearch

Transformalize ⭐ 153

Configurable Extract, Transform, and Load

Morph Kgc ⭐ 151

Powerful RDF Knowledge Graph Generation with RML Mappings

Graphar ⭐ 145

An open source, standard data file format for graph data storage and retrieval

Eel Sdk ⭐ 140

Big Data Toolkit for the JVM

Hydrograph ⭐ 138

A visual ETL development and debugging tool for big data

Ethereum Etl Postgres ⭐ 136

ETL for moving Ethereum data to PostgreSQL database

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

LinkedPipes ETL is an RDF based, lightweight ETL tool

The CSV to database command line loader

A COBOL parser and Mainframe/EBCDIC data source for Apache Spark

Česká otevřená data

Forklift ⭐ 130

Forklift: Moving big databases around. A ruby ETL tool.

Marklogic Data Hub ⭐ 129

The MarkLogic Data Hub: documentation ==>

Sync Addons ⭐ 129

Odoo Integration Modules

Easy_sql ⭐ 126

A library developed to ease the data ETL development process.

Php Etl ⭐ 125

Extract, Transform and Load data using PHP.

R package to facilitate ETL operations

Freebase Triples ⭐ 123

A methodology to process triples data from the Freebase data dumps.

Sentinel Crawler ⭐ 122

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器，分布式爬虫

Datagristle ⭐ 122

Tough and flexible tools for data analysis, transformation, validation and movement.

Neo4j Jdbc ⭐ 122

JDBC driver for Neo4j

Diffsync ⭐ 121

A utility library for comparing and synchronizing different datasets.

A Go native tabular data extraction package. Currently supports .xls, .xlsx, .csv, .tsv formats.

Empujar ⭐ 120

When you need to push data around, you push it. A node.js ETL tool.

Sequenceiq Samples ⭐ 119

SequenceIQ Hadoop examples

Datachecks ⭐ 117

Open Source Data Quality Monitoring.

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Databazel ⭐ 116

The analytical and reporting solution for MongoDB

Nextract ⭐ 114

Nextract is a Extract Transform Load (ETL) platform build on top of Node.js streams

Get customer-permissioned financial data in minutes with extensible, drop-in data connectors. Your customers & engineers will thank you.

Aws Ecs Airflow ⭐ 110

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

Pangeo Forge Recipes ⭐ 108

Python library for building Pangeo Forge recipes.

Fhir Data Pipes ⭐ 107

A collection of tools for extracting FHIR resources and analytics services on top of that data.

Deta_cache ⭐ 106

缓存cache服务器

Dataxserver ⭐ 106

为DataX(https://github.com/alibaba/DataX) 提供远程多语言调用（ThriftServer，HttpServer）分布式运行（DataX on YARN）功能

NBi is a testing framework (add-on to NUnit) for Business Intelligence and Data Access. The main goal of this framework is to let users create tests with a declarative approach based on an Xml syntax. By the means of NBi, you don't need to develop C# or Java code to specify your tests! Either, you don't need Visual Studio or Eclipse to compile your test suite. Just create an Xml file and let the framework interpret it and play your tests. The framework is designed as an add-on of NUnit but with

Peaks Consolidation ⭐ 102

The Peaks Consolidation is equipped with state-of-the-art algorithms and data structures that support high-performance databending exercises. It specializes in management accounting and consolidation, with some special topics in machine learning and bioinformatics.

Big Data ETL and Utilities for Hadoop Map Reduce, Spark and Storm

Kafka Connect ⭐ 102

equivalent to kafka-connect 🔧 for nodejs ✨🐢🚀✨

Patterns Devkit ⭐ 101

Data pipelines from re-usable components

locopy: Loading/Unloading to Redshift and Snowflake using Python.

Batch Scheduler ⭐ 96

Polygon Etl ⭐ 93

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Python ETL(Extract-Transform-Load) tool / Data migration tool

Deployml_course ⭐ 93

Репозиторий для открытого курса «Промышленная эксплуатация моделей машинного обучения»

Target Postgres ⭐ 93

A Singer.io Target for Postgres

Chronicle Etl ⭐ 92

📜 A CLI toolkit for extracting and working with your digital history

Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)

101-200 of 964 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.