Awesome Open Source

Programming Languages

Search results for python etl

330 search results found

Projeto_etl_rfb_ibge_anp ⭐ 38

PYTHON E POSTGRESQL - EXTRACT TRANSFORM LOAD - ETL - DADOS PÚBLICOS DA RECEITA FEDERAL DO BRASIL - RFB, INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA - IBGE E AGÊNCIA NACIONAL DO PETRÓLEO, GÁS NATURAL E BIOCOMBUSTÍVEIS - ANP - PYTHON E POSTGRESQL

Data transformation framework for LinkML data models

A simple and out-of-box toolkit to handle data work

Wikirepo ⭐ 36

Python based Wikidata framework for easy dataframe extraction

multiprocessing enabled out-of-memory data analysis library for tabular data.

Ether_sql ⭐ 35

A python library to push ethereum blockchain data into an sql database.

Fhirpack ⭐ 34

FHIR Python Analysis Client and Kit (FHIRPACK) is a general purpose FHIR client that simplifies the access, analysis and representation of FHIR and EHR data using PANDAS, an ETL philosophy and a functional syntax. It was initially developed at the IKIM and HDDBS in Germany. Read more at https://zenodo.org/record/8006589

Pandas To Postgres ⭐ 33

Copy Pandas DataFrames and HDF5 files to PostgreSQL database

A Python client for interacting with Knack applications

Dagster Polars ⭐ 32

Polars integration for Dagster

Write data & AI pipelines in (SQL, Spark, Pandas) and deploy to the cloud, simplified

data wrangling simplicity, complete audit transparency, and at speed

Hive Metastore Client ⭐ 32

A client for connecting and running DDLs on hive metastore.

Blast is a data orchestration tool that can run SQL and Python against Google BigQuery and Snowflake. It supports templating with Jinja, data quality tests, query validation, environment management and more.

Stellar Etl Airflow ⭐ 31

Airflow DAGs for the Stellar ETL project

Business intelligence for software development by python.

Gluestick ⭐ 29

A small Python module containing quick utility functions for standard ETL processes.

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

News_scrapy_redis ⭐ 28

Cookiecutter R Project ⭐ 28

Basic cookiecutter template for R projects

Utilities for data cleaning and ETL processing

Datayoga ⭐ 27

streaming data pipeline platform

Covalent Slurm Plugin ⭐ 26

Executor plugin interfacing Covalent with Slurm

Aws Auto Terminate Idle Emr ⭐ 26

AWS Auto Terminate Idle AWS EMR Clusters Framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.

Sample_etl_structure ⭐ 26

Forklift ⭐ 26

🚜📦✨ Slinging data all over the place 🚜📦✨

Python_mozetl ⭐ 26

ETL jobs for Firefox Telemetry

Alto is a versatile data integration tool that allows you to easily run Singer plugins, build and cache PEX files encapsulating those plugins, and create a data reservoir whereby you can extract once and replay to as many destinations as you want.

Fhir Pipe ⭐ 25

Populate FHIR-compliant objects using SQL databases and processing rules

Health Graph ⭐ 25

Graph of health and pharm data.

Sql_to_ibis ⭐ 25

A Python package that parses sql and converts it to ibis expressions

Arthur Redshift Etl ⭐ 25

ELT Code for your Data Warehouse

Yandex Tracker Exporter ⭐ 24

ETL tool for Yandex.Tracker. Export, transform and load issue metadata, changelog and agile metrics to Clickhouse storage.

Tweetsolaping ⭐ 24

implementing an end-to-end tweets ETL/Analysis pipeline.

Airflow Jupyter Docker Compose ⭐ 24

Orchestration of data science and earth observation models in Apache Airflow, scale-up with Celery Executor, experiment with jupyter notebook using a docker containers composition

Python_tutorials ⭐ 24

Python Notes on IPython Notebook files.

Django Etl Sync ⭐ 24

Django ETL deriving rules from models and forms

Blockchain Etl Streaming ⭐ 23

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Nodestream ⭐ 23

A Fast, Declarative, and Extensible ETL Framework for Graph Databases.

Gamechanger Data ⭐ 23

GAMECHANGER aspires to be the Department’s trusted solution for evidence-based, data-driven decision-making across the universe of DoD requirements

Iati Datastore ⭐ 23

An open-source datastore for IATI data with RESTful web API providing XML, JSON, CSV plus ETL tools

Sql Based Etl With Apache Spark On Amazon Eks ⭐ 23

A solution that provides declarative data processing capability, and workflow orchestration automation to help your business users (such as analysts and data scientists) access their data and create meaningful insights without the need for manual IT processes.

Warn Scraper ⭐ 23

Command-line interface for downloading WARN Act notices of qualified plant closings and mass layoffs from state government websites

Postgresql utilities for ETL and data analysis

Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.

Whakapai ⭐ 22

Various Python Data Science Projects available in PyPi

Aws Glue Docker ⭐ 22

🐋 Docker image for AWS Glue Spark/Python

Airflow Snowflake ⭐ 22

Code to be contributed to the Apache Airflow (incubating) project for ETL workflow management for integrating with the Snowflake Data Warehouse.

Singer Runner ⭐ 22

A CLI and library to run Singer Taps and Targets

Forklift ⭐ 22

🚚 ETL for Spark and Airflow

Etl_manager ⭐ 21

A python package to create a database on the platform using our moj data warehousing framework

Spark Movies Etl ⭐ 21

Spark data pipeline that ingests and transforms movie ratings data.

Appendfeaturestolayer ⭐ 21

Processing plugin-based provider for QGIS 3 that adds an algorithm for appending/updating features from a source vector layer to an existing target vector layer.

Dataviva Etl ⭐ 21

Extract / Transform / Load Scripts for databases used in Dataviva Project

ETL toolkit for 2.5 million electronic nonprofit tax returns released by the IRS.

Target Redshift ⭐ 20

A Singer.io Target for Redshift

Taskflow ⭐ 20

An advanced yet simple system to run your background tasks and workflows

Aws_glue_etl_docker ⭐ 20

Helper library to run AWS Glue ETL scripts docker container for local testing of development in a Jupyter notebook

Ethereum2 Etl ⭐ 20

Python scripts for ETL (extract, transform and load) jobs for Ethereum 2.0 beacon blocks, attestations, deposits, slashings, validators, committees. Data is available in Google BigQuery

Jira Database Etl ⭐ 20

🚹 💾 Script to import issues from a JIRA instance into a database.

application framework for ETL(ELT) pipeline, process

Data Refinery ⭐ 19

Data transformation

⚗️ Instill Model contains components for AI model orchestration

Amazon S3 Step Functions Ingestion Orchestration ⭐ 19

Design pattern for orchestrating an incremental data ingestion pipeline using AWS Step Functions from an on premise location into an Amazon S3 datalake bucket

Airflow Tutorial ⭐ 19

Tutorial like code for how to deploy airflow using docker and how to use the DockerOperator.

Cardano Py ⭐ 18

Python3 lib and cli for operating a Cardano Passive Node and using the API's. (PRE-ALPHA)

Tableau Extraction ⭐ 18

📈➡️💾 A Flask application which extends Tableau to be used as an ETL tool.

Dolthub Etl Jobs ⭐ 17

ETL jobs that DoltHub maintained that load public data into DoltHub.

Rivery_cli ⭐ 17

Django Data Migration ⭐ 17

Data migration framework for Django that migrates legacy data into your new django app

Klaytn Etl ⭐ 17

Python scripts for ETL (extract, transform and load) jobs for Klaytn blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions.

Tap Airbyte Wrapper ⭐ 17

A Singer tap that wraps Airbyte sources allowing them to be consumed by Singer targets

Covalent Aws Plugins ⭐ 16

Executor plugins interfacing Covalent with various AWS compute platforms

Airflowetl ⭐ 16

Blog post on ETL pipelines with Airflow

Phila Airflow ⭐ 16

A data transformation package for deep learning with Autonomio, Keras and TensorFlow.

Sparklanes ⭐ 16

A lightweight data processing framework for Apache Spark

Covid News ⭐ 16

A data engineering personal project for applying some of my skills

Smartpipeline ⭐ 16

A framework for rapid development of robust data pipelines following a simple design pattern

This is the ETL lib package. It provides an API to munge and prepare JSON, TSV and other data using Apache Tika and JSON parsing/loading for ETL via Apache OODT (or other libs) into Apache Solr.

Airflow Provider Fivetran Async ⭐ 15

A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran

Airflowdatapipeline ⭐ 15

Example of an ETL Pipeline using Airflow

Automated_etl_google_cloud Social_dashboard ⭐ 15

A dashboard is worth a thousand words => https://datastudio.google.com/reporting/755f3183-d

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python

Archivekit ⭐ 14

ArchiveKit manages data and documents during ETL processes, either on a local file system or on S3.

Sheetwork ⭐ 14

A handy package to load Google Sheets to your database right from the CLI and with easy configuration via YAML files.

Etl Airflow S3 ⭐ 14

ETL of newspaper article keywords using Apache Airflow, Newspaper3k, Quilt T4 and AWS S3

Covid 19 ⭐ 14

Data ETL & Analysis on the global and Mexican datasets of the COVID-19 pandemic.

Atd Data Publishing ⭐ 13

Python scripts for Austin Transportation's ETL tasks

Docker Etl ⭐ 13

Collection of dockerized ETL jobs managed by data engineering.

Bootcamp Igti Analista De Dados ⭐ 13

Bootcamp online analista de dados disponibilizado pelo IGTI – Instituto de Gestão e Tecnologia da Informação

Jp Ocr Prunned Cnn ⭐ 13

Attempting feature map prunning on a CNN trained for Japanese OCR

Airflow Ml Prediction ⭐ 13

Running ECS task for ML prediction orchestrated by Airflow

Bigquery To Pubsub ⭐ 13

A tool for streaming time series data from a BigQuery table to a Pub/Sub topic

Flowmaster ⭐ 13

ETL flow framework based on Yaml configs in Python

Etl Master ⭐ 13

This is a pytorch implementation of our paper: "Towards Equivalent Transformation of User Preferences in Cross Domain Recommendation"

Extract Transform Load toolkit (python).

Data Tools ⭐ 12

Common Python tools and utilities for data engineering, ETL, Exploration, etc. made opensource and packaged, making it easy to use in any environment.

Mara Example Project 1 ⭐ 12

Runnable e-commerce mini data warehouse based on Python, PostgreSQL & Metabase, template for new projects

Related Searches

Python Jupyter Notebook (17,496)

Python Dataset (14,792)

Python Docker (14,113)

Python Machine Learning (14,099)

Python Command Line (12,663)

Python Database (10,521)

Python Artificial Intelligence (8,580)

Python Amazon Web Services (7,946)

Python Paper (6,550)

Python Pandas (6,193)

101-200 of 330 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.