Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 19 Etl Apache Spark Open Source Projects

Open source projects categorized as Etl Apache Spark
Categories > Data Processing > Apache Spark
Goodreads_etl_pipeline ⭐ 593
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
most recent commit 4 years ago
Cuelake ⭐ 266
Use SQL to build ELT pipelines on a data lakehouse.
most recent commit 2 years ago
Hydrograph ⭐ 138
A visual ETL development and debugging tool for big data
most recent commit 4 years ago
Cobrix ⭐ 131
A COBOL parser and Mainframe/EBCDIC data source for Apache Spark
dependent packages 2total releases 43most recent commit 3 months ago
Flowman ⭐ 85
Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pipelines.
dependent packages 24total releases 65most recent commit 5 months ago
Spark ⭐ 65
Open Source D-APM (Data-Application Performance Monitoring) for Apache Spark
most recent commit 3 months ago
Spark Etl ⭐ 62
Apache Spark based ETL Engine
most recent commit 8 years ago
Apachespark ⭐ 59
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
most recent commit 2 years ago
Datapipelines Essentials Python ⭐ 45
Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations
most recent commit a year ago
Spark Ref Architecture ⭐ 38
Reference Architectures for Apache Spark
most recent commit 7 years ago
Daflow ⭐ 24
Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.
most recent commit 4 years ago
Sql Based Etl With Apache Spark On Amazon Eks ⭐ 23
A solution that provides declarative data processing capability, and workflow orchestration automation to help your business users (such as analysts and data scientists) access their data and create meaningful insights without the need for manual IT processes.
most recent commit 3 months ago
Sparklanes ⭐ 16
A lightweight data processing framework for Apache Spark
total releases 5most recent commit 4 years ago
Datacooker Etl ⭐ 10
Data transformation framework for ETL processing with SQL-like syntax and GIS extensions, based on Apache Spark
most recent commit 3 months ago
Apache Spark Etl Pipeline Example ⭐ 8
Demonstration of using Apache Spark to build robust ETL pipelines while taking advantage of open source, general purpose cluster computing.
most recent commit 6 years ago
Spark Etl Framework ⭐ 7
A generic ETL framework with Spark_SQL for transforming data by constructing pipelines with Yaml/Json/Xml
most recent commit 8 months ago
Spark Databricks ⭐ 6
🔥 Master Apache Spark & Databricks! Dive into a world of big data with exclusive insights from Udemy courses, personal notes, and practical guides. Whether you're starting out or scaling new heights in data engineering, this is your ultimate resource hub! 🌟🚀
most recent commit 4 months ago
Stock_streaming_pipeline_project ⭐ 5
Built a real-time streaming pipeline to extract stock data, using Apache Nifi, Debezium, Kafka, and Spark Streaming. Loaded the transformed data into Glue database and created real-time dashboards using Power BI and Tableau with Athena. The pipeline is orchestrated using Airflow.
most recent commit 7 months ago
Spark Structured Streaming Kafka ⭐ 5
Spark Structured Streaming + Kafka + Delta pipeline.
most recent commit 8 months ago
The 2 Most Depended On Etl Apache Spark Open Source Projects
Cobrix ⭐ 131
A COBOL parser and Mainframe/EBCDIC data source for Apache Spark
dependent packages 2total releases 43latest release October 16, 2023most recent commit 3 months ago
The 3 Latest Releases In Etl Apache Spark Open Source Projects
Cobrix ⭐ 131
A COBOL parser and Mainframe/EBCDIC data source for Apache Spark
dependent packages 2total releases 43latest release October 16, 2023most recent commit 3 months ago
Sparklanes ⭐ 16
A lightweight data processing framework for Apache Spark
total releases 5latest release January 31, 2019most recent commit 4 years ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.