Awesome Open Source

Programming Languages

The Top 23 Spark Data Lake Open Source Projects

Open source projects categorized as Spark Data Lake

Categories > Data Lake

Categories > Data Processing > Spark

Upserts, Deletes And Incremental Processing on Big Data.

dependent packages 58total releases 21most recent commit 3 months ago

Lakesoul ⭐ 2,248

LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.

dependent packages 1total releases 7most recent commit 3 months ago

Kyuubi ⭐ 1,849

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

dependent packages 24total releases 11most recent commit 3 months ago

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

dependent packages 15total releases 10most recent commit a year ago

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

total releases 1most recent commit 3 months ago

Goodreads_etl_pipeline ⭐ 593

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

most recent commit 4 years ago

Marmaray ⭐ 444

Generic Data Ingestion & Dispersal Library for Hadoop

most recent commit 2 years ago

Data Engineering Projects ⭐ 322

Personal Data Engineering Projects

most recent commit a year ago

Smart Data Lake ⭐ 87

Smart Automation Tool for building modern Data Lakes and Data Pipelines

dependent packages 8total releases 26most recent commit 3 months ago

Apachespark ⭐ 59

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

most recent commit 2 years ago

Lighthouse ⭐ 54

Lighthouse is a library for data lakes built on top of Apache Spark. It provides high-level APIs in Scala to streamline data pipelines and apply best practices.

dependent packages 3total releases 10most recent commit 3 years ago

Anyscale ⭐ 49

anyscale roadmap

most recent commit 3 years ago

Datapipelines Essentials Python ⭐ 45

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

most recent commit a year ago

Real Time Data Warehouse ⭐ 29

Real-time Data Warehouse with Apache Flink & Apache Kafka & Apache Hudi

most recent commit 2 years ago

Enceladus ⭐ 28

Dynamic Conformance Engine

dependent packages 4total releases 40most recent commit 4 months ago

Data Engineer Nanodegree Projects Udacity ⭐ 27

Projects done in the Data Engineer Nanodegree Program by Udacity.com

most recent commit 2 years ago

Jobanalytics_and_search ⭐ 22

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

most recent commit 2 years ago

Sparkprogramminginscala ⭐ 18

Apache Spark Course Material

most recent commit 4 years ago

Data Mill ⭐ 16

A K8s-based infrastructure for analytics

most recent commit 4 years ago

Data Pipeline from the Global Historical Climatology Network DataSet

most recent commit a year ago

Kyuubi Docker ⭐ 9

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

most recent commit 4 months ago

Awesome Data Pipeline ⭐ 6

Awesome list for datapipeline

most recent commit a year ago

Bigdata Platform ⭐ 6

End to end big data project, that aims to show how to implement different big data layers, from the infrastructure layer to the end user one. [HADOOP][Spark][Kafka][Cassandra][Ansible][Jupyter

most recent commit 4 months ago

The 7 Most Depended On Spark Data Lake Open Source Projects

Upserts, Deletes And Incremental Processing on Big Data.

dependent packages 58total releases 21latest release November 11, 2023most recent commit 3 months ago

Kyuubi ⭐ 1,849

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

dependent packages 24total releases 11latest release November 03, 2023most recent commit 3 months ago

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

dependent packages 15total releases 10latest release December 05, 2018most recent commit a year ago

Smart Data Lake ⭐ 87

Smart Automation Tool for building modern Data Lakes and Data Pipelines

dependent packages 8total releases 26latest release October 25, 2023most recent commit 3 months ago

Enceladus ⭐ 28

Dynamic Conformance Engine

dependent packages 4total releases 40latest release March 23, 2023most recent commit 4 months ago

Lighthouse ⭐ 54

Lighthouse is a library for data lakes built on top of Apache Spark. It provides high-level APIs in Scala to streamline data pipelines and apply best practices.

dependent packages 3total releases 10latest release August 21, 2020most recent commit 3 years ago

Lakesoul ⭐ 2,248

LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.

dependent packages 1total releases 7latest release October 12, 2023most recent commit 3 months ago

The 8 Latest Releases In Spark Data Lake Open Source Projects

Upserts, Deletes And Incremental Processing on Big Data.

dependent packages 58total releases 21latest release November 11, 2023most recent commit 3 months ago

Kyuubi ⭐ 1,849

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

dependent packages 24total releases 11latest release November 03, 2023most recent commit 3 months ago

Smart Data Lake ⭐ 87

Smart Automation Tool for building modern Data Lakes and Data Pipelines

dependent packages 8total releases 26latest release October 25, 2023most recent commit 3 months ago

Lakesoul ⭐ 2,248

LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.

dependent packages 1total releases 7latest release October 12, 2023most recent commit 3 months ago

Enceladus ⭐ 28

Dynamic Conformance Engine

dependent packages 4total releases 40latest release March 23, 2023most recent commit 4 months ago

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

total releases 1latest release June 01, 2022most recent commit 3 months ago

Lighthouse ⭐ 54

Lighthouse is a library for data lakes built on top of Apache Spark. It provides high-level APIs in Scala to streamline data pipelines and apply best practices.

dependent packages 3total releases 10latest release August 21, 2020most recent commit 3 years ago

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

dependent packages 15total releases 10latest release December 05, 2018most recent commit a year ago

Categories

Application Programming Interfaces

Artificial Intelligence

Cloud Computing

Command Line Interface

Computer Science

Configuration Management

Content Management

Data Processing

Integrated Development Environments

Learning Resources

Lists Of Projects

Machine Learning

Operating Systems

Package Managers

Programming Languages

Runtime Environments

Software Architecture

Software Development

Software Performance

Software Quality

Text Processing

User Interface Components

Version Control

Web User Interface

Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source. All rights reserved.