Awesome Open Source

Programming Languages

The Top 23 Hadoop Apache Spark Open Source Projects

Open source projects categorized as Hadoop Apache Spark

Categories > Data Processing > Apache Spark

Categories > Data Processing > Hadoop

Spark ⭐ 37,661

Apache Spark - A unified analytics engine for large-scale data processing

dependent packages 939total releases 46most recent commit 3 months ago

Bigdl ⭐ 4,728

Accelerate LLM with low-bit (FP4 / INT4 / FP8 / INT8) optimizations using bigdl-llm

dependent packages 10total releases 16most recent commit 3 months ago

Docker Spark ⭐ 1,783

Apache Spark docker image

most recent commit 2 years ago

Dr Elephant ⭐ 1,301

Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark

most recent commit a year ago

Livy is an open source REST interface for interacting with Apache Spark from anywhere

dependent packages 5total releases 4most recent commit 4 years ago

Flintrock ⭐ 627

A command-line tool for launching Apache Spark clusters.

total releases 14most recent commit 5 months ago

Dist Keras ⭐ 611

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

total releases 2most recent commit 6 years ago

Data Lineage Tracking And Visualization Solution

dependent packages 6total releases 14most recent commit 5 months ago

Spark Jupyter Aws ⭐ 255

A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support

most recent commit 6 years ago

Sparkrdma ⭐ 191

RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark

most recent commit 5 years ago

Learning Hadoop And Spark ⭐ 160

Companion to Learning Hadoop and Learning Spark courses on Linked In Learning

most recent commit 6 months ago

Bigdata Playground ⭐ 154

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

most recent commit 5 years ago

Hdfs_fdw ⭐ 131

PostgreSQL foreign data wrapper for HDFS

most recent commit 9 months ago

Griffon Vm ⭐ 129

Griffon Data Science Virtual Machine

most recent commit 2 years ago

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

total releases 27most recent commit 10 months ago

Spark With Python ⭐ 98

Fundamentals of Spark with Python (using PySpark), code examples

most recent commit 4 years ago

Mongo Spark ⭐ 93

Example application on how to use mongo-hadoop connector with Spark

most recent commit 10 years ago

Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pipelines.

dependent packages 24total releases 65most recent commit 5 months ago

Pig on Apache Spark

most recent commit 9 years ago

Docker Spark ⭐ 77

🚢 Docker image for Apache Spark

most recent commit 4 years ago

Euphoria ⭐ 74

Euphoria is an open source Java API for creating unified big-data processing flows. It provides an engine independent programming model which can express both batch and stream transformations.

total releases 4most recent commit 4 years ago

Apachespark ⭐ 59

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

most recent commit 2 years ago

Serverless Spark Workshop ⭐ 56

Solution Accelerators for Serverless Spark on GCP, the industry's first auto-scaling and serverless Spark as a service

most recent commit 6 months ago

The 5 Most Depended On Hadoop Apache Spark Open Source Projects

Spark ⭐ 37,661

Apache Spark - A unified analytics engine for large-scale data processing

dependent packages 939total releases 46latest release May 09, 2021most recent commit 3 months ago

Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pipelines.

dependent packages 24total releases 65latest release October 16, 2023most recent commit 5 months ago

Bigdl ⭐ 4,728

Accelerate LLM with low-bit (FP4 / INT4 / FP8 / INT8) optimizations using bigdl-llm

dependent packages 10total releases 16latest release April 19, 2021most recent commit 3 months ago

Data Lineage Tracking And Visualization Solution

dependent packages 6total releases 14latest release July 25, 2019most recent commit 5 months ago

Livy is an open source REST interface for interacting with Apache Spark from anywhere

dependent packages 5total releases 4latest release April 12, 2017most recent commit 4 years ago

The 9 Latest Releases In Hadoop Apache Spark Open Source Projects

Flintrock ⭐ 627

A command-line tool for launching Apache Spark clusters.

total releases 14latest release November 27, 2023most recent commit 5 months ago

Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pipelines.

dependent packages 24total releases 65latest release October 16, 2023most recent commit 5 months ago

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

total releases 27latest release November 17, 2022most recent commit 10 months ago

Spark ⭐ 37,661

Apache Spark - A unified analytics engine for large-scale data processing

dependent packages 939total releases 46latest release May 09, 2021most recent commit 3 months ago

Bigdl ⭐ 4,728

Accelerate LLM with low-bit (FP4 / INT4 / FP8 / INT8) optimizations using bigdl-llm

dependent packages 10total releases 16latest release April 19, 2021most recent commit 3 months ago

Data Lineage Tracking And Visualization Solution

dependent packages 6total releases 14latest release July 25, 2019most recent commit 5 months ago

Euphoria ⭐ 74

Euphoria is an open source Java API for creating unified big-data processing flows. It provides an engine independent programming model which can express both batch and stream transformations.

total releases 4latest release January 20, 2018most recent commit 4 years ago

Dist Keras ⭐ 611

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

total releases 2latest release October 26, 2017most recent commit 6 years ago

Livy is an open source REST interface for interacting with Apache Spark from anywhere

dependent packages 5total releases 4latest release April 12, 2017most recent commit 4 years ago

Popular Data Processing Categories

Jupyter Notebook

Categories

Application Programming Interfaces

Artificial Intelligence

Cloud Computing

Command Line Interface

Computer Science

Configuration Management

Content Management

Data Processing

Integrated Development Environments

Learning Resources

Lists Of Projects

Machine Learning

Operating Systems

Package Managers

Programming Languages

Runtime Environments

Software Architecture

Software Development

Software Performance

Software Quality

Text Processing

User Interface Components

Version Control

Web User Interface

Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source. All rights reserved.