Awesome Open Source
Awesome Open Source
Combined Topics
hdfs
x
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
The Top 315 Hdfs Open Source Projects on Github
Categories
>
Data Storage
>
Hdfs
Seaweedfs
⭐
12,883
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, Hadoop, WebDAV, encryption, Erasure Coding.
Bigdata Notes
⭐
9,567
大数据入门指南 ⭐️
God Of Bigdata
⭐
5,081
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...
Juicefs
⭐
3,733
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
Smart_open
⭐
2,221
Utils for streaming large files (S3, HDFS, gzip, bz2...)
Ibis
⭐
1,606
A pandas-like deferred expression system, with first-class SQL support
Tiledb
⭐
1,195
The Universal Storage Engine
Hdfs
⭐
1,055
A native go client for HDFS
Bigdata Interview
⭐
1,041
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结
Snakebite
⭐
804
A pure python HDFS client
Sparta
⭐
521
Real Time Analytics and Data Pipelines based on Spark Streaming
Devops Python Tools
⭐
463
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Kafka Connect Hdfs
⭐
423
Kafka Connect HDFS connector
Kafka Connect Ui
⭐
409
Web tool for Kafka Connect |
Bigdata
⭐
358
💎🔥大数据学习笔记
Divolte Collector
⭐
275
Divolte Collector
Repository
⭐
273
个人学习知识库涉及到数据仓库建模、实时计算、大数据、Java、算法等。
Hdfs
⭐
217
API and command line interface for HDFS
Bigdata_docker
⭐
211
Big Data Ecosystem Docker
Storagetapper
⭐
196
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Dcos Commons
⭐
162
DC/OS SDK is a collection of tools, libraries, and documentation for easy integration of technologies such as Kafka, Cassandra, HDFS, Spark, and TensorFlow with DC/OS.
Wradlib
⭐
157
weather radar data processing - python package
Bigdata File Viewer
⭐
142
A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.
Hsuntzu
⭐
134
HDFS compress tar zip snappy gzip uncompress untar codec hadoop spark
Elasticctr
⭐
125
ElasticCTR，即飞桨弹性计算推荐系统，是基于Kubernetes的企业级推荐系统开源解决方案。该方案融合了百度业务场景下持续打磨的高精度CTR模型、飞桨开源框架的大规模分布式训练能力、工业级稀疏参数弹性调度服务，帮助用户在Kubernetes环境中一键完成推荐系统部署，具备高性能、工业级部署、端到端体验的特点，并且作为开源套件，满足二次深度开发的需求。
Skein
⭐
116
A tool and library for easily deploying applications on Apache YARN
Tiledb Py
⭐
115
Python interface to the TileDB storage engine
Dynamometer
⭐
110
A tool for scale and performance testing of HDFS with a specific focus on the NameNode.
Spark With Python
⭐
98
Fundamentals of Spark with Python (using PySpark), code examples
Wifi
⭐
95
基于wifi抓取信息的大数据查询分析系统
Hadoop Hdfs Fsimage Exporter
⭐
95
Exports Hadoop HDFS content statistics to Prometheus
Nnanalytics
⭐
92
NameNodeAnalytics is a self-help utility for scouting and maintaining the namespace of an HDFS instance.
Kafka Connect Fs
⭐
86
Kafka Connect FileSystem Connector
Hdfs Shell
⭐
85
HDFS Shell is a HDFS manipulation tool to work with functions integrated in Hadoop DFS
Camus
⭐
79
Mirror of Linkedin's Camus
Ros_hadoop
⭐
75
Hadoop splittable InputFormat for ROS. Process rosbag with Hadoop Spark and other HDFS compatible systems.
Euphoria
⭐
74
Euphoria is an open source Java API for creating unified big-data processing flows. It provides an engine independent programming model which can express both batch and stream transformations.
Pyhdfs
⭐
73
Python HDFS client
Devops Perl Tools
⭐
68
25+ DevOps CLI Tools - Anonymizer, SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles), Hadoop HDFS & Hive tools, Solr/SolrCloud CLI, Nginx stats & HTTP(S) URL watchers for load-balanced web farms, Linux tools etc.
Rumble
⭐
68
⛈️ RumbleDB 1.15.0 "Ivory Palm" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
Tiledb R
⭐
65
R interface to the TileDB Universal Storage Engine
Big Data Engineering Coursera Yandex
⭐
63
Big Data for Data Engineers Coursera Specialization from Yandex
Rainbow
⭐
61
A data layout optimization framework for wide tables stored on HDFS. See rainbow's webpage
Cloud Note
⭐
59
基于分布式的云笔记（参考某道云笔记），数据存储在redis与hbase中
Fluent Plugin Webhdfs
⭐
56
Hadoop WebHDFS output plugin for Fluentd
Geodocker
⭐
55
Central repository for the GeoDocker project
Flume Canal Source
⭐
54
Flume NG Canal source
Monix Connect
⭐
52
A set of connectors for Monix. 🔛
Docker Hadoop
⭐
52
Docker image for main Apache Hadoop components (Yarn/Hdfs)
Teraslice
⭐
47
Scalable data processing pipelines in JavaScript
Pxf
⭐
41
Platform Extension Framework: Federated Query Engine
Jsr203 Hadoop
⭐
41
A Java NIO file system provider for HDFS
Getl
⭐
40
A tool for developing and testing ETL and ELT processes for automating the capture, delivery and processing of information in data warehouses on the MicroFocus Vertica platform.
Samples
⭐
38
This repository contains open-source sample applications for IBM Streams.
Etl Light
⭐
38
A light Kafka to HDFS/S3 ETL library based on Apache Spark
Hadoop Tools
⭐
37
Tools for working with Hadoop, written with performance in mind.
Bigdatatools
⭐
35
tools for bigData
Hdfs Netdisc
⭐
33
基于Hadoop的分布式云存储系统 🌴
Pucket
⭐
29
Bucketing and partitioning system for Parquet
Cluster Pack
⭐
28
A library on top of either pex or conda-pack to make your Python code easily available on a cluster
Aaocp
⭐
27
一个对用户行为日志进行分析的大数据项目
Sansa Notebooks
⭐
25
Interactive Spark Notebooks for running SANSA examples.
Tiledb Go
⭐
25
Go Interface to the TileDB storage manager
Fsbrowser
⭐
25
Fast desktop client for Hadoop Distributed File System
Hadooplearning
⭐
25
全套大数据基础学习教程，包含最基础的centos、maven。大数据主要包含hdfs、mr、yarn、hbase、kafka、scala、sparkcore、sparkstreaming、sparksql。教程包含所有的源代码演示以及在线文档说明。
Cobol To Hive
⭐
24
Serde for Cobol Layout to Hive table
Hfsa
⭐
24
Hadoop FSImage Analyzer (HFSA)
Learning Spark
⭐
22
零基础学习spark，大数据学习
Trino Storage
⭐
22
Storage connector for Trino
Node Webhdfs
⭐
21
A WebHDFS module for Node.js.
Hbase Meta Repair
⭐
20
Repair hbase metadata table from hdfs.
Oci Hdfs Connector
⭐
19
HDFS Connector for Oracle Cloud Infrastructure
Hive_to_es
⭐
18
同步Hive数据仓库数据到Elasticsearch的小工具
Stream Loader
⭐
18
Components for building stream loaders from Kafka to arbitrary storages
Datax Src
⭐
16
DataX 是异构数据广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。
Real_time_social_media_mining
⭐
16
DevOps pipeline for Real Time Social/Web Mining
Fastdata Cluster
⭐
15
Fast Data Cluster (Apache Cassandra, Kafka, Spark, Flink, YARN and HDFS with Vagrant and VirtualBox)
Yandex Big Data Engineering
⭐
15
Datasqueeze
⭐
14
Hadoop utility to compact small files
Gaffer Docker
⭐
14
Gaffer Docker images and associated Helm charts for deploying on Kubernetes
Bigkube
⭐
14
Minikube for big data with Scala and Spark
Bigdata Fun
⭐
14
A complete (distributed) BigData stack, running in containers
Wasp
⭐
14
WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.
Spark Fits
⭐
14
FITS data source for Spark SQL and DataFrames
Local Hashicorp Stack
⭐
14
Local Hashicorp Stack for DevOps Development without Hypervisor or Cloud
Ucz Dfs
⭐
14
A distributed file system written in Rust.
Leaflet_heatmap
⭐
13
简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .
Bigdata Doc
⭐
13
大数据学习笔记，学习路线，技术案例整理。
Robot
⭐
13
Robot is a framework base on flink below v1.5, serve for 'oshit team', develop in python3.6。
Py Hdfs Mount
⭐
13
Mount HDFS with fuse, works with kerberos!
Bigdata Skilltree
⭐
12
Spark、flink、HBase、Hive、flume集成了一些Hadoop的原生api的一些demo（如HDFS、MapReduce：目前就这两个）；同时测试一些异常功能
Sbk
⭐
12
Storage Benchmark Kit
Taller_sparkr
⭐
12
Taller SparkR para las Jornadas de Usuarios de R
Dumping Machine
⭐
12
Dumping Machine is an application which dumps Kafka Avro topics to S3 or HDFS as Parquet
Bigdataguide
⭐
11
一份详细的大数据开发所需要掌握的核心知识点，基础 | 架构 | 源码，让你自学从此不求人。 有相关问题可以添加公众号：努力的老刘，联系老刘！
Git Influencer
⭐
11
Insight Data Engineering project: A platform built in HDFS, Spark and Airflow to help you to find social influencers from GitHub Network.
Dcos Jupyterlab Service
⭐
11
JupyterLab Notebook for Mesosphere DC/OS
Tramp Hdfs
⭐
11
Browser HDFS in Emacs using Tramp
Kafka Connect Field And Time Partitioner
⭐
11
Kafka Connect Store Partitioner by custom fields and time
Veloxdfs
⭐
11
🚲 Consistent Hashing FS for Big Data
1-100 of 315 projects
Next >
Related Projects
Hadoop Hdfs Projects (152)
Spark Hdfs Projects (98)
Java Hdfs Projects (96)
Python Hdfs Projects (60)
Hive Hdfs Projects (56)
Advertising
📦 9
All Projects
Application Programming Interfaces
📦 120
Applications
📦 181
Artificial Intelligence
📦 72
Blockchain
📦 70
Build Tools
📦 111
Cloud Computing
📦 79
Code Quality
📦 28
Collaboration
📦 30
Command Line Interface
📦 48
Community
📦 81
Companies
📦 60
Compilers
📦 60
Computer Science
📦 74
Configuration Management
📦 39
Content Management
📦 167
Control Flow
📦 197
Data Formats
📦 77
Data Processing
📦 266
Data Storage
📦 132
Economics
📦 60
Frameworks
📦 198
Games
📦 122
Graphics
📦 103
Hardware
📦 148
Integrated Development Environments
📦 47
Learning Resources
📦 147
Legal
📦 28
Libraries
📦 119
Lists Of Projects
📦 21
Machine Learning
📦 336
Mapping
📦 61
Marketing
📦 15
Mathematics
📦 55
Media
📦 228
Messaging
📦 97
Networking
📦 304
Operating Systems
📦 84
Operations
📦 120
Package Managers
📦 52
Programming Languages
📦 229
Runtime Environments
📦 96
Science
📦 42
Security
📦 375
Social Media
📦 26
Software Architecture
📦 70
Software Development
📦 68
Software Performance
📦 57
Software Quality
📦 127
Text Editors
📦 45
Text Processing
📦 131
User Interface
📦 310
User Interface Components
📦 465
Version Control
📦 29
Virtualization
📦 68
Web Browsers
📦 38
Web Servers
📦 25
Web User Interface
📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.
