Awesome Open Source
Awesome Open Source
Combined Topics
hdfs x

The Top 2,224 Hdfs Open Source Projects

Cat ⭐ 16,631
CAT 作为服务端项目基础组件，提供了 Java, C/C++, Node.js, Python, Go 等多语言客户端，已经在美团点评的基础架构中间件框架（MVC框架，RPC框架，数据库框架，缓存框架等，
total releases 5most recent commit 21 days ago
Seaweedfs ⭐ 14,480
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
dependent packages 1total releases 296most recent commit 5 hours ago
Bigdata Notes ⭐ 11,183
大数据入门指南 ⭐️
most recent commit 3 months ago
Datax ⭐ 11,073
DataX是阿里云DataWorks数据集成的开源版本。
most recent commit 7 days ago
Mycat Server ⭐ 9,266
most recent commit a month ago
God Of Bigdata ⭐ 6,749
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive.
most recent commit 11 days ago
Juicefs ⭐ 5,343
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
dependent packages 1total releases 113most recent commit 7 hours ago
Tensorflowonspark ⭐ 3,781
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
total releases 32most recent commit a month ago
Smart_open ⭐ 2,431
Utils for streaming large files (S3, HDFS, gzip, bz2...)
dependent packages 137total releases 52most recent commit a month agopypi smart-open} Downloads
Ibis ⭐ 1,814
Expressive analytics in Python at any scale.
dependent packages 15total releases 30most recent commit 5 hours agopypi ibis-framework} Downloads
Xlearning ⭐ 1,627
AI on Hadoop
most recent commit 2 years ago
Docker Hadoop ⭐ 1,594
Apache Hadoop docker image
most recent commit 2 months ago
Poseidon ⭐ 1,543
A search engine which can hold 100 trillion lines of log data.
most recent commit 5 years ago
Drake ⭐ 1,472
Data workflow tool, like a "Make for data"
total releases 9most recent commit a month ago
Tiledb ⭐ 1,327
The Universal Storage Engine
dependent packages 6total releases 70most recent commit 5 hours ago
Hdfs ⭐ 1,150
A native go client for HDFS
dependent packages 46total releases 26most recent commit 2 months ago
Nagios Plugins ⭐ 1,051
450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...
most recent commit 9 days ago
Bigdata Interview ⭐ 1,041
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop
most recent commit 9 months ago
Snakebite ⭐ 854
A pure python HDFS client
dependent packages 12total releases 63most recent commit a month agopypi snakebite} Downloads
Sqoop ⭐ 820
Mirror of Apache Sqoop
most recent commit a year ago
Hopsworks ⭐ 709
Hopsworks - Data-Intensive AI platform with a Feature Store
most recent commit 12 hours ago
Hawq ⭐ 649
Apache HAWQ
most recent commit 2 days ago
Repository ⭐ 554
知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。
most recent commit 9 hours ago
Devops Python Tools ⭐ 550
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
most recent commit 9 days ago
Sparta ⭐ 521
Real Time Analytics and Data Pipelines based on Spark Streaming
most recent commit 3 years ago
Juicesync ⭐ 520
A tool to move your data between any clouds or regions.
dependent packages 1total releases 30most recent commit 12 days ago
Minos ⭐ 508
Minos is beyond a hadoop deployment system.
most recent commit 4 months ago
Docker Hadoop Spark Workbench ⭐ 503
[EXPERIMENTAL] This repo includes deployment instructions for running HDFS/Spark inside docker containers. Also includes spark-notebook and HDFS FileBrowser.
most recent commit 2 years ago
Kafka Connect Hdfs ⭐ 437
Kafka Connect HDFS connector
most recent commit 21 days ago
Kafka Connect Ui ⭐ 432
Web tool for Kafka Connect |
most recent commit 6 months ago
Storm Yarn ⭐ 419
Storm-yarn enables Storm clusters to be deployed into machines managed by Hadoop YARN.
most recent commit 2 years ago
Daily Deeplearning ⭐ 411
🔥机器学习/深度学习/Python/算法面试/自然语言处理教程/剑指offer/machine learning/deeplearning/Python/Algorithm interview/NLP Tutorial
most recent commit 10 months ago
Kite ⭐ 366
Kite SDK
dependent packages 17total releases 62most recent commit 2 years ago
Bigdata ⭐ 358
💎🔥大数据学习笔记
most recent commit 3 years ago
Spindle ⭐ 333
Next-generation web analytics processing with Scala, Spark, and Parquet.
most recent commit 7 years ago
Golang Distributed Filesystem ⭐ 312
HDFS-alike in Go. Written to learn the language and get a job.
most recent commit 3 years ago
Packetpig ⭐ 309
Packetpig - Open Source Big Data Security Analytics
most recent commit 4 years ago
Tensorspark ⭐ 302
TensorFlow on Spark
most recent commit 5 years ago
Divolte Collector ⭐ 275
Divolte Collector
most recent commit a year ago
Storagetapper ⭐ 269
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
total releases 4most recent commit 4 months ago
Faunus ⭐ 259
Graph Analytics Engine
total releases 10most recent commit 8 years ago
Hadoop Mini Clusters ⭐ 251
hadoop-mini-clusters provides an easy way to test Hadoop projects directly in your IDE
dependent packages 5total releases 15most recent commit 2 years ago
Hops ⭐ 246
Hops Hadoop is a distribution of Apache Hadoop with distributed metadata.
most recent commit 5 months ago
Hadoopy ⭐ 244
Python MapReduce library written in Cython. Visit us in #hadoopy on freenode. See the link below for documentation and tutorials.
total releases 3most recent commit 6 years ago
Hdfs ⭐ 240
API and command line interface for HDFS
most recent commit 2 months ago
Hdfs Mount ⭐ 234
A tool to mount HDFS as a local Linux file system
most recent commit 3 years ago
Pysparkling ⭐ 234
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
dependent packages 1total releases 68most recent commit a year agopypi pysparkling} Downloads
Omniduct ⭐ 231
A toolkit providing a uniform interface for connecting to and extracting data from a wide variety of (potentially remote) data stores (including HDFS, Hive, Presto, MySQL, etc).
total releases 72most recent commit 11 days ago
Bigdata_docker ⭐ 226
Big Data Ecosystem Docker
most recent commit 4 months ago
Zdh_web ⭐ 220
大数据采集,抽取平台
most recent commit 20 days ago
Zeppelin Notebooks ⭐ 206
Gallery of Apache Zeppelin notebooks
most recent commit 3 years ago
Hadoop Attack Library ⭐ 200
A collection of pentest tools and resources targeting Hadoop environments
most recent commit 5 years ago
Bigdata File Viewer ⭐ 192
A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.
most recent commit a month ago
Wifiprobeanalysis ⭐ 189
基于WIFI探针的商业大数据分析技术
most recent commit 2 years ago
Wradlib ⭐ 187
weather radar data processing - python package
most recent commit 11 hours ago
Spark Notes ⭐ 183
most recent commit 3 years ago
Terrapin ⭐ 168
Serving system for batch generated data sets
most recent commit 5 years ago
Rumble ⭐ 165
⛈️ RumbleDB 1.18.0 "Scarlet Ixora" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
total releases 4most recent commit a month ago
Dcos Commons ⭐ 162
DC/OS SDK is a collection of tools, libraries, and documentation for easy integration of technologies such as Kafka, Cassandra, HDFS, Spark, and TensorFlow with DC/OS.
total releases 7most recent commit a year ago
Sequins ⭐ 161
A key/value store for serving static batch data
total releases 13most recent commit 8 months ago
Magpie ⭐ 156
Magpie contains a number of scripts for running Big Data software in HPC environments, including Hadoop and Spark. There is support for Lustre, Slurm, Moab, Torque. LSF, and more.
most recent commit 4 months ago
Rlink Rs ⭐ 155
High-performance Stream Processing Framework. An alternative to Apache Flink.
dependent packages 10total releases 64most recent commit 5 months agocargo rlink} Downloads
Bigdata In Practice ⭐ 154
大数据实践项目 Hadoop、Spark、Kafka、Hbase、Flink.....
most recent commit 3 years ago
Ipython Spark Docker ⭐ 151
most recent commit 6 years ago
Deeplog ⭐ 142
Pytorch Implementation of DeepLog.
most recent commit 2 years ago
Tiledb Py ⭐ 138
Python interface to the TileDB storage engine
dependent packages 13total releases 82most recent commit 5 hours agopypi tiledb} Downloads
Kube Yarn ⭐ 135
Running YARN on Kubernetes with PetSet controller.
most recent commit 4 years ago
Distributed Graph Analytics ⭐ 135
Distributed Graph Analytics (DGA) is a compendium of graph analytics written for Bulk-Synchronous-Parallel (BSP) processing frameworks such as Giraph and GraphX. The analytics included are High Betweenness Set Extraction, Weakly Connected Components, Page Rank, Leaf Compression, and Louvain Modularity.
most recent commit 3 years ago
Hsuntzu ⭐ 134
HDFS compress tar zip snappy gzip uncompress untar codec hadoop spark
most recent commit 4 years ago
Lambda Arch ⭐ 134
A full big data pipeline (Lambda Architecture) with Spark, Kafka, HDFS and Cassandra.
most recent commit 4 days ago
Portainer ⭐ 130
Apache Mesos framework for building Docker images on a cluster of machines
most recent commit 5 years ago
Hdfs Shell ⭐ 129
HDFS Shell is a HDFS manipulation tool to work with functions integrated in Hadoop DFS
most recent commit 2 months ago
Hadoopdemo ⭐ 128
Hadoop简单应用案例，包括MapReduce、单词统计、HDFS基本操作、web日志分析、Zoo
most recent commit 3 years ago
Skein ⭐ 126
A tool and library for easily deploying applications on Apache YARN
dependent packages 6total releases 23most recent commit a month agopypi skein} Downloads
Elasticctr ⭐ 125
ElasticCTR，即飞桨弹性计算推荐系统，是基于Kubernetes的企业级推荐系统开源解决方案
most recent commit 2 years ago
Gowfs ⭐ 122
A Go client binding for Hadoop HDFS using WebHDFS.
most recent commit 2 years ago
Hdfs_fdw ⭐ 121
PostgreSQL foreign data wrapper for HDFS
most recent commit 2 days ago
Rubix ⭐ 121
Cache File System optimized for columnar formats and object stores
dependent packages 8total releases 39most recent commit a year ago
Vagrant Hadoop Spark Cluster ⭐ 121
Vagrant project to spin up a cluster of 4 32-bit CentOS6.5 Linux virtual machines with Hadoop v2.6.0 and Spark v1.1.1
most recent commit 6 years ago
Difacto_dmlc ⭐ 118
Distributed FM and LR based on Parameter Server with Ftrl
most recent commit 5 years ago
Kylin Docker ⭐ 116
This repository trackes the code and files for building docker image with Apache Kylin.
most recent commit 4 years ago
Mpich2 Yarn ⭐ 112
Running MPICH2 on Yarn
most recent commit 5 years ago
Kubernetes Yarn ⭐ 111
most recent commit 7 years ago
Hadron ⭐ 110
Construct and run Hadoop MapReduce programs in Haskell
most recent commit 5 years ago
Dynamometer ⭐ 110
A tool for scale and performance testing of HDFS with a specific focus on the NameNode.
most recent commit 3 years ago
Hadoop Hdfs Fsimage Exporter ⭐ 105
Exports Hadoop HDFS content statistics to Prometheus
most recent commit 11 days ago
Cobrix ⭐ 105
A COBOL parser and Mainframe/EBCDIC data source for Apache Spark
most recent commit 17 days ago
Play Videos In Hdfs ⭐ 102
This project realizes playing videos storing in HDFS(Hadoop) in the web page online.在线播放HDFS中视频文件
most recent commit 4 years ago
Nnanalytics ⭐ 99
NameNodeAnalytics is a self-help utility for scouting and maintaining the namespace of an HDFS instance.
most recent commit a month ago
Spark With Python ⭐ 98
Fundamentals of Spark with Python (using PySpark), code examples
most recent commit 2 years ago
Kafka Connect Fs ⭐ 98
Kafka Connect FileSystem Connector
most recent commit 5 months ago
Wifi ⭐ 95
基于wifi抓取信息的大数据查询分析系统
most recent commit 5 years ago
Ssm ⭐ 94
Smart Storage Management for Big Data, a comprehensive hot/cold data optimized solution
most recent commit 4 months ago
My Tutorial ⭐ 93
我想构建形成自己的知识的体系，工作职位是大数据，所以主要还是以大数据为主，从主流框架Hadoop，S 大数据开发是很繁琐的，正确的运行环境是成功的第一步，所以我尽量从搭建，部署，开发整个流程都做出来，单
most recent commit 5 months ago
Plsc ⭐ 92
Paddle Large Scale Classification Tools，supports ArcFace, CosFace, PartialFC, Data Parallel + Model Parallel.
most recent commit 3 months ago
Jaws Spark Sql Rest ⭐ 92
most recent commit 5 years ago
Correlation Approximation ⭐ 90
Spark implementation of the Google Correlate algorithm to quickly find highly correlated vectors in huge datasets
most recent commit 6 years ago
Focusbigdata ⭐ 89
【大数据成神之路学习路径+面经+简历】
most recent commit a month ago
Cuesheet ⭐ 84
A framework for writing Spark 2.x applications in a pretty way
most recent commit 5 years ago
Kafka Hadoop Loader ⭐ 84
Hadoop Job for schemaless incremental loading of messages from Kafka topics onto hdfs with configurable output partitioning. 🚫
most recent commit 6 years ago
1-100 of 2,224 projects
Related Awesome Lists
Hadoop Hdfs Projects (1,082)
Java Hdfs Projects (752)
Spark Hdfs Projects (573)
Python Hdfs Projects (355)
Hive Hdfs Projects (338)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,046,513)
Python  (794,038)
Java  (384,745)
Php  (284,342)
Typescript  (238,372)
C Plus Plus  (235,678)
Ruby  (221,647)
C  (180,021)
Shell  (169,271)
C Sharp  (161,668)
Golang  (155,705)
Swift  (64,594)
Objective C  (57,180)
R  (57,097)
Rust  (55,814)
Dart  (51,233)
Kotlin  (45,308)
Lua  (33,168)
Matlab  (30,832)
Perl  (30,482)
Language  (30,083)
Scala  (28,582)
Haskell  (20,223)
Clojure  (19,065)
Powershell  (18,770)
Bash  (18,586)
Coffeescript  (17,335)
Elixir  (16,549)
Assembly  (14,508)
Processing  (13,295)
Julia  (12,509)
Basic  (10,772)
Flow  (10,528)
Emacs Lisp  (10,358)
Erlang  (9,057)
Scheme  (8,442)
Groovy  (8,426)
Solidity  (7,543)
Ocaml  (6,660)
Pascal  (6,371)
Hack  (6,365)
Lisp  (6,345)
Fortran  (5,475)
Elm  (5,468)
Common Lisp  (5,285)
Dlang  (4,742)
Shell Script  (4,626)
Python Library  (4,446)
Crystal  (4,373)
Scripting  (4,199)
Top Projects
Freecodecamp ⭐ 346,085
996.icu ⭐ 262,270
Free Programming Books ⭐ 233,047
Coding Interview University ⭐ 219,725
Awesome ⭐ 201,499
Vue ⭐ 195,787
Developer Roadmap ⭐ 194,812
Public Apis ⭐ 192,026
React ⭐ 188,079
System Design Primer ⭐ 180,726
Tensorflow ⭐ 164,981
Bootstrap ⭐ 157,299
You Dont Know Js ⭐ 152,909
Cs Notes ⭐ 150,780
Ohmyzsh ⭐ 145,101
Build Your Own X ⭐ 141,447
Flutter ⭐ 140,685
Javascript Algorithms ⭐ 140,055
Python ⭐ 136,032
Gitignore ⭐ 133,164
Linux ⭐ 131,866
Vscode ⭐ 131,762
Awesome Python ⭐ 126,837
Javascript ⭐ 122,717
Javaguide ⭐ 121,487
Python 100 Days ⭐ 117,585
Computer Science ⭐ 112,893
Youtube Dl ⭐ 110,029
Fucking Algorithm ⭐ 106,048
The Art Of Command Line ⭐ 105,874
React Native ⭐ 102,754
Electron ⭐ 101,722
D3 ⭐ 100,901
Go ⭐ 99,254
30 Seconds Of Code ⭐ 95,528
Create React App ⭐ 95,021
Axios ⭐ 93,400
Awesome Selfhosted ⭐ 88,479
Kubernetes ⭐ 88,475
Free Programming Books Zh_cn ⭐ 88,386
Node ⭐ 87,657
Next.js ⭐ 86,824
Terminal ⭐ 83,149
Deno ⭐ 82,461
Three.js ⭐ 82,021
Angular ⭐ 81,395
Awesome Go ⭐ 80,742
Typescript ⭐ 80,677
Ant Design ⭐ 80,313
Material Ui ⭐ 78,463
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.