Awesome Open Source
Awesome Open Source
Combined Topics
hadoop x
python x

The Top 481 Python Hadoop Open Source Projects

Spark ⭐ 33,580
Apache Spark - A unified analytics engine for large-scale data processing
dependent packages 882total releases 46most recent commit a day ago
Data Science Ipython Notebooks ⭐ 22,662
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
most recent commit 5 months ago
Luigi ⭐ 15,838
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
dependent packages 67total releases 74most recent commit 18 days agopypi luigi} Downloads
Pdf ⭐ 15,635
编程电子书，电子书，编程书籍，包括C，C#，Docker，Elasticsearch，Git，Had
most recent commit 3 months ago
Deeplearning4j ⭐ 12,568
Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learning using automatic differentiation.
dependent packages 109total releases 52most recent commit 2 days ago
It_book ⭐ 8,543
本项目收藏这些年来看过或者听过的一些不错的常用的上千本书籍，没准你想找的书就在这里呢，包含了互联网行
most recent commit 10 months ago
H2o 3 ⭐ 5,918
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
dependent packages 30total releases 229most recent commit a day ago
School Of Sre ⭐ 5,614
At LinkedIn, we are using this curriculum for onboarding our entry-level talents into the SRE role.
most recent commit 2 months ago
Bigdl ⭐ 3,994
Building Large-Scale AI Applications for Distributed Big Data
dependent packages 10total releases 16most recent commit a day ago
Tensorflowonspark ⭐ 3,781
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
total releases 32most recent commit 4 months ago
Ibis ⭐ 1,940
Expressive analytics in Python at any scale.
dependent packages 15total releases 30most recent commit a day agopypi ibis-framework} Downloads
Ambari ⭐ 1,564
Mirror of Apache Ambari
total releases 2most recent commit 9 months ago
Nagios Plugins ⭐ 1,067
450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...
most recent commit a day ago
Studybooks ⭐ 999
我的学习资料，包括书籍、网址等
most recent commit 8 years ago
Coding Now ⭐ 893
学习记录的一些笔记，以及所看得一些电子书eBooks、视频资源和平常收纳的一些自己认为比较好的博客、
most recent commit 5 months ago
Snakebite ⭐ 854
A pure python HDFS client
dependent packages 12total releases 63most recent commit 4 months agopypi snakebite} Downloads
Cdap ⭐ 660
An open source framework for building data analytic applications.
dependent packages 26total releases 62most recent commit 2 days ago
Dist Keras ⭐ 611
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
total releases 2most recent commit 4 years ago
Flintrock ⭐ 601
A command-line tool for launching Apache Spark clusters.
total releases 13most recent commit 3 months ago
Devops Python Tools ⭐ 580
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
most recent commit 6 days ago
Aws Glue Libs ⭐ 461
AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
most recent commit 4 months ago
Tuiblogs ⭐ 443
优秀的计算机编程类博客和文章 share excellent blogs and sites
most recent commit 4 years ago
Bigtop ⭐ 435
Mirror of Apache Bigtop
most recent commit a day ago
Gather Deployment ⭐ 345
Gathers scalable Tensorflow, Python infrastructure deployment and practices, 100% Docker.
most recent commit 4 months ago
Elasticluster ⭐ 317
Create clusters of VMs on the cloud and configure them with Ansible.
total releases 12most recent commit 5 months ago
Hadoopy ⭐ 244
Python MapReduce library written in Cython. Visit us in #hadoopy on freenode. See the link below for documentation and tutorials.
total releases 3most recent commit 7 years ago
Sagemaker Spark ⭐ 228
A Spark library for Amazon SageMaker.
total releases 32most recent commit a year ago
Maas ⭐ 218
Official MAAS repository mirror (may be out of date). Development happens in Launchpad (https://git.launchpad.net/maas/).
most recent commit 9 days ago
Hadoop Attack Library ⭐ 200
A collection of pentest tools and resources targeting Hadoop environments
most recent commit 5 years ago
Zohmg ⭐ 175
Zohmg is a data store for aggregation of multi-dimensional time series data, built on top of Hadoop, Dumbo and HBase.
most recent commit 10 years ago
Avenir ⭐ 162
Set of Machine Learning and Stochastic Optimazion tools based on Hadoop, Spark and Storm https://pkghosh.wordpress.com/
most recent commit a month ago
Airflow Pipeline ⭐ 155
An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR
most recent commit a month ago
Bigdata Playground ⭐ 154
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
most recent commit 4 years ago
Ipython Spark Docker ⭐ 151
most recent commit 6 years ago
Spatialanalytics ⭐ 134
Where 2.0 Workshop Code: Spatial Analysis of Tweets using Hadoop, Pig, Python & Mechanical Turk. Slides here: http://www.slideshare.net/kevinweil/spatial-analyt
most recent commit 12 years ago
Griffon Vm ⭐ 129
Griffon Data Science Virtual Machine
most recent commit 8 days ago
Skein ⭐ 126
A tool and library for easily deploying applications on Apache YARN
dependent packages 6total releases 23most recent commit 4 months agopypi skein} Downloads
Aut ⭐ 117
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
total releases 25most recent commit 2 months ago
Hprof2flamegraph ⭐ 108
Flame Graph visualization for Java (HPROF, Honest-profiler)
total releases 6most recent commit 4 years ago
Introtohadoopandmr__udacity_course ⭐ 103
🐘 Source code for assignments of Udacity course "Introduction to Hadoop and MapReduce"
most recent commit 8 years ago
Hadoop Yarn Api Python Client ⭐ 99
Python client for Hadoop® YARN API
dependent packages 3total releases 21most recent commit 7 months agopypi yarn-api-client} Downloads
Spark With Python ⭐ 98
Fundamentals of Spark with Python (using PySpark), code examples
most recent commit 2 years ago
Ambari Hue Service ⭐ 97
Ambari stack service for easily installing and managing Hue on HDP cluster
most recent commit 3 years ago
Annotated Wikiextractor ⭐ 88
Simple Wikipedia plain text extractor with article link annotations and Hadoop support.
most recent commit 11 years ago
Briefly ⭐ 85
Briefly - A Python Meta-programming Library for Job Flow Control
most recent commit 4 years ago
Dlflow ⭐ 84
DLFlow is a deep learning framework.
most recent commit 2 years ago
Tf Yarn ⭐ 82
Train TensorFlow models on YARN in just a few lines of code!
total releases 15most recent commit 4 months ago
Solutions Google Compute Engine Cluster For Hadoop ⭐ 81
This sample app will get up and running quickly with a Hadoop cluster on Google Compute Engine. For more information on running Hadoop on GCE, read the papers at https://cloud.google.com/resources/.
most recent commit 5 years ago
Data Pipeline ⭐ 79
Data pipeline is a tool to run Data loading pipelines. It is an open sourced app engine app that users can extend to suit their own needs. Out of the box it will load files from a source, transform them and then output them (output might be writing to a file or loading them into a data analysis tool). It is designed to be modular and support various sources, transformation technologies and output types. The transformations can be chained together to form complex pipelines.
most recent commit 8 years ago
Resilient Ml Research Platform ⭐ 76
most recent commit 6 days ago
Pyhdfs ⭐ 73
Python HDFS client
dependent packages 5total releases 8most recent commit a year agopypi PyHDFS} Downloads
Bisheserver ⭐ 69
本系统是我的毕业设计项目，题目为“基于用户画像的电影推荐系统的设计与实现”。主要是以Django作为
most recent commit 3 months ago
Geoprocessing Tools For Hadoop ⭐ 68
The Hadoop GP Toolbox provides tools to exchange features between a Geodatabase and Hadoop and run Hadoop workflow jobs.
most recent commit 4 years ago
Python Hdfs ⭐ 67
HDFS client for Python
most recent commit 12 years ago
Apache Spark Hands On ⭐ 64
Educational notes,Hands on problems w/ solutions for hadoop ecosystem
most recent commit 4 years ago
Airflow Spark ⭐ 64
Docker with Airflow and Spark standalone cluster
most recent commit 6 months ago
Pystratus ⭐ 63
Python-based utility for managing various distributed services on cloud providers
most recent commit 9 years ago
Dask Yarn ⭐ 61
Deploy dask on YARN clusters
dependent packages 1total releases 17most recent commit 2 months agopypi dask-yarn} Downloads
Coursework ⭐ 59
most recent commit 4 years ago
Knit ⭐ 54
Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead
dependent packages 1total releases 11most recent commit 4 years agopypi knit} Downloads
Apachespark ⭐ 54
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
most recent commit 2 months ago
Snabler ⭐ 54
Parallel Algorithms in Python for Hadoop/Mapreduce
most recent commit 10 years ago
Spark Training ⭐ 52
Repository used for Spark Trainings
most recent commit 10 months ago
Hadoop_vision ⭐ 49
Example code for "Web-Scale Computer Vision using MapReduce for Multimedia Data Mining"
most recent commit 12 years ago
Mylearningnotes ⭐ 49
Because its never late to start taking notes and 'public' it...
most recent commit a month ago
Cqu_bigdata ⭐ 48
重庆大学计算机学院“大数据课程群”实验及PPT
most recent commit 2 months ago
Qds Sdk Py ⭐ 47
Python SDK for accessing Qubole Data Service
dependent packages 13total releases 69most recent commit 2 years agopypi qds_sdk} Downloads
Pydrill ⭐ 46
Python Driver for Apache Drill.
total releases 8most recent commit a year ago
Ossocr ⭐ 46
gathering point for open source OCR scripts and diffs
most recent commit 8 years ago
Medicare Demo ⭐ 44
A demo of how to use PageRank with Hadoop and SociaLite to identify anomalies in Healthcare Data
most recent commit 7 years ago
Ibis ⭐ 44
IBIS is a workflow creation-engine that abstracts the Hadoop internals of ingesting RDBMS data.
most recent commit 4 years ago
Icwsm2010_tutorial ⭐ 44
example code for "Large-scale social media analysis with Hadoop" tutorial presented at ICWSM 2010
most recent commit 12 years ago
Machine_learning_in_action_py3 ⭐ 44
Important book about the machine learning algorithms, and introduces the application of those who use these algorithms and tools, and how to use them in a real environment. This book and other books, behind the other books are long on machine learning theory knowledge, the book happened to be more discussion on how to use coded machine learning algorithms.
most recent commit 2 years ago
Webhdfs Py ⭐ 43
Python Client for WebHDFS REST API
most recent commit 7 years ago
Python Devops ⭐ 41
gathers Python stack for DevOps, these are usually my basic templates use for my implementations, so, feel free to use it and evolve it! Everything is Docker!
most recent commit 3 years ago
Pybigdata ⭐ 39
使用 python 操作大数据的各种组件
most recent commit 2 months ago
Slinky ⭐ 39
Slinky, a high-performance web crawler / text analytics in Python, Redis, Hadoop, R, Gephi
most recent commit 12 years ago
Til ⭐ 38
Today I Learned
most recent commit 10 months ago
Hadoop Multi Server Ansible ⭐ 38
Multi-server deployment of Hadoop using Ansible
most recent commit 7 years ago
Big Data Exploration ⭐ 37
[Archive] Intern project - Big Data Exploration using MongoDB - This Repository is NOT a supported MongoDB product
most recent commit 4 years ago
Swordfish ⭐ 37
Open-source distribute workflow schedule tools, also support streaming task.
most recent commit 5 years ago
Test.fm ⭐ 36
Testing framework for Collaborative Filtering
most recent commit 7 years ago
Luiti ⭐ 35
A time task management framework, support multiple projects, built on top of luigi.
most recent commit 7 years ago
Hadoop_exporter ⭐ 35
A hadoop exporter for prometheus, scrape hadoop metrics (including HDFS, YARN, MAPREDUCE, HBASE. etc.) from hadoop components jmx url.
most recent commit 2 years ago
Hortonworks Sandbox ⭐ 33
hortonworks-sandbox
most recent commit 5 years ago
Telemetry Analysis Service ⭐ 33
Telemetry Analysis Service
most recent commit 3 years ago
Ansible Ambari ⭐ 33
Quickly deploy Hadoop with the help of Ansible and Apache Ambari
most recent commit 7 years ago
Bigdata Docker Compose ⭐ 33
Hadoop, Hive, Spark, Zeppelin and Livy: all in one Docker-compose file.
most recent commit 2 years ago
Jydoop ⭐ 32
Efficient Hadoop Map-Reduce in Python
most recent commit 8 years ago
Jmxtrans Lib ⭐ 32
JMXTrans configuration for hadoop/cassandra/zookeeper
most recent commit 7 years ago
Matrix Hadoop Tutorial ⭐ 31
A set of tutorial codes about matrix methods in Hadoop
most recent commit 9 years ago
Gparml ⭐ 31
Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models.
most recent commit 9 years ago
Hadoop Job Analyzer ⭐ 31
most recent commit 8 years ago
Dataprocess ⭐ 30
慕课网公开课：数据预处理
most recent commit 3 years ago
Webarchive Indexing ⭐ 30
Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
most recent commit 5 years ago
Awesome Tools ⭐ 29
curated list of awesome tools and libraries for specific domains
most recent commit 2 months ago
Gads7 ⭐ 28
GA Data Science NY Section 7
most recent commit 8 years ago
Compute Hadoop Java Python ⭐ 28
This software demonstrates one way to create and manage a cluster of Hadoop nodes running on Google Compute Engine.
most recent commit 7 years ago
Eggo ⭐ 28
Ready-to-go Parquet-formatted public 'omics datasets
most recent commit 7 years ago
Pydatapreprocessing ⭐ 26
《Python数据预处理技术与实践》源码下载
most recent commit 3 years ago
1-100 of 481 projects
Related Awesome Lists
Python Dataset Projects (14,792)
Python Pytorch Projects (14,667)
Python Machine Learning Projects (14,099)
Python Tensorflow Projects (13,736)
Python Deep Learning Projects (13,092)
Python Learning Projects (10,661)
Python Jupyter Notebook Projects (8,507)
Python Neural Nets Projects (7,502)
Python Html Projects (6,892)
Python Deep Projects (6,552)
Python Keras Projects (6,284)
Python Docker Projects (4,744)
Python Cloud Projects (4,599)
Javascript Python Projects (4,528)
Python Pipeline Projects (4,199)
Python Data Science Projects (3,737)
Python Redis Projects (3,473)
Python Mysql Projects (3,470)
Python Conda Projects (3,079)
Python Gpu Projects (2,830)
Python Postgresql Projects (2,800)
Python Version Projects (2,600)
Python Sqlalchemy Projects (2,448)
Python Role Projects (2,423)
Python Pandas Projects (2,377)
Python Aws Projects (2,184)
Java Hadoop Projects (2,117)
Python Artificial Intelligence Projects (2,116)
Python Numpy Projects (2,087)
Python Elasticsearch Projects (1,920)
Python Java Projects (1,851)
Python Sqlite Projects (1,806)
Python Security Projects (1,733)
Python Apache Projects (1,438)
Python Dependencies Projects (1,423)
Python Database Projects (1,373)
Python Matplotlib Projects (1,366)
Python Json Projects (1,338)
Python Parallel Projects (1,198)
Spark Hadoop Projects (1,188)
Python Scikit Learn Projects (1,151)
Hadoop Hdfs Projects (1,082)
Python Sql Projects (1,075)
Python Kafka Projects (1,053)
Python Rabbitmq Projects (981)
Python Kubernetes Projects (980)
Python Git Projects (957)
Python Github Projects (904)
Python Cloudformation Projects (874)
Hadoop Mapreduce Projects (851)
Python Jenkins Projects (821)
Python R Projects (770)
Shell Hadoop Projects (766)
Python Networking Projects (751)
Python Monitoring Projects (720)
Python Spark Projects (586)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,072,919)
Python  (820,561)
Java  (391,895)
Php  (287,508)
Typescript  (255,004)
C Plus Plus  (241,814)
Ruby  (222,128)
C  (183,606)
Shell  (173,092)
C Sharp  (166,391)
Golang  (160,615)
Swift  (65,849)
Rust  (59,377)
R  (58,670)
Objective C  (57,112)
Dart  (54,061)
Kotlin  (47,624)
Lua  (34,629)
Matlab  (31,405)
Perl  (30,580)
Language  (29,990)
Scala  (28,784)
Haskell  (20,491)
Powershell  (19,276)
Clojure  (19,184)
Bash  (18,781)
Coffeescript  (17,256)
Elixir  (16,766)
Assembly  (14,775)
Processing  (13,285)
Julia  (13,012)
Basic  (10,721)
Flow  (10,535)
Emacs Lisp  (10,467)
Erlang  (9,062)
Groovy  (8,464)
Scheme  (8,415)
Solidity  (8,342)
Ocaml  (6,749)
Pascal  (6,486)
Hack  (6,426)
Lisp  (6,337)
Fortran  (5,580)
Elm  (5,475)
Common Lisp  (5,325)
Dlang  (4,745)
Shell Script  (4,685)
Python Library  (4,467)
Crystal  (4,390)
Scripting  (4,216)
Top Projects
Freecodecamp ⭐ 351,148
996.icu ⭐ 263,365
Free Programming Books ⭐ 244,273
Coding Interview University ⭐ 228,551
Awesome ⭐ 211,770
Developer Roadmap ⭐ 205,499
Public Apis ⭐ 204,617
Vue ⭐ 198,463
React ⭐ 192,917
System Design Primer ⭐ 191,492
Tensorflow ⭐ 166,973
Build Your Own X ⭐ 160,771
Bootstrap ⭐ 158,895
You Dont Know Js ⭐ 157,194
Cs Notes ⭐ 153,924
Ohmyzsh ⭐ 148,574
Javascript Algorithms ⭐ 147,277
Flutter ⭐ 143,388
Python ⭐ 141,170
Gitignore ⭐ 136,695
Awesome Python ⭐ 136,585
Linux ⭐ 136,029
Vscode ⭐ 135,160
Javaguide ⭐ 126,035
Javascript ⭐ 125,359
Python 100 Days ⭐ 121,533
Computer Science ⭐ 121,209
Youtube Dl ⭐ 112,200
The Art Of Command Line ⭐ 108,745
Fucking Algorithm ⭐ 108,035
React Native ⭐ 104,196
Electron ⭐ 102,988
Go ⭐ 102,591
D3 ⭐ 101,796
30 Seconds Of Code ⭐ 100,507
Awesome Selfhosted ⭐ 97,978
Create React App ⭐ 96,376
Axios ⭐ 95,042
Free Programming Books Zh_cn ⭐ 94,762
Kubernetes ⭐ 90,962
Next.js ⭐ 90,806
Node ⭐ 89,359
Awesome Go ⭐ 85,464
Terminal ⭐ 84,600
Deno ⭐ 84,450
Three.js ⭐ 84,235
Angular ⭐ 83,120
Typescript ⭐ 83,050
Ant Design ⭐ 81,365
Material Ui ⭐ 80,347
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.