Awesome Open Source
Awesome Open Source
Combined Topics
machine-learning x
spark x

The Top 423 Machine Learning Spark Open Source Projects

Categories > Machine Learning > Machine Learning
Data Science Ipython Notebooks ⭐ 23,924
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
most recent commit 3 months ago
Horovod ⭐ 12,968
Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.
dependent packages 10total releases 72most recent commit 21 hours agopypi horovod} Downloads
Angel ⭐ 6,625
A Flexible and Powerful Parameter Server for large-scale machine learning
dependent packages 7total releases 23most recent commit 2 months ago
H2o 3 ⭐ 6,119
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
dependent packages 30total releases 232most recent commit 21 hours ago
Pipeline ⭐ 4,140
PipelineAI Kubeflow Distribution
total releases 85most recent commit 3 months ago
Synapseml ⭐ 3,890
Simple and Distributed Machine Learning
dependent packages 1total releases 5most recent commit a day agopypi synapseml} Downloads
Tensorflowonspark ⭐ 3,781
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
total releases 32most recent commit 9 months ago
Mage Ai ⭐ 3,220
🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.
total releases 9most recent commit a day ago
Spark Nlp ⭐ 3,066
State of the Art Natural Language Processing
dependent packages 2total releases 90most recent commit 20 hours ago
Scio ⭐ 2,420
A Scala API for Apache Beam and Google Cloud Dataflow.
dependent packages 34total releases 82most recent commit 20 hours ago
Transmogrifai ⭐ 2,099
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
dependent packages 3total releases 9most recent commit a year ago
Spark ⭐ 1,869
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
dependent packages 16total releases 20most recent commit 2 days agonuget Microsoft.Spark} Downloads
Benchm Ml ⭐ 1,839
A minimal benchmark for scalability, speed and accuracy of commonly used open source implementations (R packages, Python scikit-learn, H2O, xgboost, Spark MLlib etc.) of the top machine learning algorithms for binary classification (random forests, gradient boosted trees, deep neural networks etc.).
most recent commit 4 months ago
Oryx ⭐ 1,759
Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
dependent packages 2total releases 14most recent commit 2 years ago
Spark Ml Source Analysis ⭐ 1,710
spark ml 算法原理剖析以及具体的源码实现分析
most recent commit 4 years ago
Home ⭐ 1,635
ApacheCN 开源组织：公告、介绍、成员、活动、交流方式
most recent commit 5 days ago
Petastorm ⭐ 1,543
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
dependent packages 6total releases 77most recent commit a month agopypi petastorm} Downloads
Seldon Server ⭐ 1,420
Machine Learning Platform and Recommendation Engine built on Kubernetes
total releases 44most recent commit 3 years ago
Optimus ⭐ 1,333
🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
total releases 27most recent commit 11 days ago
Spark Py Notebooks ⭐ 1,227
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
most recent commit 5 years ago
Fugue ⭐ 1,136
A unified interface for distributed computing. Fugue executes SQL, Python, and Pandas code on Spark, Dask and Ray without any rewrites.
dependent packages 9total releases 83most recent commit 2 days agopypi fugue} Downloads
Spark Sklearn ⭐ 1,039
(Deprecated) Scikit-learn integration package for Apache Spark
total releases 8most recent commit 3 years ago
Sparkling Water ⭐ 940
Sparkling Water provides H2O functionality inside Spark cluster
dependent packages 4total releases 163most recent commit a day ago
Around Dataengineering ⭐ 926
A Data Engineering & Machine Learning Knowledge Hub
most recent commit 3 months ago
Sparklyr ⭐ 886
R interface for Apache Spark
dependent packages 26total releases 46most recent commit 2 months ago
Sparkctr ⭐ 769
CTR prediction model based on spark(LR, GBDT, DNN)
most recent commit 3 years ago
Zingg ⭐ 685
Scalable identity resolution, entity resolution, data mastering and deduplication using ML
most recent commit 8 days ago
Machinelearning ⭐ 684
Machine learning resources，including algorithm, paper, dataset, example and so on.
most recent commit 3 years ago
Dist Keras ⭐ 611
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
total releases 2most recent commit 5 years ago
Sparklearning ⭐ 573
Learning Apache spark,including code and data .Most part can run local.
most recent commit a year ago
Onedal ⭐ 537
oneAPI Data Analytics Library (oneDAL)
dependent packages 3total releases 7most recent commit 3 days ago
Data Science Learning Resources ⭐ 490
A collection of data science and machine learning resources that I've found helpful (I only post what I've read!)
most recent commit 7 days ago
Featran ⭐ 461
A Scala feature transformation library for data science and machine learning
dependent packages 11total releases 34most recent commit 3 days ago
Agile_data_code_2 ⭐ 423
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
most recent commit a year ago
Machinelearning ⭐ 406
Machine Learning
most recent commit 6 years ago
Stockinference Spark ⭐ 376
Stock inference engine using Spring XD, Apache Geode / GemFire and Spark ML Lib.
most recent commit 7 years ago
Mlcraft ⭐ 369
Low-code metrics store, modern open-source alternative to Looker
most recent commit 2 days ago
Complete Life Cycle Of A Data Science Project ⭐ 369
Complete-Life-Cycle-of-a-Data-Science-Project
most recent commit 5 days ago
Ytk Learn ⭐ 349
Ytk-learn is a distributed machine learning library which implements most of popular machine learning algorithms(GBDT, GBRT, Mixture Logistic Regression, Gradient Boosting Soft Tree, Factorization Machines, Field-aware Factorization Machines, Logistic Regression, Softmax).
most recent commit 7 months ago
Sk Dist ⭐ 267
Distributed scikit-learn meta-estimators in PySpark
total releases 12most recent commit 2 years ago
Openuba ⭐ 264
A robust, and flexible open source User & Entity Behavior Analytics (UEBA) framework used for Security Analytics. Developed with luv by Data Scientists & Security Analysts from the Cyber Security Industry. [PRE-ALPHA]
most recent commit 2 months ago
Sagemaker Spark ⭐ 263
A Spark library for Amazon SageMaker.
total releases 36most recent commit 5 months ago
Jpmml Sparkml ⭐ 258
Java library and command-line application for converting Apache Spark ML pipelines to PMML
total releases 118most recent commit 8 months ago
Hydro Serving ⭐ 248
MLOps Platform
most recent commit a year ago
Data_science_blogs ⭐ 232
A repository to keep track of all the code that I end up writing for my blog posts.
most recent commit 2 months ago
Geni ⭐ 225
A Clojure dataframe library that runs on Spark
total releases 33most recent commit 9 months ago
Glow ⭐ 220
An open-source toolkit for large-scale genomic analysis
dependent packages 1total releases 8most recent commit 3 days ago
Rasterframes ⭐ 201
Geospatial Raster support for Spark DataFrames
dependent packages 3total releases 16most recent commit 8 days ago
Isolation Forest ⭐ 191
A Spark/Scala implementation of the isolation forest unsupervised outlier detection algorithm.
total releases 10most recent commit a month ago
Rumble ⭐ 183
⛈️ RumbleDB 1.20.0 "Honeylocust" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
total releases 4most recent commit 10 days ago
Book Resources ⭐ 181
most recent commit 5 years ago
Whylogs Java ⭐ 179
Profile and monitor your ML data pipeline end-to-end
dependent packages 2total releases 5most recent commit a year ago
Spark Ml Streaming ⭐ 175
Visualize streaming machine learning in Spark
most recent commit 6 years ago
Mydatascienceportfolio ⭐ 172
Applying Data Science and Machine Learning to Solve Real World Business Problems
most recent commit 3 years ago
Setl ⭐ 170
A simple Spark-powered ETL framework that just works 🍺
total releases 4most recent commit a month ago
Qb ⭐ 160
QANTA Quiz Bowl AI
most recent commit a year ago
Opaque Sql ⭐ 159
An encrypted data analytics platform
most recent commit a year ago
Cumf_als ⭐ 157
CUDA Matrix Factorization Library with Alternating Least Square (ALS)
most recent commit 4 years ago
Scalable Data Science Platform ⭐ 153
Content for architecting a data science platform for products using Luigi, Spark & Flask.
most recent commit 3 years ago
Spark Ext ⭐ 147
Spark Extension : ML transformers, SQL aggregations, etc that are missing in Apache Spark
most recent commit 7 years ago
Bigdata ⭐ 142
hadoop,hbase,storm,spark,etc..
most recent commit 3 years ago
Albedo ⭐ 142
A recommender system for discovering GitHub repos, built with Apache Spark
most recent commit 3 years ago
Ml ⭐ 135
sourced.ml is a library and command line tools to build and apply machine learning models on top of Universal Abstract Syntax Trees
dependent packages 1total releases 19most recent commit 4 years agopypi sourced-ml} Downloads
Sparkling Graph ⭐ 134
SparklingGraph provides easy to use set of features that will give you ability to proces large scala graphs using Spark and GraphX.
dependent packages 4total releases 7most recent commit 3 years ago
Data Science ⭐ 134
Projects and awesome list for all Data Science fields
most recent commit 2 days ago
Lift ⭐ 129
The LinkedIn Fairness Toolkit (LiFT) is a Scala/Spark library that enables the measurement of fairness in large scale machine learning workflows.
total releases 6most recent commit 2 years ago
Rikai ⭐ 127
Parquet-based ML data format optimized for working with unstructured data
dependent packages 5total releases 36most recent commit 2 months agopypi rikai} Downloads
Data Science Tutorials ⭐ 123
Python Tutorials for Data Science
most recent commit 16 days ago
Data Algorithms With Spark ⭐ 119
O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian
most recent commit 15 days ago
Spark Mllib Twitter Sentiment Analysis ⭐ 103
🌟 ✨ Analyze and visualize Twitter Sentiment on a world map using Spark MLlib
most recent commit 6 years ago
Ml Resource ⭐ 103
A concise resource repository for machine learning
most recent commit 4 months ago
Xlearning Xdml ⭐ 101
extremely distributed machine learning
most recent commit 3 years ago
Spark Nlp Models ⭐ 100
Models and Pipelines for the Spark NLP library
most recent commit a year ago
Spark With Python ⭐ 98
Fundamentals of Spark with Python (using PySpark), code examples
most recent commit 3 years ago
Machine Learning With Spark Second Edition ⭐ 92
Machine Learning with Spark - Second Edition, by Packt
most recent commit 2 years ago
Sift ⭐ 86
Knowledge extraction from web data
most recent commit 5 years ago
Pyspark2pmml ⭐ 85
Python library for converting Apache Spark ML pipelines to PMML
dependent packages 2total releases 2most recent commit 8 months agopypi pyspark2pmml} Downloads
Mloperator ⭐ 85
Machine Learning Operator & Controller for Kubernetes
most recent commit 20 days ago
Spark_python_ml_examples ⭐ 81
Spark 2.0 Python Machine Learning examples
most recent commit 3 years ago
Mleap ⭐ 76
MLeap allows for easily putting Spark ML pipelines into production
dependent packages 2total releases 6most recent commit 6 years ago
Resilient Ml Research Platform ⭐ 76
most recent commit 6 months ago
Spark_scala_ml_examples ⭐ 75
Spark 2.0 Scala Machine Learning examples
most recent commit 3 years ago
Ros_hadoop ⭐ 75
Hadoop splittable InputFormat for ROS. Process rosbag with Hadoop Spark and other HDFS compatible systems.
most recent commit 2 years ago
Generator Mitosis ⭐ 75
A micro-service infrastructure generator based on Yeoman/Chatbot, Kubernetes/Docker Swarm, Traefik, Ansible, Jenkins, Spark, Hadoop, Kafka, etc.
total releases 17most recent commit 2 years ago
Holoclean Legacy Deprecated ⭐ 74
A Machine Learning System for Data Enrichment.
most recent commit 4 years ago
Spark Redis Ml ⭐ 66
A spark package for loading Spark ML models to Redis-ML
most recent commit 4 years ago
Mltoolkits ⭐ 65
learningOrchestra is a distributed Machine Learning integration tool that facilitates and streamlines iterative processes in a Data Science project.
most recent commit 9 months ago
Frovedis ⭐ 64
Framework of vectorized and distributed data analytics
most recent commit 7 days ago
Pypmml ⭐ 64
Python PMML scoring library
dependent packages 5total releases 14most recent commit a month agopypi pypmml} Downloads
Pyspark Twitter Stream Mining ⭐ 63
Real-time Machine Learning with Apache Spark on Twitter Public Stream
most recent commit 6 years ago
W2v ⭐ 62
Word2Vec models with Twitter data using Spark. Blog:
most recent commit 4 years ago
Data On Eks ⭐ 62
DoEKS is a tool to build, deploy and scale Data Platforms on Amazon EKS
most recent commit 14 days ago
Awesome Ai Kubernetes ⭐ 62
❄️ 🐳 Awesome tools and libs for AI, Deep Learning, Machine Learning, Computer Vision, Data Science, Data Analytics and Cognitive Computing that are baked in the oven to be Native on Kubernetes and Docker with Python, R, Scala, Java, C#, Go, Julia, C++ etc
most recent commit 4 years ago
Spark ⭐ 60
Apache Spark (Scala, PySpark, SparkR) Code, Tricks, and References
most recent commit 4 years ago
Nlp_spark ⭐ 58
Natural Language Processing with Spark's MLlib
most recent commit 5 years ago
Bigdataanalytics_infoh515 ⭐ 56
Material for the Big Data Analytics exercise classes - INFOH515 - Big Data : Distributed Data Management and Scalable Analytics - Université Libre de Bruxelles
most recent commit a year ago
Playdata Zeppelin Notebook ⭐ 55
Zeppelin 화재 뉴스 기사 분류 예제
most recent commit 6 years ago
Cloud Bigdata Book ⭐ 53
write book
most recent commit 2 years ago
Cyber Matrix Ai ⭐ 53
Collection of cyber security and "AI" relevant topics
most recent commit 5 years ago
Pravda Ml ⭐ 52
This project is used to capture machine learning pipelines created on top of Spark as OK
total releases 9most recent commit 3 months ago
1-100 of 423 projects
Related Awesome Lists
Python Machine Learning Projects (14,099)
Jupyter Notebook Machine Learning Projects (10,674)
Machine Learning Deep Projects (7,043)
Machine Learning Artificial Intelligence Projects (3,887)
Machine Learning Neural Network Projects (3,522)
Machine Learning Data Science Projects (3,395)
Scala Spark Projects (3,279)
Machine Learning Natural Language Processing Projects (3,045)
Machine Learning Nlp Projects (3,045)
Machine Learning Tensorflow Projects (2,982)
Python Spark Projects (1,930)
Dataset Machine Learning Projects (1,872)
Machine Learning Pytorch Projects (1,834)
Machine Learning Computer Vision Projects (1,763)
Machine Learning Ml Projects (1,700)
Java Spark Projects (1,613)
Algorithms Machine Learning Projects (1,611)
Machine Learning Ai Projects (1,600)
Machine Learning Keras Projects (1,567)
Machine Learning Convolutional Neural Networks Projects (1,501)
Machine Learning Neural Projects (1,443)
Javascript Machine Learning Projects (1,257)
Machine Learning Scikit Learn Projects (1,236)
Apache Spark Projects (1,207)
Spark Hadoop Projects (1,188)
Jupyter Notebook Spark Projects (1,151)
Machine Learning Network Projects (1,148)
Html Machine Learning Projects (1,055)
Spark Kafka Projects (985)
R Machine Learning Projects (911)
Spark Streaming Projects (813)
Machine Learning Numpy Projects (803)
Docker Machine Learning Projects (793)
Java Machine Learning Projects (780)
Spark Pyspark Projects (773)
Machine Learning Deep Neural Networks Projects (763)
C Plus Plus Machine Learning Projects (758)
Machine Learning Data Analysis Projects (757)
Shell Spark Projects (705)
Machine Learning Reinforcement Projects (6)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,154,708)
Python  (888,323)
Java  (411,578)
Typescript  (304,714)
Php  (297,724)
C Plus Plus  (257,665)
Ruby  (225,073)
C  (193,605)
Shell  (182,260)
C Sharp  (178,205)
Golang  (171,741)
Swift  (68,947)
Rust  (68,512)
R  (62,080)
Dart  (61,781)
Objective C  (57,059)
Kotlin  (53,231)
Lua  (38,349)
Matlab  (32,815)
Perl  (30,816)
Language  (29,928)
Scala  (29,370)
Haskell  (20,925)
Powershell  (20,379)
Clojure  (19,555)
Bash  (19,201)
Elixir  (17,444)
Coffeescript  (17,147)
Assembly  (15,654)
Julia  (13,994)
Processing  (13,325)
Emacs Lisp  (10,848)
Basic  (10,674)
Flow  (10,509)
Solidity  (10,029)
Erlang  (9,124)
Groovy  (8,612)
Scheme  (8,437)
Ocaml  (6,951)
Pascal  (6,798)
Hack  (6,596)
Lisp  (6,356)
Fortran  (5,792)
Elm  (5,567)
Common Lisp  (5,467)
Shell Script  (4,835)
Dlang  (4,803)
Python Library  (4,516)
Crystal  (4,443)
F Sharp  (4,260)
Privacy | About | Terms | Follow Us On Twitter | Top Projects

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2023 Awesome Open Source.  All rights reserved.