Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Model Serving Open Source Projects

Open source projects categorized as Model Serving
Categories > Model Serving
Vllm ⭐ 13,832
A high-throughput and memory-efficient inference and serving engine for LLMs
dependent packages 14total releases 14most recent commit 5 months agopypi vllm} Downloads
Bentoml ⭐ 6,575
The most flexible way to serve AI/ML models in production - Build Model Inference Service, LLM APIs, Inference Graph/Pipelines, Compound AI systems, Multi-Modal, RAG as a Service, and more!
dependent packages 12total releases 119most recent commit 2 months agopypi bentoml} Downloads
Deep Learning In Production ⭐ 4,138
In this repository, I will share some useful notes and references about deploying deep learning-based models in production.
most recent commit 8 months ago
Fedml ⭐ 3,946
FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, FEDML Nexus AI (https://nexus.fedml.ai) is the dedicated cloud service for generative AI
dependent packages 1total releases 342most recent commit 5 months agopypi fedml} Downloads
Kserve ⭐ 3,239
Standardized Serverless ML Inference Platform on Kubernetes
dependent packages 9total releases 20most recent commit 6 days agopypi kserve} Downloads
Envd ⭐ 1,869
🏕️ Reproducible development environment
dependent packages 2total releases 103most recent commit 5 months agopypi envd} Downloads
Lightllm ⭐ 1,417
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
most recent commit 5 months ago
Mlrun ⭐ 1,177
Machine Learning automation and tracking
total releases 486most recent commit 5 months ago
Hopsworks ⭐ 1,041
Hopsworks - Data-Intensive AI platform with a Feature Store
total releases 1most recent commit 5 months ago
Truss ⭐ 776
The simplest way to serve AI/ML models in production
dependent packages 2total releases 163most recent commit 5 months agopypi truss} Downloads
Yatai ⭐ 748
Model Deployment at Scale on Kubernetes 🦄️
total releases 125most recent commit 4 months ago
Lorax ⭐ 719
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
most recent commit 5 months ago
Mosec ⭐ 661
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
dependent packages 1total releases 40most recent commit 5 months agopypi mosec} Downloads
Model_server ⭐ 618
A scalable inference server for models optimized with OpenVINO™
dependent packages 4total releases 6most recent commit 5 months agopypi ovmsclient} Downloads
Pinferencia ⭐ 473
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
total releases 15most recent commit a year ago
Stable Diffusion Deploy ⭐ 373
Learn to serve Stable Diffusion models on cloud infrastructure at scale. This Lightning App shows load-balancing, orchestrating, pre-provisioning, dynamic batching, GPU-inference, micro-services working together via the Lightning Apps framework.
most recent commit 9 months ago
Fastapi Ml Skeleton ⭐ 307
FastAPI Skeleton App to serve machine learning models production-ready.
most recent commit 6 months ago
Onediffusion ⭐ 293
OneDiffusion: Run any Stable Diffusion models and fine-tuned weights with ease
total releases 2most recent commit 5 months ago
Chitra ⭐ 219
A multi-functional library for full-stack Deep Learning. Simplifies Model Building, API development, and Model Deployment.
dependent packages 1total releases 38most recent commit 7 months agopypi chitra} Downloads
Kafka With Akka Streams Kafka Streams Tutorial ⭐ 191
Code samples for the Lightbend tutorial on writing microservices with Akka Streams, Kafka Streams, and Kafka
most recent commit 5 years ago
Rtp Llm ⭐ 157
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
most recent commit 5 months ago
Zoltar ⭐ 140
Common library for serving TensorFlow, XGBoost and scikit-learn models in production.
dependent packages 12total releases 17most recent commit a year ago
Fate Serving ⭐ 131
A scalable, high-performance serving system for federated learning models
most recent commit 8 months ago
The 10 Most Depended On Model Serving Open Source Projects
Vllm ⭐ 13,832
A high-throughput and memory-efficient inference and serving engine for LLMs
dependent packages 14total releases 14latest release December 03, 2023most recent commit 5 months agopypi vllm} Downloads
Zoltar ⭐ 140
Common library for serving TensorFlow, XGBoost and scikit-learn models in production.
dependent packages 12total releases 17latest release April 21, 2022most recent commit a year ago
Kserve ⭐ 3,239
Standardized Serverless ML Inference Platform on Kubernetes
dependent packages 9total releases 20latest release November 17, 2023most recent commit 6 days agopypi kserve} Downloads
Model_server ⭐ 618
A scalable inference server for models optimized with OpenVINO™
dependent packages 4total releases 6latest release September 18, 2023most recent commit 5 months agopypi ovmsclient} Downloads
Envd ⭐ 1,869
🏕️ Reproducible development environment
dependent packages 2total releases 103latest release September 01, 2023most recent commit 5 months agopypi envd} Downloads
Truss ⭐ 776
The simplest way to serve AI/ML models in production
dependent packages 2total releases 163latest release December 07, 2023most recent commit 5 months agopypi truss} Downloads
Surround ⭐ 22
Surround is a framework for building AI driven microservices in Python, https://surround.readthedocs.io/en/latest/
dependent packages 1total releases 17latest release August 30, 2021most recent commit a year agopypi surround} Downloads
Mosec ⭐ 661
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
dependent packages 1total releases 40latest release December 05, 2023most recent commit 5 months agopypi mosec} Downloads
The 10 Latest Releases In Model Serving Open Source Projects
Truss ⭐ 776
The simplest way to serve AI/ML models in production
dependent packages 2total releases 163latest release December 07, 2023most recent commit 5 months agopypi truss} Downloads
Mlrun ⭐ 1,177
Machine Learning automation and tracking
total releases 486latest release December 07, 2023most recent commit 5 months ago
Mosec ⭐ 661
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
dependent packages 1total releases 40latest release December 05, 2023most recent commit 5 months agopypi mosec} Downloads
Vllm ⭐ 13,832
A high-throughput and memory-efficient inference and serving engine for LLMs
dependent packages 14total releases 14latest release December 03, 2023most recent commit 5 months agopypi vllm} Downloads
Fastdeploy ⭐ 90
Deploy DL/ ML inference pipelines with minimal extra code.
total releases 56latest release November 29, 2023most recent commit 7 months ago
Machine Learning Api ⭐ 7
Hopsworks Machine Learning Api 🚀 Model management with a model registry and model serving
dependent packages 1total releases 29latest release November 23, 2023most recent commit 6 months agopypi hsml} Downloads
Kserve ⭐ 3,239
Standardized Serverless ML Inference Platform on Kubernetes
dependent packages 9total releases 20latest release November 17, 2023most recent commit 6 days agopypi kserve} Downloads
Console ⭐ 20
⛅ Versatile Data Pipeline (VDP) console website
total releases 4latest release November 15, 2023most recent commit 5 months ago
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.