Awesome Open Source

Programming Languages

Search results for python llm serving

13 search results found

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

Vllm ⭐ 13,832

A high-throughput and memory-efficient inference and serving engine for LLMs

Openllm ⭐ 7,871

Operating LLMs in production

Skypilot ⭐ 4,975

SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.

Superduperdb ⭐ 3,924

🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalable model training and vector search.

Llm Action ⭐ 2,955

本项目旨在分享大模型相关技术原理以及实战经验。

Ray Llm ⭐ 972

RayLLM - LLMs on Ray

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

Swiftinfer ⭐ 277

Efficient AI Inference & Serving

Ialacol ⭐ 127

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

Friendli Client ⭐ 23

Friendli: the fastest serving engine for generative AI such as LLMs

Ray_vllm_inference ⭐ 5

A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.

Related Searches

Python Django (28,897)

Python Machine Learning (20,195)

Python Flask (17,643)

Python Dataset (14,792)

Python Docker (14,113)

Python Tensorflow (13,736)

Python Command Line (13,351)

Python Deep Learning (13,092)

Python Jupyter Notebook (12,976)

Python Network (11,495)

1-13 of 13 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.