Awesome Open Source

Programming Languages

Search results for llm serving

24 search results found

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

Vllm ⭐ 13,832

A high-throughput and memory-efficient inference and serving engine for LLMs

Openllm ⭐ 7,871

Operating LLMs in production

Skypilot ⭐ 4,975

SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.

Superduperdb ⭐ 3,924

🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalable model training and vector search.

Llm Action ⭐ 2,955

本项目旨在分享大模型相关技术原理以及实战经验。

Ray Llm ⭐ 972

RayLLM - LLMs on Ray

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

Swiftinfer ⭐ 277

Efficient AI Inference & Serving

Ray Educational Materials ⭐ 232

This is suite of the hands-on training materials that shows how to scale CV, NLP, time-series forecasting workloads with Ray.

Rtp Llm ⭐ 157

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Runbooks ⭐ 151

Finetune LLMs on K8s by using Runbooks

Ialacol ⭐ 127

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

Llm Finetuning Large Language Models ⭐ 120

LLM (Large Language Model) FineTuning

Sugarcane Ai ⭐ 39

npm like package ecosystem for Prompts 🤖

Friendli Client ⭐ 23

Friendli: the fastest serving engine for generative AI such as LLMs

Sugarcane Ai.github.io ⭐ 13

Llm Inference Solutions ⭐ 12

A collection of all available inference solutions for the LLMs

Awesome Llm Productization ⭐ 6

Awesome-LLM-Productization: a curated list of tools/tricks/news/regulations about AI and Large Language Model (LLM) productization

Llms In Prod Workshop 2023 ⭐ 6

Deploy and Scale LLM-based applications

Anyscale Berkeley Ai Hackathon ⭐ 5

Ray and Anyscale for UC Berkeley AI Hackathon!

A self-hosted personal chatbot API with FastAPI. It allows you to interact with the Llama2 LLM (and other open-source LLMs) to have natural language conversations, generate text, and perform various language-related tasks.

Ray_vllm_inference ⭐ 5

A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.

Related Searches

Python Llm Serving (12)

Jupyter Notebook Llm Serving (8)

Pytorch Llm Serving (5)

Large Language Models Llm Serving (4)

Llama2 Llm Serving (3)

Llms Llm Serving (3)

Llamacpp Llm Serving (3)

1-24 of 24 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.