🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
Alternatives To Petals
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Tensorflow179,02432778a day ago46October 23, 20192,093apache-2.0C++
An Open Source Machine Learning Framework for Everyone
Transformers115,925642,452a day ago125November 15, 2023917apache-2.0Python
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Pytorch72,9363,3418,140a day ago39November 15, 202313,295otherPython
Tensors and Dynamic neural networks in Python with strong GPU acceleration
Cs Video Courses61,557
16 days ago2
List of Computer Science courses with video lectures.
Keras59,7986792 days ago85November 07, 2023123apache-2.0Python
Deep Learning for humans
D2l Zh51,074116 days ago51August 18, 202359apache-2.0Python
6 days ago19gpl-3.0Python
Deepfakes Software For All
a month ago547gpl-3.0Python
DeepFaceLab is the leading software for creating deepfakes.
5 days ago8September 21, 2021199agpl-3.0Python
YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite
Tensorflow Examples42,312
a year ago218otherJupyter Notebook
TensorFlow Tutorial and Examples for Beginners (support TF v1 & v2)
Alternatives To Petals
Select To Compare

Alternative Project Comparisons

Run large language models at home, BitTorrent-style.
Fine-tuning and inference up to 10x faster than offloading

Generate text with distributed Llama 2 (70B), Falcon (40B+), BLOOM (176B) (or their derivatives), and finetune them for your own tasks — right from your desktop computer or Google Colab:

from transformers import AutoTokenizer
from petals import AutoDistributedModelForCausalLM

# Choose any model available at
model_name = "petals-team/StableBeluga2"  # This one is fine-tuned Llama 2 (70B)

# Connect to a distributed network hosting model layers
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoDistributedModelForCausalLM.from_pretrained(model_name)

# Run the model as if it were on your computer
inputs = tokenizer("A cat sat", return_tensors="pt")["input_ids"]
outputs = model.generate(inputs, max_new_tokens=5)
print(tokenizer.decode(outputs[0]))  # A cat sat on a mat...

 Try now in Colab

Privacy. Your data will be processed with the help of other people in the public swarm. Learn more about privacy here. For sensitive data, you can set up a private swarm among people you trust.

Want to run Llama 2? Request access to its weights at the Meta AI website and Model Hub, then run huggingface-cli login in the terminal before loading the model. Or just try it in our chatbot app.

Any questions? Ping us in our Discord!

Connect your GPU and increase Petals capacity

Petals is a community-run system — we rely on people sharing their GPUs. You can check out available models and help serving one of them! As an example, here is how to host a part of Stable Beluga 2 on your GPU:

Linux + Anaconda. Run these commands for NVIDIA GPUs (or follow this for AMD):

conda install pytorch pytorch-cuda=11.7 -c pytorch -c nvidia
pip install git+
python -m petals.cli.run_server petals-team/StableBeluga2

Windows + WSL. Follow this guide on our Wiki.

Docker. Run our Docker image for NVIDIA GPUs (or follow this for AMD):

sudo docker run -p 31330:31330 --ipc host --gpus all --volume petals-cache:/cache --rm \
    learningathome/petals:main \
    python -m petals.cli.run_server --port 31330 petals-team/StableBeluga2

macOS + Apple M1/M2 GPU. Install Homebrew, then run these commands:

brew install python
python3 -m pip install git+
python3 -m petals.cli.run_server petals-team/StableBeluga2

 Learn more (how to use multiple GPUs, start the server on boot, etc.)

Any questions? Ping us in our Discord!

Want to host Llama 2? Request access to its weights at the Meta AI website and Model Hub, generate an access token, then add --token YOUR_TOKEN_HERE to the python -m petals.cli.run_server command.

Security. Hosting a server does not allow others to run custom code on your computer. Learn more here.

Thank you! Once you load and host 10+ blocks, we can show your name or link on the swarm monitor as a way to say thanks. You can specify them with --public_name YOUR_NAME.

How does it work?

  • You load a small part of the model, then join a network of people serving the other parts. Singlebatch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B) enough for chatbots and interactive apps.
  • You can employ any fine-tuning and sampling methods, execute custom paths through the model, or see its hidden states. You get the comforts of an API with the flexibility of PyTorch and ** Transformers**.

 Read paper             See FAQ

Tutorials, examples, and more

Basic tutorials:

  • Getting started: tutorial
  • Prompt-tune Llama-65B for text semantic classification: tutorial
  • Prompt-tune BLOOM to create a personified chatbot: tutorial

Useful tools:

Advanced guides:

  • Launch a private swarm: guide
  • Run a custom model: guide


Please see Section 3.3 of our paper.


Please see our FAQ on contributing.


Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative Inference and Fine-tuning of Large Models. arXiv preprint arXiv:2209.01188, 2022.

  title = {Petals: Collaborative Inference and Fine-tuning of Large Models},
  author = {Borzunov, Alexander and Baranchuk, Dmitry and Dettmers, Tim and Ryabinin, Max and Belkada, Younes and Chumachenko, Artem and Samygin, Pavel and Raffel, Colin},
  journal = {arXiv preprint arXiv:2209.01188},
  year = {2022},
  url = {}

This project is a part of the BigScience research workshop.

Popular Machine Learning Projects
Popular Deep Learning Projects
Popular Machine Learning Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Machine Learning
Deep Learning
Natural Language Processing
Neural Network
Distributed Systems
Language Model
Pretrained Models