Awesome Open Source
Awesome Open Source

hivemind: decentralized deep learning in PyTorch

Build status Documentation Status Gitter

Hivemind is a PyTorch library to train large neural networks across the Internet. Imagine training one huge Transformer model on thousands of computers from different universities, companies, and volunteers.

img

Key Features

  • Train neural networks of arbitrary size: parts of their layers are distributed across the participants
  • Run distributed training without master node: Distributed Hash Table allows to connect computers in a decentralized network
  • Fault-tolerant backpropagation: forward and backward passes succeed even if some nodes are unresponsive or take too long to respond

To learn more about the idea behind this library and its components, see https://learning-at-home.github.io or read the NeurIPS 2020 paper

Documentation

Contributing

Hivemind is currently at the active development stage, and we welcome all contributions from bug fixes and documentation improvements to entirely new features. If you want to contribute to hivemind, take a look at the issues or join our chat room. The Developer's guide page contains best practices, as well as description of tests and performance benchmarks.

References

You can read the paper that inspired hivemind here:

Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts (Max Ryabinin and Anton Gusev, NeurIPS 2020).

@misc{ryabinin2020crowdsourced,
      title={Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts}, 
      author={Max Ryabinin and Anton Gusev},
      year={2020},
      eprint={2002.04013},
      archivePrefix={arXiv},
      primaryClass={cs.DC}
}

The initial implementation of hivemind used to conduct experiments for the paper is available here: mryab/learning-at-home.

In the docs, we list several related projects and acknowledgements.


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
python (50,799
deep-learning (3,756
pytorch (2,195
deep-neural-networks (449
distributed-systems (282
asyncio (261
dht (47
asynchronous-programming (27

Find Open Source By Browsing 7,000 Topics Across 59 Categories