⚡Hivemind⚡ — Python Library for Decentralized Neural Network Training

Hivemind is a library for decentralized training of large neural networks (the Decentralized Mixture of Experts (DMoE)). The library functionality provides distributed training of models on the user equipment. At the moment, the library is at the pre-alpha stage.

⚡Hivemind⚡ — Python Library for Decentralized Neural Network Training
⚡Hivemind⚡ — Python Library for Decentralized Neural Network Training
Hivemind — Python Library for Decentralized Neural Network Training. Source: Github

Description of the problem

Larger neural networks are state-of-the-art in both computer vision tasks and natural language processing tasks. Examples include:

  • Pre-trained transformers — for NLP;
  • Convolutional neural networks — for computer vision;
  • GPT-3 with 175 billion parameters

Transfer learning allows you to improve the quality of models with an increase in the training sample. However, training such large models is limited by the limited computational resources. One GPT-3 training costs $4.6 million (!) in cloud GPUs. For example, only limited research teams in corporations can often contribute to the development of state-of-the-art deep learning models. Hivemind allows you to bypass this limitation and decentralize the training of large neural networks.

More about Hivemind

Hivemind uses a Decentralized Mixture of Experts (DMoE) layer to allow a group of users to distribute computation across machines.

Image for post
Image for post
DMoE layers are inherently fault-tolerant: if some of the chosen experts fail to respond, the model will simply average the remaining ones and call that dropout. Source: Github

In hivemind, all users:

  • ➡️One or more models are adopted depending on the characteristics of their iron;
  • ➡️ Participate in asynchronous model training and receive models from other users;
  • ➡️ Form a Distributed Hash Table to find out each other’s models. This is the same type of protocol that BitTorrent uses to distribute files.

Hivemind uses Kademlia-based DHT, which scales to tens of thousands of users with log search complexity.

Video demonstration

Read More

If you found this article helpful, click the💚 or 👏 button below or share the article on Facebook so your friends can benefit from it too.

Source: Github

Written by

Bioinformatician at Oncobox Inc. (@oncobox). Research Associate at Moscow Institute of Physics and Technology (@mipt_eng).

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store