⚡Hivemind⚡ — Python Library for Decentralized Neural Network Training
Hivemind is a library for decentralized training of large neural networks (the Decentralized Mixture of Experts (DMoE)). The library functionality provides distributed training of models on the user equipment. At the moment, the library is at the pre-alpha stage.
Description of the problem
Larger neural networks are state-of-the-art in both computer vision tasks and natural language processing tasks. Examples include:
- Pre-trained transformers — for NLP;
- Convolutional neural networks — for computer vision;
- GPT-3 with 175 billion parameters
Transfer learning allows you to improve the quality of models with an increase in the training sample. However, training such large models is limited by the limited computational resources. One GPT-3 training costs $4.6 million (!) in cloud GPUs. For example, only limited research teams in corporations can often contribute to the development of state-of-the-art deep learning models. Hivemind allows you to bypass this limitation and decentralize the training of large neural networks.
More about Hivemind
Hivemind uses a Decentralized Mixture of Experts (DMoE) layer to allow a group of users to distribute computation across machines.
In hivemind, all users:
- ➡️One or more models are adopted depending on the characteristics of their iron;
- ➡️ Participate in asynchronous model training and receive models from other users;
- ➡️ Form a Distributed Hash Table to find out each other’s models. This is the same type of protocol that BitTorrent uses to distribute files.
Hivemind uses Kademlia-based DHT, which scales to tens of thousands of users with log search complexity.
If you found this article helpful, click the💚 or 👏 button below or share the article on Facebook so your friends can benefit from it too.