⚡Hivemind⚡ — Python Library for Decentralized Neural Network Training

Hivemind is a library for decentralized training of large neural networks (the Decentralized Mixture of Experts (DMoE)). The library functionality provides distributed training of models on the user equipment. At the moment, the library is at the pre-alpha stage.

Hivemind — Python Library for Decentralized Neural Network Training. Source: Github

Description of the problem

Larger neural networks are state-of-the-art in both computer vision tasks and natural language processing tasks. Examples include:

• Pre-trained transformers — for NLP;
• Convolutional neural networks — for computer vision;
• GPT-3 with 175 billion parameters

Transfer learning allows you to improve the quality of models with an increase in the training sample. However, training such large models is limited by the limited computational resources. One GPT-3 training costs $4.6 million (!) in cloud GPUs. For example, only limited research teams in corporations can often contribute to the development of state-of-the-art deep learning models. Hivemind allows you to bypass this limitation and decentralize the training of large neural networks.

More about Hivemind

Hivemind uses a Decentralized Mixture of Experts (DMoE) layer to allow a group of…