MACHINE LEARNING

CowMask — Data Augmentation for Self-Supervised Models

Google Research recently published CowMask — a new SOTA masking-based augmentation method. The method allows training more stable self-supervised models.

CowMask — Data Augmentation for Self-Supervised Models
CowMask — Data Augmentation for Self-Supervised Models
CowMask — Data Augmentation for Self-Supervised Models. Source: Arxiv
CowMask results on CIFAR-100 test set, error rates as mean ± stdev of 5 independent runs. Source: Arxiv
CowMask results on CIFAR-100 test set, error rates as mean ± stdev of 5 independent runs. Source: Arxiv
Results on CIFAR-100 test set, error rates as mean ± stdev of 5 independent runs. Source: Arxiv

Consistency in semi-supervised training

Consistency regularization is a semi-supervised training technique that trains a model using a small amount of labeled data. It works in such a way that the model is trained to resist changes in unlabeled data.

How CowMask works

The researchers adapted the Mean Teacher framework and used it as a base for the approach. They used two networks: a student network and a teacher network. Both models predict vectors with probabilities for each class. The student network is trained in a standard manner using gradient descent. After each update of the parameters, the weights of the teacher network are updated so that they are an exponential moving average of the weights of the student model. At the same time, there is a hyperparameter with the help of which the stability and speed of how the teacher network follows the student network are regulated.

CowMask generation algorithm
CowMask generation algorithm
CowMask generation algorithm. Source: Arxiv

Original Paper

Interested in Deep Learning?

If you found this article helpful, click the💚 or 👏 button below or share the article on Facebook so your friends can benefit from it too.

Written by

Bioinformatician at Oncobox Inc. (@oncobox). Research Associate at Moscow Institute of Physics and Technology (@mipt_eng).

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store