REALM —📒Pre-Learning Language Model For Knowledge Extraction📒

REALM is a method for pre-training language models. With REALM from a pre-trained model, you can extract knowledge about the world directly from raw documents. In this case, the model does not need to store facts in scales. The essence of the approach is that the word representation model is trained together with the knowledge extraction model. Researchers have posted a project code for replicating experiments.

Source: Google AI Blog