#masked-language-model

Question 1: Single choice

All else being equal, does a masked language model (BERT, which predicts the 15% of tokens selected for masking) need more or fewer pretraining steps to converge than a left-to-right language model?

More: each sequence gives targets at only ~15% of positions, a left-to-right model at all of them1
Fewer: seeing context on both sides makes each prediction easier to learn2
Fewer: random masking acts as data augmentation, so every training step counts several times over3
The same, because both read the same number of tokens in each step4

#masked-language-model

Timed test