#vocabulary-pruning

Question 1: Single choice

Unigram tokenizer training starts from a large seed vocabulary and repeatedly removes the tokens whose removal increases the corpus loss least, but it never removes single-character tokens. Why?

So that any word written with the training alphabet can still be segmented1
Because single characters always have the highest unigram probability2
Because removing them would make the vocabulary grow instead of shrink3
Because the EM algorithm cannot estimate probabilities for single characters4

Timed test