Skip to content
ml-recall

#unk-token

2 questions from Tokenization

Timed test

2 questions · 2 min · score and explanations at the end

Take a test
Q 1 / 2Score 0 / 0

Question 1: Short answer

easyAI-checkedTokenization

A vocabulary class keeps every token that occurs at least min_freq times and always adds an unknown token:

counter = collections.Counter(tokens)
self.idx_to_token = list(sorted(set(['<unk>'] + reserved_tokens + [
    token for token, freq in counter.items() if freq >= min_freq])))

A word-level corpus has these counts: the 9, time 4, machine 3, traveller 3, said 2, hour 1, glass 1. How many entries does the vocabulary have with min_freq=3 and no reserved tokens?

a whole number