Skip to content
ml-recall

#vocabulary

8 questions from Tokenization, Language Modeling, NLP Tasks

Timed test

8 questions · 8 min · score and explanations at the end

Take a test
Q 1 / 8Score 0 / 0

Question 1: Short answer

easyAI-checkedTokenization

A vocabulary class keeps every token that occurs at least min_freq times and always adds an unknown token:

counter = collections.Counter(tokens)
self.idx_to_token = list(sorted(set(['<unk>'] + reserved_tokens + [
    token for token, freq in counter.items() if freq >= min_freq])))

A word-level corpus has these counts: the 9, time 4, machine 3, traveller 3, said 2, hour 1, glass 1. How many entries does the vocabulary have with min_freq=3 and no reserved tokens?

a whole number