Skip to content
ml-recall

#unknown-token

3 questions from Tokenization

Timed test

3 questions · 3 min · score and explanations at the end

Take a test
Q 1 / 3Score 0 / 0

Question 1: Short answer

easyAI-checkedTokenization

A word-level tokenizer has this vocabulary and encode method; word_tokenize splits words and punctuation into separate tokens.

vocab = {"<start>": 0, "<pad>": 1, "<unk>": 2, "to": 3, "be": 4,
         "or": 5, "not": 6, ".": 7}

def encode(text):
    ids = [vocab["<start>"]]
    for tok in word_tokenize(text):
        ids.append(vocab.get(tok, vocab["<unk>"]))
    return ids

What does encode("to be or never.") return?

a list of ids, e.g. [0, 1, 2]