Skip to content
ml-recall

#longest-match

1 question from Tokenization

Timed test

1 question · 1 min · score and explanations at the end

Take a test
Q 1 / 1Score 0 / 0

Question 1: Single choice

mediumAI-checkedTokenization

Two tokenizers share the vocabulary {t, e, s, es, st, te}. The BPE tokenizer learned its merges in this order: (1) e s → es, (2) s t → st, (3) t e → te. The WordPiece-style tokenizer instead takes, again and again, the longest vocabulary entry that starts at the current position (ignore ## prefixes). How do they tokenize "test"?

Choose one option