Skip to content
ml-recall

#merge-rules

1 question from Tokenization

Timed test

1 question · 1 min · score and explanations at the end

Take a test
Q 1 / 1Score 0 / 0

Question 1: Single choice

mediumAI-checkedTokenization

A BPE tokenizer learned these merges, in this order, from a corpus of "low", "lowest", "newer", "wider" and "new":

  1. e r → er
  2. n e → ne
  3. ne w → new
  4. l o → lo
  5. lo w → low
  6. new er → newer

It tokenizes a new word by splitting it into characters and applying the merges in the order they were learned. How is the unseen word "lowered" tokenized?

Choose one option