Skip to content
ml-recall

#utf-8

3 questions from Tokenization

Timed test

3 questions · 3 min · score and explanations at the end

Take a test
Q 1 / 3Score 0 / 0

Question 1: Short answer

mediumAI-checkedTokenization

A byte-level BPE tokenizer starts from the 256 byte values (ASCII: 116 = t, 104 = h, 101 = e, 32 = space) and learned, among others, these merges:

  • (101, 32) → token 256
  • (116, 104) → token 259
  • (259, 256) → token 275

How many bytes of text does token 275 stand for?

a whole number