Skip to content
ml-recall

#unicode

1 question from Tokenization

Timed test

1 question · 1 min · score and explanations at the end

Take a test
Q 1 / 1Score 0 / 0

Question 1: Short answer

easyAI-checkedTokenization

Byte-level BPE first encodes the text as UTF-8 and starts from one token per byte. In UTF-8, ASCII letters take 1 byte and "é" takes 2 bytes. How many initial tokens does the word "café" produce, before any merge?

a whole number