Skip to content
ml-recall

#detokenization

1 question from Tokenization

Timed test

1 question · 1 min · score and explanations at the end

Take a test
Q 1 / 1Score 0 / 0

Question 1: Single choice

mediumAI-checkedTokenization

SentencePiece tokenizes "Hello world." as [Hello] [▁wor] [ld] [.], writing the space before "world" as the symbol ▁ inside a token. What does keeping the whitespace as ▁ achieve?

Choose one option