Skip to content
ml-recall

#sentencepiece

2 questions from Tokenization

Timed test

2 questions · 2 min · score and explanations at the end

Take a test
Q 1 / 2Score 0 / 0

Question 1: Single choice

mediumAI-checkedTokenization

SentencePiece tokenizes "Hello world." as [Hello] [▁wor] [ld] [.], writing the space before "world" as the symbol ▁ inside a token. What does keeping the whitespace as ▁ achieve?

Choose one option