Skip to content
ml-recall

#pair-score

1 question from Tokenization

Timed test

1 question · 1 min · score and explanations at the end

Take a test
Q 1 / 1Score 0 / 0

Question 1: Single choice

hardAI-checkedTokenization

WordPiece (as described in the Hugging Face course) splits each word into a first character and ##-prefixed continuation characters, e.g. s ##p ##i ##n, and merges the pair with the highest

score=freq(pair)freq(first)×freq(second).\mathrm{score} = \frac{\mathrm{freq}(\text{pair})}{\mathrm{freq}(\text{first}) \times \mathrm{freq}(\text{second})}.

The corpus has the word counts pin 6, pins 2, spin 3, tin 4 and zip 1. Which pair is merged first?

Choose one option