Question 1: Single choice
mediumAI-checkedTokenizationA Unigram tokenizer has these token probabilities:
| token | s | e | a | t | se | at | sea | eat |
|---|---|---|---|---|---|---|---|---|
| 0.03 | 0.1 | 0.1 | 0.02 | 0.02 | 0.1 | 0.05 | 0.03 |
It picks the segmentation with the highest product of token probabilities. How does it segment "seat"?