#tokenizer

Question 1: Single choice

You load a pretrained GPT-2-style code model and want to use a different tokenizer with it. What must the replacement tokenizer satisfy for the pretrained model to work?

It must map text to the same token ids as the original vocabulary, since each id selects a learned embedding row1
It must have been trained on the same amount of text, whatever its vocabulary looks like2
It only needs to produce fewer tokens per sentence than the original tokenizer3
It must use the WordPiece algorithm, because GPT-style models cannot read BPE tokens4

Timed test