Skip to content
ml-recall

#tiktoken

3 questions from Tokenization

Timed test

3 questions · 3 min · score and explanations at the end

Take a test
Q 1 / 3Score 0 / 0

Question 1: Short answer

mediumAI-checkedTokenization
import tiktoken

tokenizer = tiktoken.get_encoding("gpt2")
ids = tokenizer.encode("<|endoftext|>", allowed_special={"<|endoftext|>"})
print(ids, tokenizer.n_vocab)

The last line prints [ ] and then .

Fill in every blank (hints in the boxes). Enter moves to the next blank.