Skip to content
ml-recall

#huggingface-tokenizers

1 question from Tokenization

Timed test

1 question · 1 min · score and explanations at the end

Take a test
Q 1 / 1Score 0 / 0

Question 1: Short answer

mediumAI-checkedTokenization

A WordPiece tokenizer is set up with this normalizer and pre-tokenizer (Hugging Face tokenizers):

tokenizer.normalizer = normalizers.Sequence([
    normalizers.Replace(Regex(r"[\s]"), " "),
    normalizers.Lowercase(), normalizers.NFD(), normalizers.StripAccents()])
tokenizer.pre_tokenizer = pre_tokenizers.Sequence([
    pre_tokenizers.WhitespaceSplit(),
    pre_tokenizers.Digits(individual_digits=True),
    pre_tokenizers.Punctuation()])

How many pre-tokens (pieces handed on to the WordPiece model) does the input Don't stop at 23.5% produce?

an integer