#pre-tokenization

Question 1: Short answer

A WordPiece tokenizer is set up with this normalizer and pre-tokenizer (Hugging Face tokenizers):

tokenizer.normalizer = normalizers.Sequence([
    normalizers.Replace(Regex(r"[\s]"), " "),
    normalizers.Lowercase(), normalizers.NFD(), normalizers.StripAccents()])
tokenizer.pre_tokenizer = pre_tokenizers.Sequence([
    pre_tokenizers.WhitespaceSplit(),
    pre_tokenizers.Digits(individual_digits=True),
    pre_tokenizers.Punctuation()])

How many pre-tokens (pieces handed on to the WordPiece model) does the input Don't stop at 23.5% produce?

Timed test