Question 1: Single choice
mediumAI-checkedTokenizationA BPE tokenizer learned these merges, in this order, from a corpus of "low", "lowest", "newer", "wider" and "new":
e r→ern e→nene w→newl o→lolo w→lownew er→newer
It tokenizes a new word by splitting it into characters and applying the merges in the order they were learned. How is the unseen word "lowered" tokenized?