Question 1: Short answer
easyAI-checkedTokenizationA word-level tokenizer has this vocabulary and encode method; word_tokenize splits words and
punctuation into separate tokens.
vocab = {"<start>": 0, "<pad>": 1, "<unk>": 2, "to": 3, "be": 4,
"or": 5, "not": 6, ".": 7}
def encode(text):
ids = [vocab["<start>"]]
for tok in word_tokenize(text):
ids.append(vocab.get(tok, vocab["<unk>"]))
return ids
What does encode("to be or never.") return?