#next-token-prediction

The book's dataset cuts a tokenized text into input/target pairs with a sliding window:

token_ids = list(range(100, 110))          # 10 token IDs
max_length, stride = 4, 2
inputs, targets = [], []
for i in range(0, len(token_ids) - max_length, stride):
    inputs.append(token_ids[i:i + max_length])
    targets.append(token_ids[i + 1:i + max_length + 1])
print(len(inputs), targets[1])

The last line prints (the number of pairs) and a list whose first element is .