#language-modeling

Question 1: Single choice

mediumAI-checkedPyTorch

A GPT-style model returns logits of shape (B, T, V) for input_ids of shape (B, T), and every row of input_ids starts with a <start> token. The logits at position i must predict the token at position i + 1. Which pair do you pass to F.cross_entropy (after flattening)?

logits[:, :-1] with targets input_ids[:, 1:]1
logits[:, 1:] with targets input_ids[:, :-1]2
logits with targets input_ids3
logits[:, :-1] with targets input_ids[:, :-1]4

Timed test