#lm-head

Question 1: Single choice

A causal language model has embed_tokens = Embedding(4096, 256), 8 decoder layers and lm_head = Linear(256, 4096). You run out = model(input_ids) on one sequence of 17 tokens. What is out[0] (the first output of the ...ForCausalLM model)?

A single embedding of the whole sequence, of shape (1, 4096)1
Token embeddings of shape (17, 256) from the input layer2
Next-token logits over the vocabulary, shape (1, 17, 4096)3
The hidden states of the last decoder layer, of shape (1, 17, 256)4

Timed test