#query-key-value

Question 1: Select all that apply

mediumAI-checkedAttention & Transformers

Self-attention builds a new representation of each token from the tokens of its sequence. Which statements about this mechanism are true? Select all that apply.

Unlike RNNs, self-attention lets a model process all tokens of a sequence at once, without recurrent connections1
In self-attention without a causal mask (as in an encoder), every token can influence the representation of every other token through learned attention weights2
Self-attention computes a weighted sum of value vectors, with weights based on the similarity between query and key vectors3
The cost of self-attention grows quadratically with the sequence length, because it scores every pair of tokens4
A self-attention layer outputs a single scalar for each token5

Timed test