#distance-attention

Question 1: Single choice

mediumAI-checkedAttention & Transformers

You implement distance-based attention with the score $a(\mathbf{q}, \mathbf{k}_i) = -\tfrac12\lVert\mathbf{q} - \mathbf{k}_i\rVert^2 = \mathbf{q}^\top\mathbf{k}_i - \tfrac12\lVert\mathbf{k}_i\rVert^2 - \tfrac12\lVert\mathbf{q}\rVert^2$ , followed by a softmax over the keys. Which term can you drop without changing any attention weight?

$-\tfrac12\lVert\mathbf{q}\rVert^2$ , because it is the same for every key of a given query1
$-\tfrac12\lVert\mathbf{k}_i\rVert^2$ , because the softmax normalizes away the key norms2
$\mathbf{q}^\top\mathbf{k}_i$ , because the distance information is already in the norms3
None of them, because a softmax changes when a constant is added to its inputs4

Timed test