#gaussian-kernel

Question 1: Single choice

mediumAI-checkedAttention & Transformers

All queries and keys lie on the unit sphere, $\lVert\mathbf{x}\rVert = \lVert\mathbf{x}_i\rVert = 1$ . What do the normalized Gaussian-kernel weights $\exp(-\tfrac12\lVert\mathbf{x} - \mathbf{x}_i\rVert^2) / \sum_j \exp(-\tfrac12\lVert\mathbf{x} - \mathbf{x}_j\rVert^2)$ simplify to?

$\mathrm{softmax}_i(\mathbf{x}^\top\mathbf{x}_i)$ , i.e. dot-product attention1
$\mathrm{softmax}_i(-\mathbf{x}^\top\mathbf{x}_i)$ , which favors the least similar keys2
Uniform weights $1/n$ , because all points on the sphere are equally far apart3
$\mathrm{softmax}_i(\lVert\mathbf{x} - \mathbf{x}_i\rVert)$ , a softmax of the distances4

Timed test