ml-recall

#attention

10 questions from Attention & Transformers, Graph Neural Networks, Landmark Architectures, Explainable AI, Fine-tuning & PEFT

Timed test

10 questions · 10 min · score and explanations at the end

Take a test

Q 1 / 10Score 0 / 0

Shuffle questions

Question 1: Short answer

hardAI-checkedAttention & Transformers

Scalar attention with keys equal to values: $\mathcal{D} = \{(k_i, v_i)\} = \{(0, 0), (1, 1), (2, 2)\}$ , score $a(q, k_i) = q\,k_i$ and softmax weights, so $\mathrm{Attention}(q) = \sum_i \mathrm{softmax}_i(q k_i)\, k_i$ . What is the derivative $\frac{d}{dq}\mathrm{Attention}(q)$ at $q = 0$ ? Round to 3 decimals.

Enter: check / next