ml-recall

#attention-mask

2 questions from Attention & Transformers, PyTorch

Timed test

2 questions · 2 min · score and explanations at the end

Take a test

Q 1 / 2Score 0 / 0

Shuffle questions

Question 1: Single choice

easyAI-checkedAttention & Transformers

In self-attention, $A = \mathrm{softmax}(QK^\top / \sqrt{d_k})$ row by row, and token $i$ must not attend to token $j$ . What do you do to the score in row $i$ , column $j$ so that $A_{ij} = 0$ and every row of $A$ still sums to 1?

Set it to $-\infty$ before the softmax1
Set it to $0$ before the softmax2
Set $A_{ij}$ to 0 after the softmax3
Set the value vector of token $j$ to zero instead4

Keys: 1–9 select · Enter check / next