#multi-head-attention

Question 1: Single choice

mediumAI-checkedAttention & Transformers

You want to prune the least important heads of a trained multi-head attention model to speed up prediction. Which experiment measures one head's importance most directly?

Mask that head's output at test time and measure how much the validation score drops1
Rank heads by how uniform their attention weights are, calling the most uniform the most important2
Rank heads by position, since later heads in the concatenation contribute less to $\mathbf{W}_o$ 3
Measure the norm of each head's query projection $\mathbf{W}_i^{(q)}$ at initialization4

#multi-head-attention

Timed test