#transformer

Question 1: Single choice

mediumAI-checkedAttention & Transformers

Would it be a good idea to replace scaled dot-product attention in a Transformer with additive attention, $\mathbf{w}_v^\top\tanh(\mathbf{W}_q\mathbf{q} + \mathbf{W}_k\mathbf{k})$ ?

No: queries and keys already share a dimension, and dot products run as fast matrix multiplications1
Yes: additive attention is the only scoring function that stays accurate for large key dimensions $d$ 2
Yes: the Transformer needs additive attention to combine its attention heads3
No: additive attention cannot be masked, so the decoder would see future tokens4

Timed test