#learned-embeddings

Question 1: Single choice

mediumAI-checkedAttention & Transformers

Instead of the fixed sinusoidal positional encoding, you add a learnable table $\mathbf{P} \in \mathbb{R}^{512 \times d}$ of position embeddings to the token embeddings, as BERT does. What is the main limitation compared with the sinusoidal version?

It has no row for positions past 512, while the sinusoidal formula works for any position1
It cannot be trained, because positions are integers and their embeddings receive no gradient2
It makes self-attention depend on word order, which hurts translation3
It adds one parameter per pair of positions, $O(n^2)$ in total4

Timed test