#positional-embeddings

Question 1: Select all that apply

Transformers have replaced RNNs in most language models. Which statements about the design choices behind them are true? Select all that apply.

Self-attention can model long-range dependencies better than RNNs, because information does not have to flow step by step through the sequence1
Transformers add positional embeddings because self-attention alone does not encode the order of the tokens2
Multi-head self-attention lets the model attend to different kinds of relations, with a separate set of projections in each head3
Self-attention relates the parts of the input by computing weighted combinations of all tokens, with weights from learned similarity scores4
All heads of a multi-head attention layer share the same query, key and value weights, so they learn the same relations5