#long-sequences

Question 1: Single choice

Why do Transformers struggle with very long input sequences?

Each position attends to every other: $O(n^2 d)$ compute and an $n \times n$ weight matrix per head1
Information has to pass through $O(n)$ sequential steps, so it fades over long distances2
The sinusoidal positional encoding repeats after 512 positions, so positions become ambiguous3
The number of weight parameters grows with the length of the input sequence4