#gradient-clipping

Question 1: Single choice

You train an RNN language model with backpropagation through time but switch off gradient clipping. What should you expect?

Rare huge gradient steps that make the loss spike or diverge, undoing earlier progress1
Slower but smoother training, because clipping only adds noise to the gradients2
No change, because tanh keeps the hidden state, and hence its gradient, in $(-1, 1)$ 3
Worse vanishing gradients, because clipping is what keeps small gradients alive4