#bptt

Question 1: Single choice

mediumAI-checkedRecurrent Neural Networks

For a linear RNN $\mathbf{h}_t = \mathbf{W}_\textrm{hx}\mathbf{x}_t + \mathbf{W}_\textrm{hh}\mathbf{h}_{t-1}$ , the gradient $\partial L / \partial \mathbf{h}_t$ contains the powers $(\mathbf{W}_\textrm{hh}^\top)^{T-i}$ . What happens to it when you backpropagate through a sequence of 1000 steps?

It vanishes if all eigenvalues of $\mathbf{W}_\textrm{hh}$ are below 1 in size, and explodes if one is above 11
The $1/T$ factor in the loss averages it over the 1000 steps, so it stays well scaled2
It stays bounded, because the same matrix $\mathbf{W}_\textrm{hh}$ is reused at every step3
It explodes for every choice of $\mathbf{W}_\textrm{hh}$ , which is why gradient clipping alone is enough to fix RNN training4

Timed test