#hidden-state

Question 1: Single choice

mediumAI-checkedRecurrent Neural Networks

An RNN language model approximates $P(x_t \mid x_{t-1}, \ldots, x_1)$ by $P(x_t \mid h_{t-1})$ . Why can it, in principle, condition on all earlier tokens, while an $n$ -gram model cannot?

$h_t = f(x_t, h_{t-1})$ is computed recursively, so $h_{t-1}$ depends on every earlier input1
It learns a separate weight matrix for every time step, so it keeps one per earlier token2
Its input at step $t$ is the concatenation of the one-hot vectors of all earlier tokens3
Its softmax output layer adds up the scores computed at all earlier positions4

Timed test