#learning-rate-decay

Question 1: Single choice

Adagrad accumulates $\mathbf{s}_t = \mathbf{s}_{t-1} + \mathbf{g}_t^2$ and steps by $\eta\,\mathbf{g}_t/\sqrt{\mathbf{s}_t + \epsilon}$ , so its effective learning rate decays roughly like $1/\sqrt{t}$ . Which change gives a less aggressive decay?

Use a leaky average, $\mathbf{s}_t = \gamma\mathbf{s}_{t-1} + (1 - \gamma)\mathbf{g}_t^2$ 1
Accumulate $|\mathbf{g}_t|$ instead of $\mathbf{g}_t^2$ in $\mathbf{s}_t$ , so that the sum grows more slowly2
Increase $\epsilon$ from $10^{-6}$ to $10^{-4}$ 3
Divide by $\mathbf{s}_t + \epsilon$ instead of by its square root4

#learning-rate-decay

Timed test