#exploding-gradients

Question 1: Single choice

Gradient-based hyperparameter optimization differentiates a validation loss through every SGD step of training, for example the 235 steps of one FashionMNIST epoch with minibatch size 256. Apart from the compute and memory needed to keep all those steps, what problem does this create?

The gradient passes through hundreds of chained steps, so it can vanish or explode as in a very deep network1
The gradient is always exactly zero, because the training loss sits at a minimum after one epoch2
The learning rate only enters the last SGD step, so the earlier steps contribute nothing to its gradient3
The chain rule does not apply to SGD steps, because the minibatches are drawn at random4

#exploding-gradients

Timed test