#learning-dynamics

Question 1: Single choice

A deep linear network is trained from very small random weights on hierarchically structured data (items such as canary, robin, goldfish, tuna and plants, with their features). The training loss shows a series of plateaus, each followed by a sudden drop. What happens at each drop?

The network picks up one more singular mode of the input–output map, the strongest (e.g. animal vs plant) first1
A learning-rate schedule lowers the learning rate, which produces a sudden improvement2
The network memorizes one more training example at each drop3
Random noise in the weights occasionally kicks the network out of a bad local minimum4

Timed test