#high-dimensions

Question 1: Single choice

Starting from the same weights of an MLP with tens of thousands of parameters, you compare one small gradient-descent step with 100 random perturbations of similar size. The gradient step reduces the loss; almost none of the random perturbations do. Why?

The gradient is the steepest-descent direction, and a random direction in many dimensions is nearly orthogonal to it1
Random perturbations are always larger than gradient steps, so they overshoot the minimum2
Random search only works on convex losses, and the loss of an MLP is not convex3
Only the gradient step uses the labels, so random changes of the weights cannot affect the loss4

Timed test