#adagrad

Question 1: Single choice

Adagrad is run on $f(\mathbf{x}) = 0.1x_1^2 + 2x_2^2$ and on its copy rotated by $45^\circ$ , $f(\mathbf{x}) = 0.05(x_1 + x_2)^2 + (x_1 - x_2)^2$ . Plain gradient descent behaves the same on both up to the rotation, but Adagrad converges much more slowly on the second. Why?

Its per-coordinate scaling is a diagonal preconditioner, which helps only when the steep and flat directions follow the axes1
The rotated function has larger gradients, so the accumulated sum shrinks its steps too fast2
Rotating the function makes it nonconvex, and Adagrad's guarantees need convexity3
Adagrad is designed for sparse features, and after rotation every feature is sparse4

Timed test