#adaptive-learning-rate

Question 1: Single choice

During SGD, one coordinate of the gradient is consistently much larger than all the others. What is the most sensible change to the solver?

Give each coordinate its own step size, scaled down where gradients are large1
Raise the global learning rate so that all the small coordinates also move faster2
Drop that coordinate from the update so that it stops dominating3
Use smaller minibatches so that the large coordinate averages out4