Question 1: Short answer
easyAI-checkedOptimization & CalculusPlain SGD works well with . You switch to momentum, , , with . Where the gradient is roughly constant, which gives the same step size as before?
12 questions from Optimization & Calculus, Training Deep Networks
12 questions · 12 min · score and explanations at the end
Plain SGD works well with . You switch to momentum, , , with . Where the gradient is roughly constant, which gives the same step size as before?