#orthogonalized-updates

Question 1: Single choice

A layer's gradient matrix has the reduced SVD $G = U \Sigma V^\top$ . Steepest descent under the spectral norm $\lVert \cdot \rVert_2$ (the largest singular value) solves $\operatorname*{arg\,min}_{\Delta W}\Big[\operatorname{tr}(G^\top \Delta W) + \tfrac{\lambda}{2}\lVert \Delta W \rVert_2^2\Big].$ Which update is the solution?

$-\frac{\operatorname{tr}\Sigma}{\lambda}\, U V^\top$ 1
$-\frac{1}{\lambda}\, U \Sigma V^\top$ 2
$-\frac{\sigma_{\max}}{\lambda}\, U V^\top$ 3
$-\frac{\operatorname{tr}\Sigma}{\lambda}\, \mathrm{sign}(G)$ 4

#orthogonalized-updates

Timed test