#hyperparameter-transfer

A layer maps $d_{\text{in}} = 1024$ inputs to $d_{\text{out}} = 256$ outputs. It is initialized as $W = \sqrt{d_{\text{out}} / d_{\text{in}}}\; M$, where $M \in \mathbb{R}^{256 \times 1024}$ is a random semi-orthogonal matrix (orthonormal rows). What is the spectral norm of $W$?