#loss-scaling

Question 1: Single choice

mediumAI-checkedDeep Learning Frameworks

In float16 mixed-precision training, the loss is multiplied by a large factor $S$ (e.g. 1024) before backward(), and the gradients are divided by $S$ again before the optimizer step. What problem does this solve?

Small gradients would underflow to zero in float16; scaling moves them into its range1
It raises the effective learning rate $S$ -fold to make up for float16's low precision2
It stops large gradients from overflowing to infinity in float163
It lets the optimizer update float16 weights directly, without a float32 master copy4

Timed test