#zero-init

Question 1: Single choice

A regression MLP $\hat y = \mathbf{w}_2^\top \tanh(W_1\mathbf{x} + \mathbf{b}_1) + b_2$ is initialized with all weights and biases equal to 0 and trained with gradient descent on MSE. What happens?

Training works normally, because the output bias gets a gradient and breaks the symmetry1
Only the output bias $b_2$ ever changes, so the network can only learn a constant2
The hidden units learn identical but nonzero weights, so the network behaves like one hidden unit3
The loss becomes NaN after the first step because of a division by zero4

Timed test