#non-convexity

Question 1: Single choice

NMA trains MLPs of different widths and depths from several slightly perturbed initializations. Larger networks reach a low training loss faster and more consistently across initializations. What is a real downside of relying on this overparameterization?

More memory and compute per step, and a higher risk of overfitting that must be checked on held-out data1
Larger networks always end up in worse local minima than smaller ones2
Overparameterized networks can no longer be trained with gradient descent3
Larger networks make the loss convex, which removes the benefit of momentum4

Timed test