Question 1: Short answer
mediumAI-checkedPyTorchTraining with nn.MSELoss() (the default reduction='mean') and torch.optim.SGD with
lr=0.03 works well on minibatches of 64 examples. You switch to
nn.MSELoss(reduction='sum'). Which learning rate gives exactly the same parameter updates?