#data-parallelism

Question 1: Single choice

On one GPU you trained with minibatch size 256 and learning rate 0.2. You switch to data-parallel training on 8 GPUs with 256 examples per GPU, so each update now uses 2048 examples. What is the usual starting point for the new learning rate?

Still 0.2 per GPU, which adds up to an effective rate of 1.6 without changing any code1
About 1.6, scaled up 8× along with the minibatch, often with a short warm-up2
About 0.025, scaled down 8× because each update now averages many more gradients3
Still 0.2, since the gradient is an average and so does not depend on the batch size4

Timed test