#ddp

Question 1: Single choice

mediumAI-checkedDeep Learning Frameworks

A model is trained with PyTorch's DistributedDataParallel on 4 GPUs, one process per GPU, each process with its own minibatch. Right after loss.backward() returns in every process, what does each process hold in its parameters' .grad?

The mean of all 4 processes' gradients, identical in every process1
Its own local gradient; rank 0 averages them in optimizer.step() and broadcasts new weights2
The sum of all 4 gradients, but only on rank 0; the other ranks keep their local ones3
A quarter of the averaged gradient, since each process updates a quarter of the parameters4

Timed test