#nondeterminism

Question 1: Select all that apply

mediumAI-checkedDeep Learning Frameworks

You train the same model twice on the same GPU, with the same code, data and torch.manual_seed(0). The losses match at first but slowly drift apart over a few hundred steps. Which of these can explain it? Select all that apply.

Some GPU kernels accumulate with atomic adds, whose order can vary from run to run1
If cudnn.benchmark is on, cuDNN may choose a different convolution algorithm in each run2
torch.manual_seed seeds only the CPU generator, so GPU dropout masks differ in each run3
Floating-point rounding is random, so the same additions in the same order can differ4

Timed test