#ring-allreduce

Question 1: Single choice

In ring synchronization each GPU sends gradient chunks to its neighbour, always in the same direction around the ring. The links (for example NVLink) are full duplex: they carry the same bandwidth in both directions at once. How can you make the synchronization faster?

Let only half of the GPUs send in each step of the ring, so fewer links compete for bandwidth1
Nothing can be gained, because ring synchronization already uses every link at full capacity2
Send every chunk in both directions, so that each GPU receives its data twice as early3
Split the data in two and run one ring each way, roughly halving the time4

Timed test