Skip to content
ml-recall

#allreduce

2 questions from Deep Learning Frameworks

Timed test

2 questions · 2 min · score and explanations at the end

Take a test
Q 1 / 2Score 0 / 0

Question 1: Short answer

mediumAI-checkedDeep Learning Frameworks

Four GPUs each hold a 160 MB gradient. Every GPU has its own 16 GB/s link to a PCIe switch, all links run at full speed at the same time, and sending 160 MB over one link takes 10 ms. Summing everything on GPU 0 and sending the result back takes 3×10+3×10=603 \times 10 + 3 \times 10 = 60 ms, because GPU 0's link carries all the traffic. If you instead split the gradient into four 40 MB parts and let each GPU aggregate and redistribute one part, how many milliseconds does the synchronization take?

a number of milliseconds