#second-derivatives

Question 1: Single choice

mediumAI-checkedPyTorch

Why is the full matrix of second derivatives (the Hessian) of a loss with $n$ parameters much more expensive to get from an autograd framework than its gradient?

Autograd only knows first-order rules, so second derivatives always fall back to finite differences1
The forward pass has to be rerun from scratch once for every pair of parameters $(i, j)$ 2
Backpropagation needs a scalar output, so second derivatives cannot be computed at all3
The backward pass is recorded and differentiated again, once per gradient entry, for $n^2$ numbers4

Timed test