#ridge-regression

Question 1: Single choice

hardAI-checkedLinear Models & Regularization

Your design matrix $\mathbf{X} \in \mathbb{R}^{n \times d}$ has linearly dependent columns, so $\mathbf{X}^\top\mathbf{X}$ is singular. You add independent noise $\epsilon_{ij} \sim \mathcal{N}(0, \sigma^2)$ to every entry, giving $\mathbf{X} + \mathbf{E}$ . What is the expected value of $(\mathbf{X} + \mathbf{E})^\top(\mathbf{X} + \mathbf{E})$ ?

$\mathbf{X}^\top\mathbf{X} + \sigma^2\mathbf{I}$ , one noise variance per entry1
$\mathbf{X}^\top\mathbf{X} + d\sigma^2\mathbf{I}$ , one term per feature2
$\mathbf{X}^\top\mathbf{X} + n\sigma^2\mathbf{I}$ , which is invertible3
$\mathbf{X}^\top\mathbf{X}$ , since the noise has mean zero4

Timed test