#implicit-reward

Question 1: Select all that apply

Direct Preference Optimization (DPO) trains a policy $\pi_\theta$ on a fixed dataset of preference pairs $(x, y_w, y_l)$ with the loss

$\mathcal{L}_{\text{DPO}} = -E\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]$

Which statements are correct? Select all that apply.

The loss is a Bradley–Terry likelihood in which the implicit reward replaces a learned reward model1
A larger $\beta$ lets the policy drift further away from $\pi_{\text{ref}}$ , because the preference signal is stronger2
No separate reward model is trained; the implicit reward is $\beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$ 3
The reference model $\pi_{\text{ref}}$ is no longer needed once training starts, since only $\pi_\theta$ is updated4
No responses are sampled from $\pi_\theta$ during training; the loss uses only the stored pairs5

Timed test