#kl-penalty

Question 1: Single choice

In RLHF the policy maximises the reward model's score minus a penalty $\beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\text{ref}})$ to the reference (supervised) model. Why is the KL term there?

To keep the policy near the reference model so it cannot exploit reward-model flaws1
To raise the probability of the responses with the highest reward even further than before2
To make each training step cheaper to compute3
To remove biases that were present in the pretraining data4

Timed test