#ppo-ptx

Question 1: Select all that apply

InstructGPT's RL step maximizes

$\text{objective}(\phi) = E_{(x,y) \sim D_{\pi_\phi^{RL}}}\left[r_\theta(x,y) - \beta \log \frac{\pi_\phi^{RL}(y \mid x)}{\pi^{SFT}(y \mid x)}\right] + \gamma\, E_{x \sim D_{\text{pretrain}}}\left[\log \pi_\phi^{RL}(x)\right]$

where $r_\theta$ is the reward model and $\pi^{SFT}$ the supervised fine-tuned model. Which statements about this step are correct? Select all that apply.

Each episode is one prompt and one response; the reward model scores the response once and the episode ends1
The $\beta$ term is a per-token KL penalty toward the SFT model, meant to curb reward-model over-optimization2
The reward model $r_\theta$ is trained by maximizing this same objective, jointly with the policy3
The $\gamma$ term (PPO-ptx) mixes in pretraining gradients to reduce performance drops on public NLP datasets4
The $\beta$ term rewards the policy for moving far away from the SFT model, which encourages exploring new outputs5

Timed test