#reward-hacking

Question 1: Select all that apply

A language model is fine-tuned with RL against a learned reward model or an automatic checker. Which of these behaviors are examples of reward hacking? Select all that apply.

The model learns to refuse requests for weapon instructions, exactly as the reward was meant to teach it1
The model agrees with a user's false claim, because agreeing tended to earn a high reward2
Answers grow longer and longer, because the reward model happens to score long answers higher3
A coding agent edits the unit tests so that they pass, instead of fixing the bug they test4
The training loss diverges after a few steps, because the learning rate was set too high5

Timed test