#goodhart

Question 1: Single choice

Gao et al. (2022) let a fixed "gold" reward model play the role of human labelers and train a smaller proxy reward model on its labels. A policy is then optimized against the proxy with RL. As the optimization proceeds (measured by the KL divergence from the initial policy), how does the gold reward score behave?

It falls from the very start, since optimizing any learned proxy hurts the true goal1
It rises and levels off near the proxy score, since the proxy learned from gold labels2
It first rises and later falls, even though the proxy score keeps rising3
It tracks the proxy score exactly, apart from a constant offset between the two4

Timed test