#trpo

Question 1: Single choice

hardAI-checkedPolicy Gradient & Actor-Critic

You run TRPO on a locomotion task twice: once fitting the value function $V_\phi$ with 80 gradient steps per epoch, once with 0 steps (so $V_\phi$ stays at its random initialization). The second run gets stuck early. Why?

Without value-function updates, the policy network receives no gradient at all1
The advantage estimates that weight each update are built from $V_\phi$ and become poor2
The environment's rewards are rescaled by the value function, so they stay near zero3
TRPO's trust region shrinks to zero when the value function is not trained4

Timed test