#advantage-estimation

Question 1: Single choice

hardAI-checkedPolicy Gradient & Actor-Critic

You run TRPO on a locomotion task twice: once fitting the value function $V_\phi$ with 80 gradient steps per epoch, once with 0 steps (so $V_\phi$ stays at its random initialization). The second run gets stuck early. Why?

Without value-function updates, the policy network receives no gradient at all1
The advantage estimates that weight each update are built from $V_\phi$ and become poor2
The environment's rewards are rescaled by the value function, so they stay near zero3
TRPO's trust region shrinks to zero when the value function is not trained4

#advantage-estimation

Timed test