#gae

Question 1: Single choice

mediumAI-checkedPolicy Gradient & Actor-Critic

Generalized advantage estimation computes $\hat{A}_t = \sum_{l \ge 0} (\gamma\lambda)^l\, \delta_{t+l}$ with the TD error $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ . What do the two extreme settings of $\lambda$ give?

$\lambda = 0$ switches off discounting of future rewards; $\lambda = 1$ discounts them by the usual $\gamma$ 1
$\lambda = 0$ makes every advantage estimate zero; $\lambda = 1$ gives the one-step TD error $\delta_t$ 2
$\lambda = 0$ gives the TD error $\delta_t$ ; $\lambda = 1$ gives the discounted return minus $V(s_t)$ 3
$\lambda = 0$ gives the discounted return minus $V(s_t)$ ; $\lambda = 1$ gives the TD error $\delta_t$ 4

Timed test