#eglp-lemma

Question 1: Select all that apply

hardAI-checkedPolicy Gradient & Actor-Critic

The policy gradient can be written $\nabla_\theta J = \mathbb{E}\big[\sum_t \nabla_\theta\log\pi_\theta(a_t\mid s_t)\,\Phi_t\big]$ . Which choices of the weight $\Phi_t$ give the same expected gradient? Select all that apply.

The reward-to-go minus an arbitrary function of the action taken, $\hat R_t - b(a_t)$ 1
The advantage $A^{\pi}(s_t, a_t) = Q^{\pi}(s_t, a_t) - V^{\pi}(s_t)$ 2
The reward obtained just before the action, $r_{t-1}$ 3
The reward-to-go minus a baseline that depends only on the state, $\hat R_t - b(s_t)$ 4
The total return $R(\tau)$ of the episode5

Timed test