#variance-reduction

Question 1: Single choice

Replacing $R(\tau)$ by the reward-to-go $\hat R_t$ in the policy-gradient estimator improves learning. What exactly changes?

The estimator becomes biased, but the bias is small enough to be worth it1
The expected gradient changes so that it points more directly to the optimum2
It removes the need to sample trajectories from the current policy3
The expected gradient stays the same, but its sample estimate has lower variance4