#sample-reuse

Question 1: Single choice

mediumAI-checkedPolicy Gradient & Actor-Critic

Vanilla policy gradient collects a batch of trajectories with the current policy, takes one gradient step on $\hat{E}_t[\log \pi_\theta(a_t \mid s_t)\, \hat{A}_t]$ , and then discards the batch. Why doesn't it simply take many gradient steps on the same batch, as PPO does?

PPO is an off-policy method like Q-learning, while vanilla policy gradient is on-policy, so only PPO may reuse data1
Each return in the batch may be credited to only one update, otherwise the rewards would be counted twice in the objective2
The gradient estimate assumes data from the current policy; after a step the batch is stale and reuse can wreck it3
Reusing the batch would be just as sound for vanilla policy gradient; it is skipped only to save the memory for storing it4

Timed test