#experience-replay

Question 1: Single choice

Two agents learn at the same time with independent deep Q-learning. Each treats the other as part of the environment and trains on its own experience-replay buffer. Why can transitions that agent 1 stored a while ago mislead its updates?

The agents share one team reward, so agent 1 cannot tell which agent earned it1
Agent 2 has changed its policy since, so they show dynamics that no longer hold2
With two agents acting, the buffer fills faster and old transitions are overwritten too soon3
Agent 1's own policy has changed since, and Q-learning cannot use another policy's data4

Timed test