#dyna-q

Question 1: Select all that apply

Dyna-Q adds planning to Q-learning: after each real step it updates $Q$ from the real transition, stores the observed reward and next state in a table $\text{Model}(s, a)$ , and then performs $k$ extra Q-learning updates on transitions replayed from that table. Which statements are true? Select all that apply.

With $k = 0$ it reduces to ordinary Q-learning1
Storing one outcome per state–action pair is a valid model only if the environment is deterministic2
A change in the world (e.g. a new shortcut) that is experienced once can spread through the Q-table during planning3
More planning steps always help, so $k$ should be as large as compute allows4
Planning updates use a different learning rule from the one used for real experience5

Timed test