#credit-assignment

Question 1: Short answer

In a cooperative task all agents receive one shared team reward, and COMA gives each agent its own advantage through a counterfactual baseline. Agent $a$ has three actions. With the other agents' actions $\mathbf{u}^{-a}$ held fixed, the centralised critic and agent $a$ 's policy give:

agent $a$ 's action $u'^a$	$u_1$	$u_2$	$u_3$
$Q(s, (\mathbf{u}^{-a}, u'^a))$	4.0	6.0	1.0
$\pi^a(u'^a \mid \tau^a)$	0.2	0.5	0.3

Agent $a$ actually took $u_2$ . What is its COMA advantage?

Timed test