advantage

The advantage function $A_{\theta }(s,a)$ measures how much better taking action $a$ in state $s$ is compared to the average value of that state. Specifically, it is the difference between the Q-value (expected return starting from $s$, taking action $a$) and the state-value (expected return from state $s$):

$$A(s,a)=Q(s,a)-V(s)$$

Theoretically, one could take any baseline that does not depend on the current action being taken (instead of $V$).

Choosing the advantage function for ${\Psi }_t$ yields almost the lowest possible variance, though in practice, the advantage function is not known and must be estimated. This statement can be intuitively justified by the following interpretation of the policy gradient: that a step in the policy gradient direction should increase the probability of better-than-average actions and decrease the probability of worse-than-average actions. The advantage function, by it’s definition measures whether or not the action is better or worse than the policy’s default behavior. Hence, we should choose ${\Psi }_t$ to be the advantage function, so that the gradient term ${\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$ points in the direction of increased ${\pi }_{\theta }(a_t\mid s_t)$ if and only if $A^{\pi }(s_t,a_t)>0$.

Link to original