Q-value

“Quality” of being in a certain state and taking a certain action.

$$\begin{array}{rl}Q(s,a)& =\mathbb{E}[R(s^{\prime },s,a)+\gamma V(s^{\prime })]\\ & =\sum _{s^{\prime }}P(s^{\prime }\mid s,a)(R(s^{\prime },s,a)+\gamma V(s^{\prime }))\end{array}$$

The quality function is a value function which is only implicity a function fo the future state, which allows us to do model-free RL.

We can recunstruct the state-value function from it:

$$V(s)=\underset{a}{\max }Q(s,a)$$

And use it as a policy (here greedily):

$$\pi (s,a)=\arg \underset{a}{\max }Q(s,a)$$