Q-value

“Quality” of being in a certain state and taking a certain action.

$$\begin{array}{rl}Q(s,a)& =\mathbb{E}[R(s^{\prime },s,a)+\gamma V(s^{\prime })]\\ & =\sum _{s^{\prime }}P(s^{\prime }\mid s,a)(R(s^{\prime },s,a)+\gamma V(s^{\prime }))\end{array}$$

The quality function is a value function which is only implicity a function fo the future state, which allows us to do model-free RL.

We can recover the state-value function from it by averaging over the actions the policy takes:

$$V_{\pi }(s)={\mathbb{E}}_{a\sim \pi }[Q_{\pi }(s,a)]=\sum _a\pi (a\mid s)Q_{\pi }(s,a)$$

The optimal policy always chooses the best action:

$$V_{\ast }(s)=\underset{a}{\max }{Q}_{\ast }(s,a)$$

And we can use $Q$ directly as a policy (here greedily):

$$\pi (s)=\arg \underset{a}{\max }Q(s,a)$$