upper confidence bound

Upper-Confidence-Bound Action Selection

$$A_t\doteq \underset{a}{\mathrm{argmax}}\left[Q_t(a)+c\sqrt{\frac{\ln t}{N_t(a)}}\right]$$

$c$ controls the degree of exploration.
The part with $c$ is the uncertainty term. As the number of times an action is selected increases, its uncertainty (and degree of exploration) decreases. But every time it doesn’t get selected, the likelihood of being selected increases, as $t$ increases.

The use of the natural logarithm means that the increases get smaller over time, but are unbounded; all actions will eventually be selected, but actions with lower value estimates, or that have already been selected frequently, will be selected with decreasing frequency over time.

Link to original