value iteration

We can rewrite the

Transclude of Bellman-Equation#^36a1e7

to this (literally the same thing, just making $a$ and $P$ explicit):

$$V(s)=\underset{a}{\max }\sum _{s^{\prime }}P(s^{\prime }\mid s,a)(R(s^{\prime },s,a)+\gamma V(s^{\prime }))$$

(If the MDP was deterministic, we could drop the sum)
We can initialize this value function/table poorly, and recursively / iteratively refine it.
Once that’s converged, we can extract the optimal policy:

$$\pi (s,a)=\arg \underset{a}{\max }\sum _{s^{\prime }}P(s^{\prime }\mid s,a)(R(s^{\prime },s,a)+\gamma V(s^{\prime }))$$

Here, we have to know the transition probabilities $P$, the next state $s^{\prime }$ and reward $R(s^{\prime },s,a)$, so this is model-based. We need some explicit model of the environment that tells us the expected next state and reward given the current state and action.

This is a type of dynamic programming. Given perfect information about the environment, we can find the optimal solution by DP. You can just code this algorithm up to solve low dimensional problems (e.g. tic tac toe), without neural nets.