Credit assignment problem
The typical problem formulation in reinforcement learning is to maximize the expected total reward of a policy. A key source of difficulty is the long time delay between actions and their positive or negative effect on rewards; this issue is called the credit assignment problem in the reinforcement learning literature (Minsky, 1961; Sutton & Barto 1998), and the distal reward problem in the behavioral literature (Hull, 1943).
value functions offer an elegant solution to the credit assignment problem – they allow us to estimate the goodness of an action before the delayed reward arrives.