Deciding which states to back up, in which order, during planning.
Planning is a budget of backups; where you spend them determines how fast the values improve. In a big world you can never back up everything, so the selection is what makes planning feasible at all. Strategies:
- sweep over all states, repeatedly (classical value iteration; only feasible in small worlds)
- back up states near the current one, those about to matter (trajectory sampling, MCTS-like)
- prioritized sweeping: when a backup changes a value a lot, queue the state’s predecessors next; change propagates backwards through the state graph, driven by surprise