Value learning can diverge (weights, values, and errors growing without bound) when these three conditions are met:
- function approximation … values come from shared weights; updating one state’s value moves others
- bootstrapping … the update target contains the current estimate itself (TD, Bellman backups)
- off-policy updates … states are backed up in different proportions than the policy being evaluated visits them
Why: the semi-gradient update treats the target as fixed, but the target is built from the same weights being updated.
Nudging to fix one state’s value also drags the successor values, i.e. the target, along. On-policy, the visit distribution weights states such that the update still contracts (linear TD provably converges); off-policy with function approximation, each step can grow the error instead.
Dropping any condition restores soundness:
- tabular values (no shared weights) → Q-learning converges
- no bootstrapping → monte carlo targets (full returns) are unbiased, just high variance
- on-policy updates → linear TD converges
An independent aggravator: the over noisy value estimates is biased upward (you tend to pick whichever estimate happens to be too high), and bootstrapping compounds the bias. double Q-learning cancels it.
deep RL’s standard stabilizers are patches for the triad:
- target networks (freeze the bootstrap target so it can’t chase the weights)
- reward and gradient clipping / Huber loss
- double Q
- keeping the update distribution near on-policy