error-driven update

Nearly every learning update takes this shape:

$$\text{NewEstimate}\leftarrow \text{OldEstimate}+\text{StepSize}\left[\text{Target}-\text{OldEstimate}\right]$$

Move the estimate a fraction of the way toward the target; $\left[\text{Target}-\text{OldEstimate}\right]$ = error.

These differ only in what the target is and how the step size is chosen: TD updates, Q-learning, Bellman backups under function approximation, gradient descent on squared error.

Even a plain running mean is an instance: estimating a value as the average of all samples seen so far is exactly this update with step size $\frac{1}{n}$.
With $Q_n$ … the average of the first $n-1$ samples $R_1,\dots ,R_{n-1}$:

$$\begin{array}{rl}{Q}_{n+1}& =\frac{1}{n}\sum _{i=1}^n{R}_i\\ & =\frac{1}{n}(R_n+(n-1)\underset{=Q_n}{\underbrace{\frac{1}{n-1}\sum _{i=1}^{n-1}{R}_i}})\\ & =\frac{1}{n}(R_n+(n-1)Q_n)\\ & =\frac{1}{n}(R_n+n{Q}_n-Q_n)\\ & =Q_n+\frac{1}{n}\left[R_n-Q_n\right]\end{array}$$

Decaying step size $\frac{1}{n}$ weights all samples equally (the true average).
A constant step size weights recent samples more (a recency-weighted average), which is what you want when the target is non-stationary.

Why: rewriting the update as $Q_{n+1}=(1-\alpha )Q_n+\alpha R_n$ shows that each update shrinks everything already in the estimate by $(1-\alpha )$. A sample’s final weight is therefore its step size at entry times all later shrinkages.
Constant $\alpha$: sample $R_i$ ends with weight $\alpha (1-\alpha {)}^{n-i}$, decaying exponentially with age.
${\alpha }_m=\frac{1}{m}$: entry weight $\frac{1}{i}$ times the shrinkages ${\prod }_{m=i+1}^n\frac{m-1}{m}$ telescopes to exactly $\frac{1}{n}$, the same for every sample. The later steps must be smaller for the weighting to come out equal: the $n^{\text{th}}$ sample joins an estimate that already speaks for $n-1$ samples, so it deserves a $\frac{1}{n}$ share of the say.