TD Lambda

The question of what $n$ we want to pick, i.e. how soon to start relying on predicted rewards, is handled by TD Lambda in a similar way / with a similar intuition as with reward discounting: We take a weighted average of all possible $n$-step TD targets rather than picking a single best $n$, discounting future rewards, as the further in the future we predict, the less confident we are.

Link to original

TD$(\lambda )$ decays the n-step return by a factor $0\le \lambda <1$:

$$G_t^{\lambda }=(1-\lambda )\sum _{n=1}^{\infty }{\lambda }^{n-1}{G}_t^{(n)}$$

We multiply by $(1-\lambda )$ to normalize the lambdas, i.e form a weighted average, which follows from rearranging the geometric sequence:

$$\begin{array}{rl}\Lambda & =1+\lambda +{\lambda }^2+\dots \\ \Lambda & =1+\lambda (1+\lambda +{\lambda }^2)\\ \Lambda & =1+\lambda \Lambda \\ \Lambda -\lambda \Lambda & =1\\ (1-\lambda )\cdot \Lambda & =1\end{array}$$

This is different to $\gamma$ discounting, where we want future rewards to have less total influence and thus don’t normalize the weights – the infinite sum ${\sum }_{k=0}^{\infty }{\gamma }^k=\frac{1}{1-\gamma }$ represents the maximum possible return rather than forming a weighted average of complete estimates.

TD(0) is equivalent to using only a single step actual reward and predicting the next.
TD($\infty$) is equivalent to monte carlo methods (complete trajectories of actual rewards).