Temporal Difference Learning: TD(0), TD(λ), and Eligibility Traces

🎯 Learning Objectives

• Understand how TD learning combines ideas from DP (bootstrapping) and MC (sampling)
• Implement TD(0) for policy evaluation
• Analyze the bias-variance tradeoff between MC and TD
• Extend to n-step TD and TD(λ) with eligibility traces
• Understand the forward and backward views of TD(λ)

📋 Prerequisites

• MDPs (Week 2): Returns, value functions, Bellman equations
• MC Methods (Week 4): Episode-based learning, returns
• Dynamic Programming (Week 3): Bootstrapping concept

1. 📖 Core Content

1.1 Intuition: Learning from Every Step

• Monte Carlo's sampling: Learning from real experience (no model needed)
• DP's bootstrapping: Updating estimates based on other estimates The TD update is:

• The actual reward $R$ (like MC — real experience)
• The estimated value of the next state $V(s')$ (like DP — bootstrapping) Why does this matter? TD is the reason RL scales to large problems:
• Learns from incomplete episodes (unlike MC)
• Doesn't need a model (unlike DP)
• Updates online at every step
• Works for continuing tasks (unlike MC)

1.2 TD Prediction (Policy Evaluation)

1.2.1 TD(0) Algorithm

text
Initialize V(s) arbitrarily
α ∈ (0, 1] (learning rate)
loop for each episode:
    Initialize S
    for each step of episode:
        Take action A from policy π(S)
        Observe reward R and next state S'
        V(S) ← V(S) + α[R + γ V(S') - V(S)]
        S ← S'
    until S is terminal

1.2.2 Worked Example 1: TD(0) on a Simple Chain

1.3 MC vs TD: Bias-Variance Tradeoff

Worked Example 2: MC vs TD on a Random Walk

1.4 Optimal TD Learning Rate

• $\alpha = 1/n(s)$ gives the sample mean (as in MC) but TD's bootstrapping means this isn't optimal
• Constant $\alpha$ (e.g., 0.01-0.1) works well for non-stationary problems
• Decaying $\alpha$: $\alpha_t = 1/t$ ensures convergence

1.5 N-Step TD

1.6 TD(λ): The λ-Return

• λ = 0: Only 1-step TD (TD(0)). Highest bias, lowest variance.
• λ = 1: MC (all returns equally weighted). Zero bias, highest variance.
• λ between 0 and 1: Mixture of all n-step returns.

1.6.1 Forward View (λ-Return Algorithm)

1.6.2 Backward View (Eligibility Traces)

text
Initialize V(s) = 0, e(s) = 0 for all s
α ∈ (0, 1], λ ∈ [0, 1]
loop for each episode:
    Initialize S
    for each step:
        Take action A, get R, S'
        δ = R + γ V(S') - V(S)
        e(S) ← e(S) + 1                # Accumulating trace
        for all s:
            V(s) ← V(s) + α δ e(s)
            e(s) ← γ λ e(s)            # Decay trace
        S ← S'
    until S is terminal

1.6.3 Worked Example 3: Eligibility Traces in Action

1.7 Edge Cases & Gotchas

• Trace type: Accumulating traces (add 1 each visit) can grow unbounded for frequently visited states. Replacing traces (set to 1 instead of add 1) work better in practice.
• λ = 0: Reduces to TD(0) — only the most recent state gets updated.
• λ = 1: Approximates MC — all states on the trajectory get equal credit for the TD error.
• Online λ-return: The theoretical TD(λ) with λ=1 converges to MC, but the backward-view implementation with λ=1 doesn't exactly match MC for finite episodes.
• Offline vs Online: The λ-return algorithm is an "offline" method (update at episode end). The backward view is "online" (update at each step).

1.8 Why This Matters

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Confusing TD and MC Targets

Pitfall 2: Setting λ Too High in TD(λ)

Pitfall 3: Not Resetting Eligibility Traces Between Episodes

Pitfall 4: Using Accumulating Traces in Large State Spaces

4. 📝 Practice Questions

Q1: For a 3-state chain (X→Y→Z, terminal Z with R=+1). Compare TD(0) and MC updates after one episode X→Y→Z. Use α=0.1, γ=0.9. Initial V=0.

Both methods receive the same rewards: R(X→Y)=0, R(Y→Z)=0, R(Z→terminal)=0?

Wait — in this formulation, the reward is received when entering Z. Let me assume: reward = +1 upon arriving at Z. So:

R at step 1 (X→Y): 0 R at step 2 (Y→Z): +1

MC: V(X) ← 0 + 0.1[(0 + 0.9×1 + 0.9²×0) - 0] = 0.09 V(Y) ← 0 + 0.1[(1 + 0) - 0] = 0.10 V(Z) ← 0 (terminal, no update)

TD(0): Step 1: δ = 0 + 0.9(0) - 0 = 0. V(X) ← 0 + 0.1(0) = 0. Step 2: δ = 1 + 0.9(0) - 0 = 1. V(Y) ← 0 + 0.1(1) = 0.10.

After one episode: MC updates both X and Y; TD(0) only updates Y. After many episodes: TD(0) will propagate value from Y to X gradually. Q2: In TD(0), the TD error δ = 0 on a non-terminal step. Does this mean V(s) is correct?

No! δ = 0 means $R + \gamma V(s') = V(s)$. This is the Bellman equation for the current value estimates — they satisfy the Bellman consistency condition. But the estimates could still be wrong.

Example: In the random walk, after initializing all V=0, the first step from state C gives δ = 0 + 0×0 - 0 = 0. V(C) = 0, which is wrong (true V(C) for γ=1 should be 0.5 if probability of eventually reaching the right goal is 0.5).

δ = 0 means there's no local improvement to make given the current estimates. But the estimates might be globally wrong because the reward hasn't propagated yet. Q3: For TD(λ), what happens to the eligibility trace of a state not visited for many steps? How does this relate to credit assignment?

The eligibility trace decays exponentially: $e(s) = (\gamma\lambda)^k \cdot e_0(s)$ after k steps since the last visit.

For λ=0.9, γ=0.99: decay factor per step = 0.891. After 10 steps: $0.891^{10} \approx 0.31$ — still 31% of the original trace remains. After 30 steps: $0.891^{30} \approx 0.03$ — only 3% remains.

This means recent states get more credit for current rewards. A state visited 30 steps ago receives only 3% of the TD error signal for a reward received now. This implements a recency-weighted credit assignment that's roughly exponential.

By contrast, MC gives equal weight to all states on the trajectory. TD(0) gives weight only to the immediately preceding state. Q4: Explain the bias-variance tradeoff in terms of n-step TD returns. How does increasing n affect each?

Bias: The n-step return uses $V(S_{t+n})$ which is a biased estimate of the true value of $S_{t+n}$. For small n, the bootstrap term $V(S_{t+n})$ dominates, leading to higher bias. For large n, the return is mostly composed of actual rewards, reducing bias toward zero (as n → ∞, the return is the MC return, which is unbiased).

Variance: The n-step return contains the sum of n random rewards plus one bootstrapped estimate. For small n, few random variables contribute, so variance is low. For large n, many random rewards sum up, increasing variance.

Tradeoff: Small n = high bias, low variance. Large n = low bias, high variance. The optimal n balances these based on the learning problem:

• Stochastic rewards → prefer smaller n (reduce variance)
• Poor value estimates → prefer larger n (reduce bias from bootstrapping)
• TD(λ) with λ ∈ [0,1] smoothly interpolates between these extremes Q5: Derive the relationship between TD error and Bellman error.

Bellman error for a state s: $BE(s) = V^\pi(s) - \mathbb{E}_\pi[R + \gamma V^\pi(S')|S=s]$

This is the error between the true value $V^\pi(s)$ and the Bellman equation's expectation. At convergence (when learning is complete), Bellman error = 0 for all states.

TD error (expected): $\mathbb{E}[\delta_t | S_t=s] = \mathbb{E}[R_{t+1} + \gamma V(S_{t+1})|S_t=s] - V(s)$

If $V$ = $V^\pi$, then the expected TD error equals the negative Bellman error (which is 0 at convergence). The TD error is a sample-based estimate of the Bellman error.

The TD update reduces the expected TD error by moving $V(s)$ toward $R + \gamma V(s')$. This makes the value function more Bellman-consistent, reducing the Bellman error over time. Q6: Given a random walk with 10 states, γ=0.9, reward +1 at right terminal, 0 elsewhere. Compare TD(0) and TD(λ=0.8) learning speed. Which propagates rewards faster?

TD(0) propagates the reward one state per episode:

• Episode 1: V(state 9) learns the reward (rightmost non-terminal)
• Episode 2: V(state 8) learns ≈ γ × V(9)
• ...
• Episode N: V(state 10-N) learns the reward

So it takes ~10 episodes for the reward to reach the leftmost state.

TD(λ=0.8) propagates the reward to multiple states per episode:

• Episode 1: All states visited get some update. If the agent starts at state 10 (rightmost) and goes to terminal, then states 10, 9, 8... all get eligibility traces. The TD error near the terminal is distributed to all preceding states, with recency weighting.
• After episode 1: V(10) ≈ 1, V(9) ≈ 0.9λ=0.72, V(8) ≈ 0.9²λ²=0.518, etc.

TD(λ) propagates rewards much faster than TD(0) but with higher variance. In practice, TD(λ) with λ ≈ 0.8-0.9 offers a good speed-variance tradeoff. Q7: What happens to eligibility traces when γ=0? How does this affect learning?

When γ=0, the eligibility trace decays according to $e(s) \leftarrow \gamma \lambda e(s) = 0$ for all s after each step. This means:

• At each step, only the current state S_t has e(S_t) = 1
• All other states have e(s) = 0

The TD(λ) update becomes identical to TD(0):

• Only the most recently visited state gets updated
• No credit assignment to earlier states

This makes sense: with γ=0, the agent only cares about immediate rewards. There's no need to assign credit to earlier states because future rewards don't matter. TD(λ) and TD(0) are equivalent when γ=0. Q8: In the backward view of TD(λ), explain why we update all states (not just the current state) at each step.

The backward view uses eligibility traces to assign credit to past states. The update is:

$V(s) \leftarrow V(s) + \alpha \delta_t e_t(s)$

At step t, the TD error δ_t reflects a prediction error about the current transition. States that were visited recently (high e_t(s)) contributed to the prediction that led to this error — they should share in the correction.

If we only updated the current state, we would only correct the immediate predecessor. The eligibility trace mechanism distributes the correction backward along the trajectory, proportionate to recency.

This is computationally efficient: instead of storing n-step returns for each state and updating at the end of the episode, we update every state at every step with a simple mechanism.

The cost is $O(|S|)$ per step (updating all states). For tabular settings this is fine. For function approximation, we update only the parameters that contribute to the value function gradient (typically proportional to the trace). Q9: Show that the sum of weights in the λ-return is 1.

The λ-return is: $G_t^\lambda = (1-\lambda) \sum_{n=1}^\infty \lambda^{n-1} G_{t:t+n}$

The sum of the weights: $(1-\lambda) \sum_{n=1}^\infty \lambda^{n-1} = (1-\lambda) \cdot \frac{1}{1-\lambda} = 1$

This is a geometric series with ratio λ. The factor (1-λ) normalizes the sum to 1 so that G_t^λ is a convex combination of all n-step returns.

For λ=0: weight on 1-step return = 1. All other returns get weight 0. For λ=0.5: 1-step: 0.5, 2-step: 0.25, 3-step: 0.125, ... (sum = 0.5 + 0.25 + 0.125 + ... = 1) For λ=1: All returns weighted equally for an infinite horizon — but this limit requires care because the series becomes $0 \cdot \sum_{n=1}^\infty 1 \cdot G_{t:t+n}$, which is undefined. In practice, λ=1 means the λ-return converges to the MC return as n → ∞. Q10: Compare accumulating traces and replacing traces for a state visited 3 times in a row.

Accumulating trace: Each visit adds 1 to the trace before decay.

• After 1st visit: e = 0 + 1 = 1
• Decay: e = γλ × 1
• After 2nd visit: e = γλ + 1
• Decay: e = (γλ + 1) × γλ = γ²λ² + γλ
• After 3rd visit: e = γ²λ² + γλ + 1

After 3 visits: e = 1 + γλ + γ²λ² (geometric sum). For γ=0.9, λ=0.8: e ≈ 1 + 0.72 + 0.518 = 2.238.

Replacing trace: Each visit sets the trace to 1 (before decay).

• After 1st visit: e = 1
• Decay: e = γλ × 1
• After 2nd visit: e = γλ + 1? No — replacing: e = 1 (set to 1, then decay applies after update)
• Actually, the standard replacing trace: e(s) = 1 (set to 1, not add 1)

After 3 visits with replacing traces: e = 1 (the most recent visit dominates).

Replacing traces prevent the trace from growing unbounded, which is especially important for frequently visited states. Accumulating traces can lead to "runaway" traces where a state visited many times accumulates a very large trace, causing huge updates and instability.

5. 🔗 Cross-References

• Previous: Monte Carlo Methods (Week 4) — MC as the λ=1 special case
• Next: Q-Learning & SARSA (Week 6) — TD control algorithms
• Related: DQN (Week 7) — Q-learning with neural networks
• External: Sutton & Barto, "Reinforcement Learning" — Chapter 6-7 on TD and eligibility traces Join Discord PreviousMonte Carlo MethodsNextQ-Learning & SARSA