Dynamic Programming: Policy Evaluation, Value Iteration, and Policy Iteration

🎯 Learning Objectives

• Derive and implement iterative policy evaluation for a given policy
• Understand policy improvement via the Bellman optimality equation
• Implement value iteration and policy iteration algorithms
• Analyze the convergence properties of DP methods
• Apply GPI (Generalized Policy Iteration) to solve MDPs

📋 Prerequisites

• MDPs (Week 2): States, actions, transitions, rewards, discount factor, Bellman equations
• Basic probability: Expectation, conditional probability
• Linear algebra: Systems of equations, iterative methods

1. 📖 Core Content

1.1 Intuition: What is Dynamic Programming?

1.2 Policy Evaluation (Prediction)

1.2.1 The Bellman Expectation Equation for $V^\pi$

1.2.2 Iterative Policy Evaluation Algorithm

text
Initialize V(s) = 0 for all s
Δ = ∞
while Δ > θ (small threshold):
    Δ = 0
    for each s ∈ S:
        v = V(s)
        V(s) = Σ_a π(a|s) Σ_{s',r} P(s',r|s,a) [r + γ V(s')]
        Δ = max(Δ, |v - V(s)|)
return V ≈ V^π

Worked Example 1: Policy Evaluation on a Simple Grid

• Move right with p=0.8, go to B: $r = -1$
• Move up with p=0.2 (slip), stay in A: $r = -1$ $V_1(A) = 0.8[-1 + 0.9(0)] + 0.2[-1 + 0.9(0)] = -1.0$ From state B: policy says move down.
• Move down with p=0.8, go to D: $r = 0$ (terminal)
• Move right with p=0.2, go to C: $r = -1$ $V_1(B) = 0.8[0 + 0.9(0)] + 0.2[-1 + 0.9(0)] = -0.2$ From state C: policy says move right.
• Move right with p=0.8, go to D: $r = 0$
• Move down with p=0.2, go to B: $r = -1$ $V_1(C) = 0.8[0] + 0.2[-1] = -0.2$ State D is terminal: $V_1(D) = 0$. Step 3 (k=2): Use $V_1$ to compute $V_2$. $V_2(A) = 0.8[-1 + 0.9(-0.2)] + 0.2[-1 + 0.9(-1)]$ $= 0.8[-1.18] + 0.2[-1.9] = -0.944 - 0.38 = -1.324$ $V_2(B) = 0.8[0] + 0.2[-1 + 0.9(-0.2)] = 0.2[-1.18] = -0.236$ $V_2(C) = 0.8[0] + 0.2[-1 + 0.9(-0.2)] = -0.236$ $V_2(D) = 0$ Continue until convergence (typically 50-100 iterations for this small grid).

1.3 Policy Improvement

Worked Example 2: Policy Improvement on the Grid

• Go right (current): $0.8[-1 + 0.9(-0.5)] + 0.2[-1 + 0.9(-3.5)] = 0.8(-1.45) + 0.2(-4.15) = -1.99$
• Go up: $0.8[-1 + 0.9(-3.5)] + 0.2[-1 + 0.9(-0.5)] = 0.8(-4.15) + 0.2(-1.45) = -3.61$
• Go down: $0.8[-1 + 0.9(-0.5)] + 0.2[-1 + 0.9(-3.5)] = -1.99$ (same as right due to symmetry) Going right gives the highest value. The current policy is already optimal for this simple grid.

1.4 Policy Iteration

text
Initialize π(s) arbitrarily for all s
loop:
    # Policy Evaluation
    V = V^π (using iterative policy evaluation)
    # Policy Improvement
    π'(s) = argmax_a Σ_{s',r} P(s',r|s,a) [r + γ V(s')]
    if π' == π for all s:
        return V, π
    else:
        π = π'

Convergence Properties

• Each iteration produces a strictly better policy (unless already optimal)
• Converges in finite number of iterations (bounded by $|A|^{|S|}$ possibilities)
• In practice, converges much faster than the theoretical bound

Worked Example 3: Policy Iteration for a Small MDP

• From state 1: R → state 2 (p=1), reward +1. L stays in 1 (p=1), reward 0.
• From state 2: R → state 3 (p=1), reward +1. L → state 1 (p=1), reward 0.
• From state 3: L → state 2 (p=1), reward 0. R stays in 3 (p=1), reward +5. $\gamma = 0.95$. Initial policy: $\pi_0(1)=R$, $\pi_0(2)=R$, $\pi_0(3)=R$. Iteration 1 - Evaluation: Solve $V = R^\pi + \gamma P^\pi V$: State 3: $\pi(3)=R$ → stays in 3: $V_3 = 5 + 0.95 \cdot V_3$ → $V_3 = 100$ State 2: $\pi(2)=R$ → goes to 3: $V_2 = 1 + 0.95 \cdot V_3 = 1 + 95 = 96$ State 1: $\pi(1)=R$ → goes to 2: $V_1 = 1 + 0.95 \cdot V_2 = 1 + 91.2 = 92.2$ Iteration 1 - Improvement: State 1: $Q(1,L) = 0 + 0.95(92.2) = 87.6$, $Q(1,R) = 1 + 0.95(96) = 92.2$ → Keep R State 2: $Q(2,L) = 0 + 0.95(92.2) = 87.6$, $Q(2,R) = 1 + 0.95(100) = 96$ → Keep R State 3: $Q(3,L) = 0 + 0.95(96) = 91.2$, $Q(3,R) = 5 + 0.95(100) = 100$ → Keep R Policy unchanged. Converged to optimal policy: always move right.

1.5 Value Iteration

text
Initialize V(s) = 0 for all s
Δ = ∞
while Δ > θ:
    Δ = 0
    for each s ∈ S:
        v = V(s)
        V(s) = max_a Σ_{s',r} P(s',r|s,a) [r + γ V(s')]
        Δ = max(Δ, |v - V(s)|)
# Extract optimal policy
π(s) = argmax_a Σ_{s',r} P(s',r|s,a) [r + γ V(s')]
return V, π

Key Differences from Policy Iteration

1.6 Generalized Policy Iteration (GPI)

• Evaluation pushes the value function toward matching the current policy
• Improvement pushes the policy toward being greedy with respect to the current value function The two processes interact:
• If we stop evaluation early (truncated), the improvement step is based on an approximate value function
• If we only partially improve, the evaluation starts from a near-optimal policy All modern RL algorithms (DQN, SARSA, Actor-Critic) are instances of GPI with different choices of:

1. How thoroughly to evaluate (one step, full convergence, n steps)
2. How to represent the value function (tabular, neural network)
3. How to improve (greedy, ε-greedy, gradient-based)

1.7 Edge Cases & Gotchas

• Asynchronous DP: Instead of sweeping through all states, update states in any order (prioritize states with large Bellman error). Can converge faster.
• In-place vs out-of-place: In-place updates ($V$ is updated immediately) converge faster than out-of-place (using $V_{k}$ to compute $V_{k+1}$).
• Terminal states: Must be handled carefully — their value is always 0, and they have no outgoing transitions.
• Discount factor near 1: Slower convergence because future rewards matter more, making the value propagation slower.
• Deterministic vs stochastic: Stochastic environments require more iterations because the value is an expectation over outcomes.

1.8 Why This Matters

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Using Value Iteration When Policy Iteration Would Be Faster

Pitfall 2: Incorrect Terminal State Handling

Pitfall 3: Setting the Convergence Threshold Too Strict

4. 📝 Practice Questions

Q1: For a 3×3 grid with terminal state at (3,3), discount γ=0.9, reward -1 per step, random slip probability 0.2. What's the optimal path from (1,1) to (3,3)?

The optimal path is the shortest path: right-right-down-down or down-down-right-right. Due to the slip probability, the agent may deviate, but the optimal policy points toward (3,3) greedily. The shortest path length is 4 steps. The optimal value at (1,1) would be approximately:

$V(1,1) \approx -1 - 0.9(1) - 0.9^2(1) - 0.9^3(0) = -2.71$ (ignoring slip)

With slip probability, the actual value is slightly worse (more negative) because the agent may take longer paths. Q2: Prove that policy iteration converges in a finite number of iterations for finite MDPs.

Proof sketch:

1. There are finitely many policies ($|A|^{|S|}$ possible deterministic policies)
2. Each policy iteration produces a strictly better policy (policy improvement theorem)
3. Since policies are strictly ordered by their value functions and there are finitely many, the algorithm must terminate at the optimal policy

The strict improvement follows from the policy improvement theorem: if $\pi'$ is greedy w.r.t. $V^\pi$, then $V^{\pi'}(s) \geq V^\pi(s)$ for all $s$, with strict inequality for at least one state if $\pi$ is not optimal. Q3: Modify value iteration to detect when the policy has stabilized (not just the values).

Add a policy extraction step after each complete sweep:

text
loop:
    # Value iteration update
    for each s:
        V_new(s) = max_a Σ P(s'|s,a)[R + γ V(s')]

    # Policy extraction
    for each s:
        π_new(s) = argmax_a Σ P(s'|s,a)[R + γ V_new(s')]

    if π_new == π for all s:
        return V_new, π_new

    V = V_new
    π = π_new

This stops once the policy stabilizes, which typically happens long before the values fully converge. It's more efficient than waiting for $\Delta < \theta$ on values. Q4: For an MDP with |S|=1000, |A|=5, γ=0.99, compare the computational cost of one iteration of policy evaluation vs. one iteration of value iteration.

Policy evaluation iteration: For each of 1000 states: sum over 5 actions, each summing over transition probabilities (could be up to 1000 next states). Cost: $O(|S|^2|A|) = O(1000^2 \times 5) = O(5,000,000)$

Value iteration: For each of 1000 states: take max over 5 actions, each summing over up to 1000 next states. Cost: $O(|S|^2|A|) = O(5,000,000)$

Same per-iteration cost! The difference is:

• Value iteration: typically needs ~500 iterations to converge (γ=0.99 means slow propagation) → 2.5B operations
• Policy iteration: typically ~20 iterations, each needing ~50 evaluation steps → 20 × 50 × 5M = 5B operations

So they're roughly comparable in total, but policy iteration often wins for large, highly discounted MDPs. Q5: In a deterministic MDP, how does the speed of value iteration change with the discount factor γ?

Value iteration updates propagate information one step at a time. For a reward at the goal:

• After 1 iteration: states one step from goal know the goal value
• After k iterations: states k steps from goal know the goal value

The number of iterations needed scales as $O(D / (1-\gamma))$ where D is the diameter of the MDP.

For γ=0.9: effective horizon = $1/(1-0.9) = 10$ steps. Need ~10-20 iterations. For γ=0.99: effective horizon = $1/(1-0.99) = 100$ steps. Need ~100-200 iterations. For γ=0.999: effective horizon = 1000 steps. Need ~1000 iterations.

The convergence speed is roughly linear in $\frac{1}{1-\gamma}$. Q6: Explain why value iteration can use the same update equation as policy evaluation but with a max over actions.

Policy evaluation computes $V^\pi$ for a fixed policy: $V(s) \leftarrow \sum_a \pi(a|s) \cdot Q(s,a)$.

Value iteration computes $V^*$ directly: $V(s) \leftarrow \max_a Q(s,a)$.

The max operation implicitly performs policy improvement. After each value update, the implicit policy becomes greedy with respect to the current value. This is why value iteration combines evaluation and improvement into one step — the max over actions IS the improvement step.

If you think of GPI as a cycle, value iteration moves both toward the optimal value and the optimal policy simultaneously in each update. Q7: Give an example where value iteration appears to converge (small Δ) but the policy is still suboptimal.

Consider an MDP with two actions (A and B) that give nearly identical values. At state s:

• Action A leads to reward +99 then terminal
• Action B leads to reward +100 then terminal

After few iterations:

• V(s) from A: 99
• V(s) from B: 100 max = 100, choose B. Correct.

But consider a chain MDP where states s1 → s2 → s3 → s4 (terminal, reward +1). γ = 0.9.

After 1 iteration: V(s3) = 1, V(s2) = 0, V(s1) = 0 After 2 iterations: V(s3) = 1, V(s2) = 0.9, V(s1) = 0 After 3 iterations: V(s3) = 1, V(s2) = 0.9, V(s1) = 0.81

After each iteration, Δ decreases. But at iteration 2, the value of s1 is still 0 (suboptimal!). It takes 3 iterations for the reward to propagate to s1. Before that, the policy at s1 appears optimal with respect to the (poor) value estimate.

The error is that Δ measures the maximum change in any state's value, but it doesn't measure how far the value is from optimal. A state far from the reward source might have a slowly changing value that's still far from convergence. Q8: Design a DP algorithm that balances between policy iteration and value iteration — perform a limited number of evaluation sweeps before improving.

This is modified policy iteration (or truncated policy iteration):

text
Initialize V arbitrarily, π arbitrarily
loop:
    # Truncated policy evaluation (k sweeps)
    for i = 1 to k:
        Δ = 0
        for each s ∈ S:
            v = V(s)
            V(s) = Σ_a π(a|s) Σ P(s'|s,a)[R + γ V(s')]
            Δ = max(Δ, |v - V(s)|)
        if Δ < θ_eval: break

    # Policy improvement
    stable = True
    for each s ∈ S:
        old_action = π(s)
        π(s) = argmax_a Σ P(s'|s,a)[R + γ V(s')]
        if π(s) != old_action: stable = False

    if stable: return V, π

With k = 1, this is essentially value iteration (one evaluation sweep before improvement). With k = large or convergence, it's policy iteration. The parameter k controls the tradeoff: more evaluation per improvement (closer to PI) vs. faster initial progress (closer to VI).

Typical choice: k = 5-10 evaluation sweeps per improvement step. This often gives faster overall convergence than either extreme. Q9: In a game of chess, why can't we apply DP directly? What assumptions would be violated?

DP requires:

1. Known transition probabilities: We know how pieces move, but we don't know the opponent's move probabilities. The opponent is part of the environment, and their strategy is unknown.
2. Known reward function: We know checkmate = +1, but intermediate rewards (material advantage, position evaluation) are subjective.
3. Full state enumeration: Chess has ~$10^{47}$ states. We can't iterate over all of them.
4. Markov property: The board state is Markovian (fully observed), so this assumption holds.

What we'd need:

• An approximate value function (neural network evaluation)
• Sample-based updates (self-play instead of exact expected values)
• Function approximation to generalize across similar positions

This is exactly what AlphaZero does: it uses a neural network to approximate $V^\pi(s)$ and $Q^\pi(s,a)$, trained via self-play (sampling), bypassing the need for a known transition model. Q10: Suppose you have an MDP with |S| = 10^6 states and |A| = 10 actions. Can you use DP? If not, what alternatives exist?

Can we use DP? No. Classical DP requires iterating over all states, which costs $O(|S|^2|A|)$ per iteration. With 10^6 states, each iteration costs $10^{13}$ operations — infeasible.

Alternatives:

1. Approximate DP: Use function approximation (neural network) to represent $V(s)$ instead of a table. Update only states visited during sampling.
2. Monte Carlo methods: Learn from complete episodes. No need to iterate over all states.
3. Temporal Difference learning: Learn online from partial episodes. Bootstraps from current value estimates.
4. Hierarchical RL: Decompose the MDP into subtasks, solve each with DP where feasible.
5. Sample-based planning: Use a learned model to simulate trajectories and update values only along those trajectories (Dyna-style).

The key insight: when |S| is large, we trade exactness for scalability by using sampling and function approximation.

5. 🔗 Cross-References

• Previous: MDPs (Week 2) — Bellman equations foundation
• Next: Monte Carlo Methods (Week 4) — Sampling-based value estimation
• Related: TD Learning (Week 5) — Bootstrapping without a model
• External: Sutton & Barto, "Reinforcement Learning: An Introduction" — Chapters 4-5 on DP and MC Join Discord PreviousMDPs & Bellman EquationsNextMonte Carlo Methods