Markov Decision Processes & Bellman Equations

🎯 Learning Objectives

• Formalize RL problems as MDPs
• Understand the Markov property and its implications
• Derive Bellman expectation and optimality equations
• Implement value and action-value function estimation

📋 Prerequisites

• Probability (conditional probability, expectation)
• Dynamic programming concepts

---

1. 📖 Core Content

1.1 The Agent-Environment Interface

• $S$: Set of states
• $A$: Set of actions
• $P(s'|s,a)$: Transition probability to state s' after action a in state s
• $R(s,a,s')$: Reward received after transitioning
• $\gamma \in [0,1]$: Discount factor (Diagram) Markov Property: The future is independent of the past given the present:

$$P(S_{t+1} | S_t, A_t) = P(S_{t+1} | S_1, A_1, ..., S_t, A_t)$$

1.2 Returns and Discounting

$$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + ... = \sum_{k=0}^\infty \gamma^k R_{t+k+1}$$

1. Mathematical convenience: Infinite sums converge
2. Uncertainty: Future rewards are uncertain
3. Human preference: We prefer immediate rewards
4. Avoids infinite loops (in continuing tasks)

1.3 Value Functions

$$V^\pi(s) = \mathbb{E}_\pi[G_t | S_t = s]$$ $$Q^\pi(s,a) = \mathbb{E}_\pi[G_t | S_t = s, A_t = a]$$

1.4 Bellman Expectation Equation

$$V^\pi(s) = \sum_a \pi(a|s) \sum_{s',r} P(s',r|s,a)[r + \gamma V^\pi(s')]$$ $$Q^\pi(s,a) = \sum_{s',r} P(s',r|s,a)[r + \gamma \sum_{a'} \pi(a'|s') Q^\pi(s',a')]$$

1.5 Bellman Optimality Equation

$$V^*(s) = \max_a \sum_{s',r} P(s',r|s,a)[r + \gamma V^*(s')]$$ $$Q^*(s,a) = \sum_{s',r} P(s',r|s,a)[r + \gamma \max_{a'} Q^*(s',a')]$$ $$\pi^*(s) = \arg\max_a Q^*(s,a)$$

1.6 Worked Example: Grid World

pseudo
A → B
↓   ↓
C → D

• From A: Right → B (R=0), Down → C (R=0)
• From B: Right → D (R=0), Down → D (R=+1)
• From C: Right → D (R=0), Down → D (R=+1)
• From D: Terminal (no further reward) Let's compute $V^*$ using Bellman optimality: $V^*(D) = 0$ (terminal) $V^*(B) = \max\{0.9 \times V^*(D), -1 + 0.9 \times V^*(D)\} = \max\{0, -1\} = 0$ (Right is optimal) $V^*(C) = \max\{0.9 \times V^*(D), -1 + 0.9 \times V^*(D)\} = \max\{0, -1\} = 0$ (Right is optimal) $V^*(A) = \max\{0.9 \times V^*(B), 0.9 \times V^*(C)\} = \max\{0, 0\} = 0$ (tie, either action)

---

📝 Practice Questions

Q1

<strong>Q1</strong>: In a 3-state MDP, transitioning from state 1 gives reward +5 and goes to state 2. State 2 gives reward +10 and goes to state 3 (terminal). γ=0.9. Compute V(1) and V(2).

V(3) = 0 (terminal)

V(2) = 10 + 0.9 × V(3) = 10 + 0 = 10

V(1) = 5 + 0.9 × V(2) = 5 + 0.9 × 10 = 5 + 9 = 14

So V*(1) = 14 (assuming only one action per state). The agent should expect total discounted return of 14 starting from state 1. Q2

<strong>Q2

<strong>Q2</strong>: If γ=0.5 instead of 0.9, how do the values change? What does this tell us?

V(2) = 10 + 0.5 × 0 = 10 (same — value of future is already 0) V(1) = 5 + 0.5 × 10 = 5 + 5 = 10

With lower γ: V(1) drops from 14 to 10. This shows that lower discount factors make the agent value immediate rewards more. With γ=0.5, the reward from state 1 (5) contributes more to V(1) than the delayed reward from state 2 (10 × 0.5 = 5).

γ close to 1: far-sighted agent γ close to 0: short-sighted agent Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: Why is the Bellman equation a "self-consistency" condition?

The Bellman equation expresses a recursive relationship:

• V(s) = immediate reward + discounted V(next state)
• But V(next state) depends on V(state after that), and so on

This is self-consistency because the value of a state must equal its own definition. It's not a computational formula (you can't compute V(s) from V(s') without knowing V(s') first) — it's a condition that the true value function must satisfy.

The Bellman equation becomes useful when turned into an update rule: $V_{k+1}(s) = \max_a \sum P[r + \gamma V_k(s')]$

This iterative approach converges to the true V*. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: Derive the relationship between V^π and Q^π.

By definition: $V^\pi(s) = \mathbb{E}_\pi[G_t | S_t = s]$ $Q^\pi(s,a) = \mathbb{E}_\pi[G_t | S_t = s, A_t = a]$

The relationship: $V^\pi(s) = \sum_a \pi(a|s) Q^\pi(s,a)$

The value of a state under policy π is the weighted average of action-values, weighted by the probability of taking each action under π.

Conversely: $Q^\pi(s,a) = \sum_{s',r} P(s',r|s,a)[r + \gamma V^\pi(s')]$

The value of taking action a in state s is the immediate reward plus the discounted value of the next state.

</details> * * * ## 🔗 Cross-References - **Next**: [Dynamic Programming](/notes/04-degree-electives-bsda5007-reinforcement-learning-week03-03-dynamic-programming) - **Previous**: [Multi-Armed Bandits](/notes/04-degree-electives-bsda5007-reinforcement-learning-week01-01-multi-armed-bandits) - **Video**: BSDA5007 Week 2 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Multi-Armed Bandits**](/notes/04-degree-electives-bsda5007-reinforcement-learning-week01-01-multi-armed-bandits)[Next**Dynamic Programming**](/notes/04-degree-electives-bsda5007-reinforcement-learning-week03-03-dynamic-programming)