Q-Learning and SARSA: Off-Policy and On-Policy TD Control

🎯 Learning Objectives

• Distinguish off-policy and on-policy learning
• Implement Q-learning and SARSA algorithms
• Understand the convergence conditions for Q-learning
• Compare Q-learning with SARSA on the cliff walking problem

📋 Prerequisites

• Temporal Difference learning (TD(0))
• Policy evaluation and control
• Exploration strategies

---

1. 📖 Core Content

1.1 From TD Prediction to TD Control

1.2 Off-Policy vs On-Policy

1.3 The Cliff Walking Example

• Q-learning learns optimal path (along cliff edge — shortest)
• SARSA learns safer path (away from cliff — longer but considers exploration errors) Because Q-learning follows $\max_a Q(s',a')$ during learning, it assumes optimal future actions. SARSA accounts for the fact that during execution, the agent might still explore (take random actions), so it learns a safer policy.

1.4 Convergence Conditions

1. States and actions are finite (tabular case)
2. $\sum_t \alpha_t = \infty$ and $\sum_t \alpha_t^2 < \infty$ (Robbins-Monro conditions)
3. All state-action pairs visited infinitely often
4. The learning rate decays appropriately

python
Run Inline
# runnable
import numpy as np
class QLearning:
    def __init__(self, n_states, n_actions, alpha=0.1, gamma=0.95, epsilon=0.1):
        self.Q = np.zeros((n_states, n_actions))
        self.alpha = alpha
        self.gamma = gamma
        self.epsilon = epsilon
        self.n_actions = n_actions
    def select_action(self, state):
        """ε-greedy action selection"""
        if np.random.random() < self.epsilon:
            return np.random.randint(self.n_actions)
        return np.argmax(self.Q[state])
    def update(self, state, action, reward, next_state, done):
        """Q-learning update (off-policy)"""
        best_next = np.max(self.Q[next_state]) if not done else 0
        td_target = reward + self.gamma * best_next
        td_error = td_target - self.Q[state, action]
        self.Q[state, action] += self.alpha * td_error
class SARSA:
    def __init__(self, n_states, n_actions, alpha=0.1, gamma=0.95, epsilon=0.1):
        self.Q = np.zeros((n_states, n_actions))
        self.alpha = alpha
        self.gamma = gamma
        self.epsilon = epsilon
        self.n_actions = n_actions
    def select_action(self, state):
        if np.random.random() < self.epsilon:
            return np.random.randint(self.n_actions)
        return np.argmax(self.Q[state])
    def update(self, state, action, reward, next_state, next_action, done):
        """SARSA update (on-policy)"""
        q_next = self.Q[next_state, next_action] if not done else 0
        td_target = reward + self.gamma * q_next
        td_error = td_target - self.Q[state, action]
        self.Q[state, action] += self.alpha * td_error

---

📝 Practice Questions

Q1

<strong>Q1</strong>: In Q-learning, explain why $\max_a Q(s', a')$ makes it off-policy.

The update uses $\max_a Q(s',a')$, which assumes the optimal action in state s' regardless of what the behavior policy actually does. The agent learns the value of the optimal policy while behaving sub-optimally (ε-greedy).

In contrast, SARSA uses the actual action a' taken by the behavior policy. This means SARSA's Q-values reflect the actual policy being followed (including exploration).

Off-policy = learning about policy π* while following behavior policy μ. On-policy = learning about the policy being followed. Q2

<strong>Q2

<strong>Q2</strong>: For the cliff walking problem, why does Q-learning learn a riskier path than SARSA?

Q-learning assumes optimal future actions. During learning, when the agent is near the cliff:

• Q-learning: "If I go close to the cliff, in the next step I'll take the optimal (safe) action" → learns to go along the cliff
• SARSA: "If I go close to the cliff, in the next step I might take a random action (ε probability) and fall" → learns to stay away

The key difference: SARSA accounts for the fact that the agent will continue to explore during execution, making the cliff-edge path dangerous. Q-learning assumes the agent will act optimally (no exploration) after learning.

This makes SARSA conservative and Q-learning optimistic. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: How does Expected SARSA differ from both Q-learning and standard SARSA?

Expected SARSA computes the expectation over next actions instead of using max or a single sample:

$Q(s,a) \leftarrow Q(s,a) + \alpha[r + \gamma \sum_{a'} \pi(a'|s') Q(s',a') - Q(s,a)]$

Compared to:

• Q-learning: Uses max (optimistic, off-policy)
• SARSA: Uses sample of next action (on-policy, higher variance)
• Expected SARSA: Uses expectation (lower variance, can be off/on-policy)

Expected SARSA reduces variance by considering all possible next actions weighted by their probability. With a greedy target policy, Expected SARSA becomes Q-learning. With the behavior policy, it's on-policy with less variance than SARSA. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: In Double Q-learning, why maintain two separate Q-functions?

Standard Q-learning uses $\max_a Q(s',a')$ which creates maximization bias — overestimating Q-values because max of noisy estimates is higher than the true max.

Double Q-learning maintains Q₁ and Q₂:

• With probability 0.5: update Q₁ using $\max_a Q_2(s',a')$
• Otherwise: update Q₂ using $\max_a Q_1(s',a')$

Using the other Q-function for action selection decouples the max operation from the value estimate, reducing overestimation. This is particularly important in stochastic environments where Q-value estimates have high variance.

</details> * * * ## 🔗 Cross-References - **Next**: [Deep Q-Networks](/notes/04-degree-electives-bsda5007-reinforcement-learning-week07-07-dqn) - **Previous**: [TD Learning](/notes/04-degree-electives-bsda5007-reinforcement-learning-week05-05-td-learning) - **Video**: BSDA5007 Week 6 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**TD Learning**](/notes/04-degree-electives-bsda5007-reinforcement-learning-week05-05-td-learning)[Next**Deep Q-Networks (DQN)**](/notes/04-degree-electives-bsda5007-reinforcement-learning-week07-07-dqn)