Monte Carlo Methods: First-Visit, Every-Visit, and Monte Carlo Control

🎯 Learning Objectives

• Understand why Monte Carlo methods work without a model of the environment
• Implement first-visit and every-visit Monte Carlo prediction
• Apply Monte Carlo control for optimal policy learning
• Compare on-policy and off-policy Monte Carlo methods
• Understand the bias-variance tradeoff in Monte Carlo estimation

📋 Prerequisites

• MDPs (Week 2): States, actions, rewards, episodes, returns
• Dynamic Programming (Week 3): Policy evaluation and improvement concepts
• Probability: Expectation, sample means, law of large numbers

1. 📖 Core Content

1.1 Intuition: Learning from Complete Episodes

• No transition probabilities $P(s'|s,a)$ needed
• No reward function needed (just observe rewards)
• Learn from real experience, not simulated experience

1.2 MC vs DP: The Fundamental Difference

1.3 MC Prediction (Policy Evaluation)

1.3.1 The Algorithm

text
Initialize V(s) arbitrarily, Returns(s) = empty list
loop for each episode:
    Generate episode following π: S0, A0, R1, S1, A1, R2, ..., ST
    G = 0
    for t = T-1, T-2, ..., 0:
        G = γ * G + R_{t+1}
        if S_t is first visit (first-visit MC):
            Append G to Returns(S_t)
            V(S_t) = average(Returns(S_t))

1.3.2 First-Visit vs Every-Visit MC

• First-visit MC: Lower variance, easier to analyze theoretically
• Every-visit MC: Lower bias (uses more data), but introduces correlation between updates

Worked Example 1: First-Visit MC on a Simple MDP

• State A: first visit at t=0 in episode 1 (G=3), first visit at t=0 in episode 3 (G=4). V(A) = (3+4)/2 = 3.5
• State B: first visit at t=1 in episode 1 (G=2), first visit at t=0 in episode 2 (G=2). V(B) = (2+2)/2 = 2.0
• State C: first visit at t=2 in episode 1 (G=1), first visit at t=1 in episode 2 (G=1), t=1 in episode 3 (G=1). V(C) = (1+1+1)/3 = 1.0 Every-visit MC:
• State A: visited at t=0 in episode 1 (G=3), t=0 in episode 3 (G=4). V(A) = 3.5 (same, each visited once)
• State B: visited at t=1 episode 1 (2), t=0 episode 2 (2). V(B) = 2.0 (same)
• State C: visited at t=2 episode 1 (1), t=1 episode 2 (1), t=1 episode 3 (1). V(C) = 1.0 (same) In this case, they give the same result because no state was visited multiple times in the same episode.

Worked Example 2: When First-Visit and Every-Visit Differ

• t=0 (A): G = 1 + (-1) + 1 + 2 + 0 = 3
• t=1 (B): G = -1 + 1 + 2 + 0 = 2
• t=2 (A): G = 1 + 2 + 0 = 3
• t=3 (B): G = 2 + 0 = 2
• t=4 (C): G = 0 First-visit MC:
• A: first visit at t=0 (G=3). V(A) = 3
• B: first visit at t=1 (G=2). V(B) = 2
• C: first visit at t=4 (G=0). V(C) = 0 Every-visit MC:
• A: visits at t=0 (3), t=2 (3). V(A) = 3
• B: visits at t=1 (2), t=3 (2). V(B) = 2
• C: visit at t=4 (0). V(C) = 0 Still the same! Let's try a case where they differ... Actually, with γ=1 and deterministic returns, every occurrence of A in the same episode has the same return (since the rest of the episode is the same). The difference shows when returns differ: Episode: A → B → A → C → (terminal), γ=1. Rewards: A→B: +5, B→A: -3, A→C: +10.
• t=0 (A): G = 5 + (-3) + 10 = 12
• t=1 (B): G = -3 + 10 = 7
• t=2 (A): G = 10 First-visit MC: V(A) = 12 (only t=0). Every-visit MC: V(A) = (12+10)/2 = 11. Every-visit MC uses both observations, giving a different estimate. The first visit's return (12) reflects the total discounted return from the first visit onward, while the second visit's return (10) is the remainder. First-visit MC is theoretically cleaner (each state contributes at most once per episode to avoid correlation).

1.4 Incremental MC Updates

Worked Example 3: Incremental MC Update

• After G=3: V(A) = 0 + 0.1(3-0) = 0.3
• After G=4: V(A) = 0.3 + 0.1(4-0.3) = 0.3 + 0.37 = 0.67
• After G=2: V(A) = 0.67 + 0.1(2-0.67) = 0.67 + 0.133 = 0.803
• After G=5: V(A) = 0.803 + 0.1(5-0.803) = 0.803 + 0.420 = 1.223
• After G=3: V(A) = 1.223 + 0.1(3-1.223) = 1.223 + 0.178 = 1.401 Sample mean: (3+4+2+5+3)/5 = 17/5 = 3.4. With $\alpha = 0.1$, our estimate 1.401 is far from 3.4. That's because constant $\alpha$ gives more weight to recent observations. For the true average, use $\alpha = 1/n$.

1.5 MC Control (Policy Improvement)

1.5.1 MC with Exploring Starts

text
Initialize Q(s,a) arbitrarily, π(s) arbitrarily, Returns(s,a) = empty list
loop for each episode:
    Generate episode following π, starting from a random (s,a) pair (exploring starts)
    G = 0
    for t = T-1, ..., 0:
        G = γ * G + R_{t+1}
        if (S_t, A_t) is first visit:
            Append G to Returns(S_t, A_t)
            Q(S_t, A_t) = average(Returns(S_t, A_t))
            π(S_t) = argmax_a Q(S_t, a)

1.5.2 On-Policy MC Control (ε-Soft Policies)

• With probability $1-\varepsilon$: choose greedy action ($\arg\max_a Q(s,a)$)
• With probability $\varepsilon$: choose random action

text
Initialize Q(s,a) arbitrarily
Initialize π as ε-greedy w.r.t. Q
loop for each episode:
    Generate episode following π
    G = 0
    for t = T-1, ..., 0:
        G = γ * G + R_{t+1}
        if (S_t, A_t) is first visit:
            Q(S_t, A_t) = average(Returns(S_t, A_t))
    # Policy improvement (implicit via ε-greedy)
    # Next episode uses updated ε-greedy policy

1.5.3 Off-Policy MC Control (Importance Sampling)

1.6 Bias, Variance, and Convergence

• Each episode provides one noisy return sample
• Variance grows with episode length (product of noisy rewards)
• Importance sampling variance grows exponentially with episode length

1.7 Edge Cases & Gotchas

• Non-episodic tasks: MC only works for episodic tasks (those that terminate). For continuing tasks, use TD learning.
• Episodes must terminate: If episodes don't terminate, returns are infinite. MC breaks.
• State-space coverage: MC only estimates values for visited states. States with low probability under $\pi$ are estimated poorly.
• Importance sampling degeneracy: If $\pi$ and $b$ diverge significantly, importance sampling ratios have enormous variance.
• Off-policy MC with deterministic target: If $\pi$ is deterministic and $b$ takes a different action, $\rho = 0$ and the sample is discarded.

1.8 Why This Matters

1. Off-policy learning: Understanding importance sampling for MC is prerequisite for off-policy TD (Q-learning)
2. Monte Carlo Tree Search: MCTS uses MC rollouts for state evaluation (used in AlphaGo)
3. Gradient estimation: REINFORCE uses the MC return directly for policy gradient
4. Evaluation: MC provides an unbiased baseline for comparing TD methods

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Using MC for Continuing (Non-Episodic) Tasks

Pitfall 2: Not Handling the First-Visit Check Correctly

Pitfall 3: Off-Policy MC with Extremely High Variance

Pitfall 4: Confusing On-Policy and Off-Policy MC

4. 📝 Practice Questions

Q1: For a 4-state chain MDP, three episodes are generated following a uniform random policy. Episode 1: s1→s2→s3→s4(terminal), rewards +1,+1,+2. Episode 2: s2→s3→s4(terminal), rewards +1,+2. Episode 3: s1→s2→s4(terminal), rewards +1,+3. Compute first-visit MC estimates for all states (γ=1).

Episode 1: G(t=0,s1)=1+1+2=4, G(t=1,s2)=1+2=3, G(t=2,s3)=2, G(t=3,s4)=0 Episode 2: G(t=0,s2)=1+2=3, G(t=1,s3)=2, G(t=2,s4)=0 Episode 3: G(t=0,s1)=1+3=4, G(t=1,s2)=3, G(t=2,s4)=0

First-visit MC:

• s1: visits in ep1 (4) and ep3 (4). V(s1) = (4+4)/2 = 4
• s2: visits in ep1 (3), ep2 (3), ep3 (3). V(s2) = (3+3+3)/3 = 3
• s3: visits in ep1 (2), ep2 (2). V(s3) = (2+2)/2 = 2
• s4: terminal, V(s4) = 0 Q2: Explain the difference between exploring starts and ε-greedy for MC control.

Exploring starts ensures all (s,a) pairs are visited by starting each episode from a randomly chosen state-action pair. This guarantees infinite visits to all pairs as number of episodes → ∞. It's simple but impractical — we can't control the starting state in many real environments.

ε-greedy ensures all actions are taken with at least ε/|A| probability from every state. This allows exploring starts to be dropped (the policy itself handles exploration). However, the learned policy is ε-soft (never fully greedy), so we learn a near-optimal policy, not the truly optimal one.

In practice, ε-greedy is preferred because we can anneal ε → 0 over time, converging to the optimal deterministic policy. Q3: Derive the importance sampling ratio for a trajectory of length T and show how it's used in off-policy MC.

For a trajectory $S_0, A_0, R_1, S_1, A_1, ..., S_T$:

Under behavior policy $b$: $P_b(\tau) = \prod_{t=0}^{T-1} b(A_t|S_t) P(S_{t+1}|S_t, A_t)$

Under target policy $\pi$: $P_\pi(\tau) = \prod_{t=0}^{T-1} \pi(A_t|S_t) P(S_{t+1}|S_t, A_t)$

The importance sampling ratio:

$\rho_{0:T-1} = \frac{P_\pi(\tau)}{P_b(\tau)} = \frac{\prod \pi(A_t|S_t) P(S_{t+1}|S_t, A_t)}{\prod b(A_t|S_t) P(S_{t+1}|S_t, A_t)} = \prod_{t=0}^{T-1} \frac{\pi(A_t|S_t)}{b(A_t|S_t)}$

Notice that the transition probabilities cancel — the ratio depends only on the action probabilities, not the environment dynamics.

For off-policy MC prediction:

$V(s) = \frac{\sum_{t \in \mathcal{T}(s)} \rho_{t:T-1} G_t}{\sum_{t \in \mathcal{T}(s)} \rho_{t:T-1}}$

(weighted importance sampling — divides by sum of importance weights for normalization) Q4: An MC agent learns to play Blackjack. After 10,000 episodes, state values seem stable but the agent still loses money. Diagnose the problem.

Possible diagnoses:

1. Insufficient state representation: Blackjack requires knowing the sum of cards AND the dealer's visible card AND whether you have an ace. If the agent only tracks the sum, it's missing critical information.
2. Policy is suboptimal due to low ε: If ε was annealed too quickly, the agent may have converged to a locally optimal ε-greedy policy that's still globally suboptimal.
3. Exploration didn't cover all states: Some state combinations (e.g., sum=20 with dealer showing 6) might be rare in natural play. If exploring starts wasn't used and ε is small, these states might have poor value estimates.
4. High variance in returns: Blackjack has high reward variance (win/loss is all-or-nothing). Even 10,000 episodes might not be enough for low-variance estimates, especially for rare states.
5. Incorrect discount factor: If γ < 1 in Blackjack (which is actually episodic), the agent might undervalue the current hand's true win probability.

Fix: Increase exploration, ensure state visits include all relevant (sum, dealer, ace) triples, use every-visit MC for more data efficiency, and increase the number of episodes. Q5: Compare the variance of first-visit MC and TD(0) for policy evaluation. Why is MC variance higher?

MC variance: $G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + ...$ — the sum of many random variables. If each reward has variance $\sigma^2$, then under the assumption of independent rewards:

$\text{Var}(G_t) = \sigma^2 + \gamma^2 \sigma^2 + \gamma^4 \sigma^2 + ... = \frac{\sigma^2}{1-\gamma^2}$

For γ = 0.99: Var(G) ≈ 50σ². The variance grows as the effective horizon expands.

TD(0) variance: TD(0) uses $R_{t+1} + \gamma V(S_{t+1})$, which has:

$\text{Var}(TD \text{ target}) = \text{Var}(R_{t+1}) + \gamma^2 \text{Var}(V_{estimate}) \approx \sigma^2 + \text{(small bootstrap bias)}$

TD's lower variance comes from only having one source of randomness at a time (the immediate reward), plus the current value estimate which has been averaged over many updates.

Tradeoff: MC is unbiased but high variance. TD is biased (uses current estimate) but lower variance. This is the classic bias-variance tradeoff. Q6: In on-policy MC control with ε-greedy, why can't we set ε=0 and learn the optimal policy?

If ε=0, the policy is deterministic (always greedy). The agent will only take the current best action from each state. It will never explore alternative actions that might have higher long-term value.

Without exploration:

• The agent might settle on a suboptimal action because the value of the optimal action was never accurately estimated
• State-action pairs not taken never get their Q-values updated
• The agent can't discover better actions

This is the exploration-exploitation dilemma: narrowing ε too fast trades off exploration for exploitation, potentially locking in a suboptimal policy.

Standard practice: start with ε=0.5 or ε=1.0, anneal to ε=0.01 or 0.05, and never fully set ε=0 (or do so only after very many episodes when you're confident the policy has converged). Q7: For off-policy MC, if the target policy is deterministic and the behavior policy takes a different action at time t, what happens to the importance sampling ratio and the corresponding return?

If $\pi(A_t|S_t) = 0$ (deterministic target policy chooses action a*, but behavior policy took action a' ≠ a*):

$\rho_{t:T-1} = \prod_{k=t}^{T-1} \frac{\pi(A_k|S_k)}{b(A_k|S_k)}$

At step t: $\pi(A_t|S_t) = 0$ (since the target policy would never take action A_t). This makes the entire product 0.

When $\rho = 0$, the sample contributes nothing to the weighted average. The episode's return from that point forward is effectively discarded for learning about the target policy.

This means off-policy MC is very sample-inefficient when the target and behavior policies differ significantly — most episodes are partially or entirely discarded. Q8: Propose a way to reduce the variance of MC returns without introducing bias.

Control variates: Use a baseline to reduce variance. Instead of estimating $V(s)$ from raw returns:

$V(s) \leftarrow V(s) + \alpha[(G_t - b(s)) - V(s)]$

where $b(s)$ is a baseline (e.g., current estimate of $V(s)$). Since $\mathbb{E}[b(s)]$ is subtracted and expected back, the estimate remains unbiased, but the variance is reduced if $b(s)$ correlates with $G_t$.

This is the same trick used in policy gradient methods (REINFORCE with baseline). The optimal baseline is $\mathbb{E}[G_t^2] / \mathbb{E}[G_t]$, but the simplest is the current value estimate itself.

Disadvantage: This introduces bias if the baseline is correlated with the gradient estimate. In practice, using the current value estimate as a baseline is biased but reduces variance enough that it's nearly always beneficial. Q9: Given episodes with γ=0.9. At t=0, state A, action X, reward 5. At t=1, state B, action Y, reward -2. At t=2, state C (terminal). Compute first-visit MC return for (A,X).

$G_0 = R_1 + \gamma R_2 + \gamma^2 R_3 = 5 + 0.9(-2) + 0.9^2(0) = 5 - 1.8 = 3.2$

The first-visit MC return for (A,X) is 3.2.

If this is the only episode where (A,X) appears, then Q(A,X) = 3.2.

For incremental update with α=0.2: Q_new = Q_old + 0.2(3.2 - Q_old) If Q_old = 0: Q_new = 0.64 Q10: Explain why weighted importance sampling has lower variance than ordinary importance sampling for off-policy MC.

Ordinary importance sampling:

$V(s) = \frac{1}{|\mathcal{T}(s)|} \sum_{t \in \mathcal{T}(s)} \rho_t G_t$

Each return is scaled by the importance ratio $\rho_t$. If the behavior and target policies are very different, individual $\rho_t$ can be very large (e.g., 1000), giving that single return 1000× weight. Other returns might have near-zero $\rho_t$. The result: estimator variance is dominated by the occasional huge $\rho_t$, leading to extremely high variance.

Weighted importance sampling:

$V(s) = \frac{\sum_{t \in \mathcal{T}(s)} \rho_t G_t}{\sum_{t \in \mathcal{T}(s)} \rho_t}$

The denominator normalizes the weights so they sum to 1. This bounds the maximum contribution of any single sample to at most 1 (if all other weights are 0). The variance is much lower because no single sample can dominate.

Tradeoff: Weighted IS is biased (estimate converges to $V^\pi$ only asymptotically, not in expectation for finite samples), but has finite variance. Ordinary IS is unbiased but has infinite variance in many practical cases.

5. 🔗 Cross-References

• Previous: Dynamic Programming (Week 3) — GPI framework that MC implements
• Next: TD Learning (Week 5) — Combining MC and DP ideas
• Related: Q-Learning & SARSA (Week 6) — Off-policy and on-policy TD control
• External: Sutton & Barto, "Reinforcement Learning" — Chapters 5-6 on MC and TD Join Discord PreviousDynamic ProgrammingNextTD Learning