Actor-Critic Methods: A2C, A3C, and Advantage Estimation

🎯 Learning Objectives

• Understand why combining policy gradients with value functions improves sample efficiency
• Derive the advantage function and its role in reducing gradient variance
• Implement Advantage Actor-Critic (A2C)
• Understand A3C's asynchronous architecture
• Analyze the bias-variance tradeoff in advantage estimation

📋 Prerequisites

• Policy Gradients (Week 8): REINFORCE, score function, baseline
• TD Learning (Week 5): Value function estimation, TD error
• Deep Learning basics: Neural network optimization, sharing parameters

1. 📖 Core Content

1.1 Intuition: Why Combine Actor and Critic?

• Actor: The policy $\pi_\theta(a|s)$ that decides which action to take
• Critic: The value function $V_\phi(s)$ or $Q_\phi(s,a)$ that evaluates how good the state/action is The critic provides a low-variance baseline for the actor, reducing gradient variance. The actor improves the policy based on the critic's evaluation. This is the most widely used architecture in modern RL (PPO, SAC, TD3 are all actor-critic methods).

1.2 The Policy Gradient with a Critic

1.2.1 Why Advantage?

1.3 Estimating the Advantage

1.3.1 TD Error as Advantage

1.3.2 Monte Carlo Advantage

1.3.3 Generalized Advantage Estimation (GAE)

• $\lambda = 0$: 1-step TD advantage (high bias, low variance)
• $\lambda = 1$: Monte Carlo advantage (low bias, high variance)
• $\lambda = 0.95$: typical in practice (balanced tradeoff)

1.4 A2C: Advantage Actor-Critic

1.4.1 A2C Loss Functions

1.4.2 A2C Algorithm

text
Initialize actor π_θ and critic V_φ networks
loop for T steps:
    for each worker i in parallel:
        Run policy πθ for n steps
        Store (s_t, a_t, r_t, s_{t+1}) in trajectory buffer
    Compute advantages using GAE
    for each worker i:
        Compute actor loss, critic loss, entropy bonus
    Gradient step on combined loss
    Update θ, φ

Worked Example 1: A2C Advantage Computation

• Step 1: s₁ → a₁ → r₁=1, s₂
• Step 2: s₂ → a₂ → r₂=0, s₃
• Step 3: s₃ → a₃ → r₃=2, s₄ (terminal) Current critic values: V(s₁)=0.5, V(s₂)=0.3, V(s₃)=0.1, V(s₄)=0. γ=0.9. Compute TD errors: δ₁ = 1 + 0.9(0.3) - 0.5 = 1 + 0.27 - 0.5 = 0.77 δ₂ = 0 + 0.9(0.1) - 0.3 = 0 + 0.09 - 0.3 = -0.21 δ₃ = 2 + 0.9(0) - 0.1 = 1.9 Compute GAE(γ=0.9, λ=0.95): A₁ = δ₁ + 0.9(0.95)(-0.21) + (0.9·0.95)²(1.9) = 0.77 + 0.855(-0.21) + 0.731(1.9) = 0.77 - 0.180 + 1.389 = 1.979 A₂ = δ₂ + 0.9(0.95)(1.9) = -0.21 + 1.625 = 1.415 A₃ = δ₃ = 1.9 The advantages are highest for a₁ and a₃ because they led to high rewards. a₂ had negative TD error (the actual reward was worse than expected), but the GAE positive future advantage compensates.

1.5 A3C: Asynchronous Advantage Actor-Critic

Key Differences from A2C

1.6 A2C vs A3C vs DQN vs PPO

1.7 Edge Cases & Gotchas

• Shared vs separate networks: Actor and critic can share early layers (common in A2C/A3C). This speeds learning but may cause interference — fix by orthogonal initialization.
• GAE λ sensitivity: The choice of λ significantly affects performance. Too low λ (0-0.9): high bias, potentially too conservative. Too high λ (0.99-1): high variance, unstable learning.
• Entropy coefficient: If entropy bonus is too large, the policy never converges (stays random). If too small, it converges prematurely to a suboptimal policy.
• Value function as baseline only: The advantage should center at ~0. If advantages are consistently positive, the value function is underestimating; if consistently negative, it's overestimating.
• n-step returns: Instead of GAE, using n-step returns for advantage ($\sum_{k=0}^{n-1} \gamma^k r_{t+k} + \gamma^n V(s_{t+n}) - V(s_t)$) is simpler and works well.

1.8 Why This Matters

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Not Normalizing Advantages

Pitfall 2: Sharing Too Many Parameters Between Actor and Critic

Pitfall 3: Setting Entropy Coefficient Too High or Too Low

Pitfall 4: Incorrect GAE Computation at Episode Boundaries

4. 📝 Practice Questions

Q1: Derive why any state-dependent baseline b(s) doesn't bias the policy gradient.

The policy gradient with baseline:

$\nabla_\theta J = \mathbb{E}_{s \sim d^\pi, a \sim \pi}[\nabla_\theta \log \pi_\theta(a|s) \cdot (Q^\pi(s,a) - b(s))]$

The bias introduced by b(s) is:

$\text{Bias} = \mathbb{E}_{s,a}[\nabla_\theta \log \pi_\theta(a|s) \cdot b(s)]$

$= \mathbb{E}_{s \sim d^\pi}[b(s) \cdot \mathbb{E}_{a \sim \pi}[\nabla_\theta \log \pi_\theta(a|s)]]$

Now, $\mathbb{E}_{a \sim \pi}[\nabla_\theta \log \pi_\theta(a|s)] = \sum_a \pi_\theta(a|s) \nabla_\theta \log \pi_\theta(a|s)$

$= \sum_a \nabla_\theta \pi_\theta(a|s) = \nabla_\theta \sum_a \pi_\theta(a|s) = \nabla_\theta 1 = 0$

Therefore, the bias term = 0 for any b(s) that doesn't depend on a. The optimal baseline that minimizes variance is $b(s) = V^\pi(s)$, making $A(s,a) = Q^\pi(s,a) - V^\pi(s)$ the advantage function. Q2: For GAE with λ=0, show that $A_t^{GAE} = \delta_t$ (TD error). For λ=1, show that $A_t^{GAE} = G_t - V(s_t)$ (MC advantage).

λ=0: $A_t^{GAE} = \sum_{l=0}^\infty (\gamma \cdot 0)^l \delta_{t+l} = \delta_t + 0 + 0 + ... = \delta_t$

This is exactly the 1-step TD error, giving highest bias/lowest variance.

λ=1: $A_t^{GAE} = \sum_{l=0}^\infty (\gamma \cdot 1)^l \delta_{t+l} = \delta_t + \gamma \delta_{t+1} + \gamma^2 \delta_{t+2} + ...$

Expanding: $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ $\gamma \delta_{t+1} = \gamma(r_{t+1} + \gamma V(s_{t+2}) - V(s_{t+1}))$ $\gamma^2 \delta_{t+2} = \gamma^2(r_{t+2} + \gamma V(s_{t+3}) - V(s_{t+2}))$

Summing (telescoping): = $r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + ... - V(s_t) = G_t - V(s_t)$

This is the MC advantage, giving lowest bias/highest variance. Q3: In A2C with 8 parallel workers, each collecting n=5 steps per update, what's the effective batch size? How does this compare to a single-worker A2C with n=40 steps?

With 8 workers × 5 steps = 40 (s, a, r) transitions per update. With 1 worker × 40 steps = 40 transitions per update.

The batch size in terms of transitions is the same (40).

However, the multi-worker approach has two advantages:

1. Diverse exploration: Each worker experiences different states, reducing correlation between samples. This is like having a more diverse replay buffer.
2. Wall-clock efficiency: With 8 workers, collecting 5 steps each takes ~5 step-times. Single worker needs 40 step-times (the environment step is often the bottleneck).

Multi-worker A2C gives better gradient estimates (lower variance) because the batch contains more diverse, less correlated samples. Q4: The entropy bonus in A2C is -c₂·H(π). Will this increase or decrease exploration? What happens if c₂ is too large?

The entropy $H(\pi) = -\sum_a \pi(a|s) \log \pi(a|s)$ is maximized when the policy is uniform (max entropy) and minimized when it's deterministic (zero entropy).

The loss includes -c₂·H(π), meaning we're subtracting entropy. Since we minimize the total loss, this encourages higher entropy (exploration).

If c₂ is too large:

• The policy remains near-uniform (never specializes)
• The agent never learns to exploit good actions
• The policy gradient signal is overwhelmed by the entropy bonus
• The agent effectively acts randomly

If c₂ is too small:

• The policy converges quickly to a deterministic policy
• The agent may converge to a suboptimal deterministic policy without exploring enough

Typical values: c₂ = 0.01 to 0.1 for continuous control, c₂ = 0.001 to 0.01 for Atari. Q5: Explain why A3C's asynchronous updates can lead to "gradient staleness" and how it's mitigated.

Gradient staleness: Worker A reads global parameters at time t, computes gradients for 5 steps, and sends updates at time t+5. Meanwhile, Workers B, C, D also computed and applied their gradients. Worker A's gradients were computed using stale parameters (from t) and are being applied to potentially very different parameters (at t+5).

This is like using a map from last week to navigate today — the terrain may have changed.

Mitigations:

1. n-step returns limit staleness: Each worker only collects n steps (e.g., 5-20) before updating, limiting the staleness window.
2. RMSProp/Adam with global learning rate: Adaptive optimizers are more robust to stale gradients.
3. Trust region methods (PPO): By clipping the surrogate objective, PPO limits how much each update can change the policy, making it more robust to stale gradients.
4. IMPALA's V-trace: Corrects for the off-policyness introduced by stale parameters using importance sampling.
5. Synchronous A2C avoids staleness entirely: all workers sync before each update, ensuring gradients are computed on the same parameters being updated.

Q6: An A2C agent's advantages are all positive (range [0.5, 2.0]) for an entire training run. What does this indicate?

Consistently positive advantages mean that the critic (value function) is underestimating the value of every state. For every state visited, the actual return is better than predicted.

This indicates a systematic bias in the critic:

1. The critic is too pessimistic: It consistently underestimates V(s)
2. The actor is outperforming expectations: The actual returns are better than what the critic expects

Consequences:

• The policy gradient update is always positive (increase log probability of all actions)
• Relative differences between actions are preserved (since advantages scale differently per state-action)
• The learning signal is weaker than it should be (advantages should be centered around 0)

Fix: Normalize advantages to have zero mean across the batch. This is standard practice in A2C implementations. Q7: Compare A2C with a batch size of 32 (single worker, 32 steps per update) vs A2C with batch size 32 (4 workers, 8 steps each).

The multi-worker version generally gives better gradients (lower variance) and faster wall-clock training. This is why A2C/A3C are preferred over single-worker variants. Q8: Derive the actor loss gradient for a single (s,a) transition with advantage A.

The actor loss for a single transition is:

$\mathcal{L}_{actor}(s,a) = -\log \pi_\theta(a|s) \cdot A$

The gradient is:

$\nabla_\theta \mathcal{L}_{actor} = -\nabla_\theta \log \pi_\theta(a|s) \cdot A$

For a Gaussian policy (continuous action space): $\pi_\theta(a|s) = \mathcal{N}(\mu_\theta(s), \sigma_\theta^2(s))$

$\log \pi_\theta(a|s) = -\frac{(a-\mu_\theta(s))^2}{2\sigma_\theta^2(s)} - \log(\sqrt{2\pi}\sigma_\theta(s))$

$\nabla_\theta \log \pi_\theta = \frac{a - \mu_\theta(s)}{\sigma_\theta^2(s)} \cdot \nabla_\theta \mu_\theta(s) - \frac{(a - \mu_\theta(s))^2 - \sigma_\theta^2(s)}{\sigma_\theta^3(s)} \cdot \nabla_\theta \sigma_\theta(s)$

For a categorical policy (discrete actions): $\pi_\theta(a|s) = \text{softmax}(h_\theta(s))_a$

$\nabla_\theta \log \pi_\theta(a|s) = \nabla_\theta h_\theta(a|s) - \sum_{a'} \pi_\theta(a'|s) \nabla_\theta h_\theta(a'|s)$

where $h_\theta(a|s)$ is the logit for action a.

The advantage A scales the gradient — actions with positive advantage are made more likely; actions with negative advantage are made less likely. Q9: In A2C, the critic loss $\mathcal{L}_{critic} = (G_t - V_\phi(s_t))^2$ is the MSE between the return and the predicted value. Why use MSE rather than absolute error?

MSE has a squared penalty (quadratic in error), while MAE has a linear penalty. For value estimation:

1. Gradient magnitude: MSE gradient is proportional to the error $(G_t - V_\phi(s_t))$, so large errors produce larger updates. This helps the critic quickly correct large mistakes.
2. Sensitivity to outliers: MSE is more sensitive to outliers (returns far from the predicted value). This is actually desirable in RL because rare, high-return episodes are informative — we want the critic to learn from them quickly.
3. Convergence properties: MSE corresponds to the variance-minimizing estimator (assuming Gaussian noise). The critic aims to estimate $\mathbb{E}[G_t|S_t=s]$, and under MSE, the minimizer is the conditional expectation.
4. Smoothness: MSE has smooth gradients everywhere, making optimization easier.

The downside of MSE is that for very large errors (e.g., initial training), gradients can be too large, causing training instability. This is one reason advantage normalization is used — it keeps the critic targets and predictions in a reasonable range. Q10: Explain the "deadly triad" in actor-critic methods with function approximation.

The "deadly triad" refers to the combination of three elements that can cause instability and divergence in RL:

1. Function approximation: Representing V(s) or π(a|s) with a neural network (not a table)
2. Bootstrapping: Updating based on current estimates (TD learning)
3. Off-policy learning: Learning about one policy from data generated by another

Actor-critic methods with off-policy data (e.g., using a replay buffer) combine all three:

1. Neural network parameterization (function approximation)
2. TD error for the critic update (bootstrapping)
3. Behavior policy differs from target policy when using stale data (off-policy)

Manifestations:

• Value function may diverge (grow without bound)
• Policy may oscillate between very different behaviors
• Learning may get stuck in poor local optima

Mitigations:

• A2C avoids off-policy learning: On-policy updates using fresh data from current policy
• PPO: Constrains policy changes to a trust region
• Target networks: Stabilize bootstrapping targets (used in DQN, SAC, TD3)
• Gradient clipping: Prevents huge updates from rare large errors

On-policy actor-critic (A2C, PPO) avoids the off-policy part of the triad, which is a key reason for their stability.

5. 🔗 Cross-References

• Previous: Policy Gradients (Week 8) — REINFORCE and baseline concept
• Next: PPO (Week 10) — Trust-region actor-critic
• Related: DQN (Week 7) — Value-based RL comparison
• External: Schulman et al., "Generalized Advantage Estimation" (ICLR 2016) Join Discord PreviousPolicy GradientsNextProximal Policy Optimization