Proximal Policy Optimization: Clipped Surrogate and Trust Regions

🎯 Learning Objectives

• Understand why unconstrained policy updates can collapse performance
• Derive and implement the PPO clipped surrogate objective
• Compare PPO-Clip and PPO-Penalty variants
• Implement PPO with multiple epochs of minibatch updates
• Apply PPO to continuous and discrete action spaces

📋 Prerequisites

• Actor-Critic Methods (Week 9): A2C, advantage estimation
• Policy Gradients (Week 8): REINFORCE, policy gradient theorem
• Trust Region Concepts: Why large policy updates are dangerous

1. 📖 Core Content

1.1 Intuition: Why Not Just Use A2C with Larger Learning Rates?

• Stable: Rarely catastrophically fails
• Sample efficient: Reuses data for multiple updates
• Simple: No complex second-order optimization (unlike TRPO)
• General: Works for discrete, continuous, and image-based tasks

1.2 The Problem with Large Policy Updates

1. The policy changes dramatically
2. New policy generates very different state distribution
3. Old advantage estimates no longer apply
4. Policy gets worse, which changes distribution further
5. Collapse The core issue: our loss function approximates the true return only for the current policy. For a very different policy, the loss is inaccurate.

1.3 PPO's Solution: Clipped Surrogate Objective

1.3.1 The Probability Ratio

1.3.2 The Clipped Objective

• If $A_t > 0$ (good action): The objective increases with $r_t$, but only up to $1+\epsilon$. Beyond that, the gradient is 0 — no incentive to make a good action even more likely.
• If $A_t < 0$ (bad action): The objective decreases as $r_t$ drops, but only to $1-\epsilon$. Below that, the gradient is 0 — no incentive to make a bad action even less likely. The clipping acts as a regularizer: it removes the incentive to change the policy beyond a certain amount per update, regardless of how large the advantage is.

Worked Example 1: PPO Clipping in Action

• If r < 1-ε: clip to lower bound (1-ε)A < rA → min chooses (1-ε)A (lower) → objective is reduced
• If r > 1+ε: clip to upper bound (1+ε)A > rA → min chooses rA (lower) → objective is reduced Wait, I got confused. Let me re-examine. For A > 0:
• clip(r, 1-ε, 1+ε)A =
  • If r < 1-ε: (1-ε)A (which is < rA since A>0)
  • If r between bounds: rA
  • If r > 1+ε: (1+ε)A (which is < rA since A>0)
• min(rA, clip(r, 1-ε, 1+ε)A) will pick the smaller value
  • If r < 1-ε: (1-ε)A < rA → pick (1-ε)A (clipped, lower bound)
  • If r between: both are rA → pick rA
  • If r > 1+ε: (1+ε)A < rA → pick (1+ε)A (clipped, upper bound) For A < 0:
• clip(r, 1-ε, 1+ε)A =
  • If r < 1-ε: (1-ε)A (which is > rA since A<0 and 1-ε < r means multiply by negative flips sign)
  • If r between bounds: rA
  • If r > 1+ε: (1+ε)A (which is > rA since A<0)
• min(rA, clip(r, 1-ε, 1+ε)A) will pick the smaller value
  • If r < 1-ε: rA < (1-ε)A → pick rA (unclipped)
  • If r between: both rA → pick rA
  • If r > 1+ε: rA < (1+ε)A → pick rA (unclipped) So for negative advantages, the clipping doesn't activate! The gradient can freely push the probability down. This asymmetry is intentional: decreasing bad action probability is always safe. Wait no, that's not right either. Let me be very precise. For A < 0, r < 1-ε:
• clip(r)A = (1-ε)A
• rA - (1-ε)A = A(r - (1-ε))
• Since A < 0 and r - (1-ε) < 0, A(r - (1-ε)) > 0 (product of two negatives)
• So rA > (1-ε)A
• min(rA, (1-ε)A) = (1-ε)A (clipped) OK I was wrong. Let me redo: For A < 0, r < 1-ε:
• A is negative, r < 1-ε
• rA = (large negative × negative?) → positive? No.
• r < 1-ε, so r is small. A is negative. rA is a small negative × negative = hmm... Actually, A < 0 means the advantage is negative. r is a ratio ≥ 0 (it's a ratio of probabilities). For A < 0, r < 1-ε:
• rA is negative (since r>0, A<0)
• clip(r)A = (1-ε)A is also negative (since 1-ε>0, A<0)
• Since r < 1-ε and A < 0: rA > (1-ε)A (multiplying inequality by negative flips it, but we need to be careful: r < 1-ε and A < 0, so rA = r×A > 1-ε×A = (1-ε)A... actually no. Multiplying an inequality by a negative flips it. A < 0, so r < 1-ε → rA > (1-ε)A.)
• So min(rA, (1-ε)A) = (1-ε)A (clipped!) For A < 0, r > 1+ε:
• clip(r)A = (1+ε)A
• r > 1+ε and A < 0: rA < (1+ε)A (multiplying by negative flips the inequality)
• So min(rA, (1+ε)A) = rA (unclipped) Ah, so for negative A:
• If r is very low (much less likely): clip to (1-ε)A (makes objective clip to less negative, i.e., the gradient to reduce probability is clipped)
• If r is very high (much more likely): no clipping (gradient flows freely to reduce probability) So PPO clips BOTH increasing AND decreasing probability by too much. Good. Let me just present a clearer version.

1.4 PPO Algorithm

text
for iteration = 1 to N:
    # Collect data with current policy
    for actor = 1 to N_workers:
        Run policy π_θ_old for T timesteps
        Store (s_t, a_t, r_t, s_{t+1})
    # Compute advantages
    Compute GAE advantages A_t using V_φ(s_t)
    # Optimize surrogate objective (multiple epochs)
    for epoch = 1 to K:
        # Shuffle data into minibatches
        for minibatch in data:
            # Compute ratio
            r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t)
            # Clipped surrogate
            L_CLIP = min(r_t * A_t, clip(r_t, 1-ε, 1+ε) * A_t)
            # Value function loss
            L_V = (V_φ(s_t) - G_t)^2  # or use clipped value loss
            # Entropy bonus
            L_S = H(π_θ(a|s_t))
            # Combined loss
            L = -L_CLIP + c1 * L_V - c2 * L_S
            # Update θ, φ
            θ ← θ + α ∇_θ L
            φ ← φ + α_φ ∇_φ L_V
    # Update old policy
    θ_old ← θ

1.4.1 Key Hyperparameters

1.5 PPO-Penalty (Adaptive KL)

• If $D_{KL} > d_{\text{target}} \times 1.5$: $\beta \leftarrow \beta \times 2$ (penalize more)
• If $D_{KL} < d_{\text{target}} / 1.5$: $\beta \leftarrow \beta / 2$ (relax penalty) This automatically adjusts the constraint based on observed KL divergence.

1.6 PPO-Clip vs PPO-Penalty

1.7 Edge Cases & Gotchas

• Clipping behavior: If all ratios are within [1-ε, 1+ε], the clipping has no effect. This means either: (a) the policy is already optimal, or (b) the learning rate is too low. Check the fraction of clipped samples (target: 10-20%).
• Value function clipping: PPO often uses a clipped value loss to match the policy clipping:

• Multiple epochs with same data: The same trajectory is reused for K epochs. This is off-policy learning (the behavior policy is π_old, the target is π). PPO's clipping protects against the distribution mismatch.
• Early termination (early stopping): If the KL divergence exceeds a threshold (e.g., 0.01 per update), early-stop the inner loop. This is a safety check.

1.8 Why This Matters

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Very Low Clipping Fraction

Pitfall 2: Too Many Epochs Without Early Stopping

Pitfall 3: Not Normalizing States or Advantages

Pitfall 4: Continuous Action Noise Schedule

4. 📝 Practice Questions

Q1: For a state-action pair with old probability 0.1, new probability 0.5, advantage +3.0, ε=0.2, compute the clipped objective contribution.

$r = 0.5/0.1 = 5.0$

clip(r, 0.8, 1.2) = 1.2

Unclipped: r·A = 5.0 × 3.0 = 15.0 clipped: clip(r)·A = 1.2 × 3.0 = 3.6

min(15.0, 3.6) = 3.6

The clipped contribution is 3.6. Without clipping, the objective contribution would be 15.0, creating a huge gradient that would push the probability even higher. The clipping caps the contribution at 3.6, providing no gradient to increase the ratio beyond 1.2. Q2: Explain why PPO reuses the same trajectory for K epochs (multiple gradient updates) while A2C uses it once.

A2C: On-policy. Each transition is used for exactly one gradient update, then discarded. This is strictly on-policy but sample-inefficient.

PPO: Uses importance sampling to be approximately on-policy. The probability ratio $r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$ corrects for the fact that the data was collected from an older policy.

PPO can safely reuse data for multiple epochs because:

1. The ratio $r_t(\theta)$ downweights transitions where the new policy differs from the old policy
2. The clipping mechanism prevents the objective from being dominated by a few transitions with extreme ratios
3. The KL divergence is bounded (implicitly via clipping, explicitly via early stopping)

This reuse improves sample efficiency by 3-15× compared to A2C (depending on K). Q3: In PPO-Clip, if ε=0.5, what's the effective range of the probability ratio before clipping kicks in?

[1-ε, 1+ε] = [0.5, 1.5]

A ratio of 0.5 means the new policy is half as likely to take this action as the old policy. A ratio of 1.5 means the new policy is 1.5× more likely.

Larger ε (0.5) allows more policy change per update. This is suitable for:

• Environments with very stable dynamics (simulated robotics)
• When you trust your advantage estimates
• Tasks requiring fast learning

Smaller ε (0.1) allows less change. Suitable for:

• Noisy environments (Atari)
• When advantage estimates are unreliable
• More conservative learning Q4: A PPO agent trains for 10M steps and the clipped fraction is 0% (none of the ratios exceed [1-ε, 1+ε]). What could be wrong?

If no ratios exceed the clipping bounds:

1. Learning rate too low: The policy isn't changing, so all ratios ≈ 1.0. Increase learning rate.
2. Network capacity too low: The network can't learn the optimal policy. Increase network size.
3. Vanishing gradients: The policy is saturated (e.g., actions are very confident). Check for tanh saturation or softmax degeneracy.
4. Convergence: The policy might actually be optimal. But this is unlikely after 10M steps if the clipping is exactly 0%.
5. Entropy coefficient too high: If entropy bonus dominates, the policy stays near-uniform. Check entropy over training.

In practice, the clipped fraction should be 10-30% for healthy PPO training. Q5: Compare PPO's handling of continuous vs discrete action spaces.

Discrete actions (e.g., Atari):

• Policy outputs logits for each action, converted to probabilities via softmax
• Ratio $r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$ is straightforward (ratio of two probabilities)
• Clipping applies directly to probability ratios

Continuous actions (e.g., robotic control):

• Policy outputs mean $\mu_\theta(s_t)$ and log_std $\sigma_\theta$ for each action dimension
• Action probability: $\pi_\theta(a_t|s_t) = \mathcal{N}(a_t|\mu_\theta(s_t), \sigma_\theta^2)$
• Ratio $r_t(\theta) = \frac{\mathcal{N}(a_t|\mu_\theta, \sigma_\theta^2)}{\mathcal{N}(a_t|\mu_{\theta_{\text{old}}}, \sigma_{\theta_{\text{old}}}^2)}$ — ratio of probability densities
• Clipping same as discrete; ratio can vary more smoothly

For continuous actions, PPO often benefits from:

• Tanh squashing: Apply tanh to action samples to bound them within [-1, 1]
• Log-probability correction: Adjust log-probability for the tanh transformation
• State-dependent standard deviation: Instead of fixed log_std, predict it from state Q6: Explain the relationship between PPO's clipping and TRPO's trust region constraint.

TRPO (Trust Region Policy Optimization) constrains policy updates using KL divergence:

$\max_\theta \mathbb{E}\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} A(s,a)\right] \text{ s.t. } \mathbb{E}[D_{KL}(\pi_{\theta_{\text{old}}} \| \pi_\theta)] \leq \delta$

This requires computing the KL constraint and solving it with conjugate gradient — complex and computationally expensive.

PPO-Clip achieves a similar effect without explicit KL computation:

1. TRPO limits KL divergence explicitly — guarantees the new policy stays within a trust region
2. PPO limits probability ratio — implicitly limits KL, since KL is bounded by a function of the ratio

For small changes, $D_{KL}(\pi_{\theta_{\text{old}}} \| \pi_\theta) \approx \mathbb{E}[(r_t - 1)^2] / 2$. So clipping r to [1-ε, 1+ε] approximately limits KL.

PPO-Penalty even more explicitly uses KL (adding a penalty term), making it even closer to TRPO but with first-order optimization.

Key difference: TRPO's constraint is "hard" (must be satisfied). PPO's clipping is "soft" (can be violated, but clipped gradient discourages it). PPO is simpler and faster; TRPO has stronger theoretical guarantees. Q7: Design a PPO agent for a continuous control task where actions are torques in [-1, 1] and state is joint angles and velocities. Describe the network architecture.

Network architecture:

pseudo
State (e.g., 24 dims: joint angles + velocities)
    │
    ├── Linear(256) ──► LayerNorm ──► tanh
    │
    ├── Linear(256) ──► LayerNorm ──► tanh
    │
    ├── Policy Head ──► Linear(128) ──► tanh
    │       │
    │       ├── μ head: Linear(action_dim) → tanh (to bound [-1, 1])
    │       │
    │       └── log_std: Parameter(action_dim) or Linear(action_dim)
    │
    └── Value Head ──► Linear(128) ──► tanh ──► Linear(1)

Key design choices:

1. LayerNorm: Helps stabilize training with varying input distributions
2. Tanh activation: Bounds activations, prevents extreme outputs
3. Separate policy/value heads: Sharing low-level features, separate high-level processing
4. Tanh on μ: Ensures action means are within [-1, 1]
5. Learned log_std: Can be state-independent (single parameter) or state-dependent (network output)

Training details:

• γ = 0.99 (typically OK for control tasks with reasonable episode lengths)
• GAE λ = 0.95
• ε = 0.2
• K = 10 epochs per iteration
• Entropy coefficient = 0.01 (supplementary bonus, not critical for continuous control) Q8: Why does PPO benefit from early stopping in the inner loop?

PPO's inner loop performs K epochs over the same trajectory data. Each epoch applies gradients that change θ further from θ_old.

After many epochs, even with clipping, the KL divergence between π_θ and π_θ_old can grow large. When KL is large:

1. The importance sampling correction (ratio r_t) becomes unreliable
2. The state distribution mismatch between old and new policy introduces bias
3. The clipping is a blunt instrument — it prevents extreme ratios but doesn't prevent gradual drift

Early stopping monitors the approximate KL (e.g., average KL per minibatch). If KL > threshold (e.g., 0.01 * 1.5), stop the inner loop. This provides a safety net.

Without early stopping, you'd need very conservative parameters (small ε, small K). Early stopping allows more aggressive learning: if KL stays low, keep optimizing; if KL spikes, stop and collect fresh data. Q9: In RLHF (Reinforcement Learning from Human Feedback), PPO is used to fine-tune language models. How does PPO need to be modified for this setting?

RLHF fine-tunes a language model to maximize a reward score from a human preference model. Key modifications:

1. KL penalty with original model: The PPO loss includes a KL penalty between the current model and the original (unfine-tuned) model: $L = \mathbb{E}[r_t·A_t] - \beta·D_{KL}(\pi_{\text{orig}} \| \pi_\theta)$ This prevents the model from diverging too far from the original pretrained model.
2. Per-token PPO: The policy generates text one token at a time. The action is the next token, the state is the sequence so far. PPO operates per-token, but the reward is only available at the end of the sequence.
3. Credit assignment: The advantage for each token is computed using GAE over the sequence. Only the final reward signal (from the reward model) is available — it's assigned to each token proportionally to recency.
4. No value function sharing: The value function (critic) is a separate head added to the language model. It's trained to predict the expected reward from any point in the sequence.
5. Batched generations: The LM generates K completion samples per prompt. All are scored by the reward model. The PPO update uses these scored generations.
6. Reference model: A frozen copy of the original LM serves as both (a) the KL constraint target and (b) the behavior policy for importance sampling.

This approach is used for models like ChatGPT, Claude, and Llama-2-Chat. Q10: For a robotic reaching task, PPO with ε=0.2 fails to learn (policy stays near-random). Propose 3 potential fixes.

1. Increase ε to 0.3-0.5: If the policy is stuck at random, the allowed change per update (ε=0.2) might be too restrictive. A larger ε allows more aggressive improvement.
2. Reduce K (epochs) and increase learning rate: If K is high (e.g., 80), the policy might be repeatedly optimizing the same data without changing. Reduce K to 3-5 and increase learning rate by 5-10×.
3. Check reward scaling: If rewards are very small (e.g., -0.001 per step), advantages are tiny, and PPO's gradient is negligible. Scale rewards so that the typical advantage magnitude is ~1.0. Or normalize advantages within the batch.
4. Fix exploration: If the policy's action noise (log_std) collapses to near-zero, the policy doesn't explore and stays at the initial poor policy. Initialize log_std to 0 (std=1) and don't let it drop below -2 (std=0.14) initially.
5. Increase GAE λ: If λ is too low (e.g., 0.8), the advantage estimate is myopic. Increase λ to 0.97 to capture longer-term consequences of actions.
6. Add a dense reward shaping: If the sparse reaching reward (0 or 1 at the end) is too hard to optimize, add a shaping reward based on distance to the target:

$r_{\text{shaped}} = r_{\text{sparse}} + \alpha·(\text{dist}_{t-1} - \text{dist}_t)$

This gives the agent immediate feedback for moving toward the target.

5. 🔗 Cross-References

• Previous: Actor-Critic Methods (Week 9) — A2C foundation
• Next: Exploration Methods (Week 11) — Beyond PPO's entropy bonus
• Related: Policy Gradients (Week 8) — Baseline policy gradient concepts
• External: Schulman et al., "Proximal Policy Optimization Algorithms" (arXiv 2017)
• External: Schulman et al., "Trust Region Policy Optimization" (ICML 2015) Join Discord PreviousActor-Critic MethodsNextExploration Methods