Policy Gradients: REINFORCE and Actor-Critic

🎯 Learning Objectives

• Understand why policy gradients are needed beyond value-based methods
• Derive the policy gradient theorem
• Implement REINFORCE with baseline
• Explain advantage estimation in actor-critic methods

📋 Prerequisites

• Neural network basics
• Q-learning concepts
• Probability and expectation

---

1. 📖 Core Content

1.1 Why Policy Gradients?

• Naturally handles continuous action spaces
• Can learn stochastic policies
• Better convergence properties (follows true gradient)
• Can incorporate domain knowledge via policy architecture

1.2 The Policy Gradient Theorem

• The score function $\nabla_\theta \log \pi_\theta(a|s)$ (direction to increase action probability)
• The action value $Q^{\pi_\theta}(s,a)$ (how good was this action?)

1.3 REINFORCE

python
Run Inline
def reinforce(env, policy, episodes=1000, gamma=0.99, lr=1e-3):
    """REINFORCE (Monte Carlo Policy Gradient)"""
    optimizer = torch.optim.Adam(policy.parameters(), lr=lr)
    for episode in range(episodes):
        states, actions, rewards = [], [], []
        # Collect one episode
        state = env.reset()
        done = False
        while not done:
            action = policy.sample_action(state)
            next_state, reward, done = env.step(action)
            states.append(state)
            actions.append(action)
            rewards.append(reward)
            state = next_state
        # Compute discounted returns
        returns = []
        G = 0
        for r in reversed(rewards):
            G = r + gamma * G
            returns.insert(0, G)
        returns = torch.tensor(returns)
        # Normalize returns (reduce variance)
        returns = (returns - returns.mean()) / (returns.std() + 1e-8)
        # Compute loss and update
        loss = 0
        for t in range(len(states)):
            log_prob = policy.get_log_prob(states[t], actions[t])
            loss += -log_prob * returns[t]  # Negative for gradient ascent
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return policy

1.4 Actor-Critic

python
Run Inline
class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=256):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU()
        )
        self.actor = nn.Linear(hidden_dim, action_dim)  # Policy logits
        self.critic = nn.Linear(hidden_dim, 1)          # Value
    def forward(self, state):
        features = self.fc(state)
        action_logits = self.actor(features)
        value = self.critic(features)
        return action_logits, value

---

📝 Practice Questions

Q1

<strong>Q1

<strong>Q1

<strong>Q1</strong>: Derive the policy gradient for a Gaussian policy $\pi_\theta(a|s) = \mathcal{N}(\mu_\theta(s), \sigma^2)$.

The log probability: $\log \pi_\theta(a|s) = -\frac{1}{2}\log(2\pi\sigma^2) - \frac{(a-\mu_\theta(s))^2}{2\sigma^2}$

Gradient w.r.t. $\mu$: $\nabla_\mu \log \pi_\theta(a|s) = \frac{a - \mu_\theta(s)}{\sigma^2}$

The policy gradient: $\nabla_\theta J = \mathbb{E}[\frac{a - \mu_\theta(s)}{\sigma^2} \cdot Q(s,a)]$

This means:

• If $Q(s,a) > 0$ and $a > \mu$ (action better than average, higher than mean): gradient increases $\mu$
• If $Q(s,a) < 0$ and $a > \mu$ (action worse than average, higher than mean): gradient decreases $\mu$

The Gaussian policy naturally balances exploration (via $\sigma$) and exploitation (via $\mu$). Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2</strong>: Why does adding a baseline (like V(s)) reduce variance in policy gradients?

The REINFORCE gradient: $\nabla J = \mathbb{E}[\nabla \log \pi(a|s) \cdot Q(s,a)]$

If we subtract a baseline b(s) that doesn't depend on a: $\nabla J = \mathbb{E}[\nabla \log \pi(a|s) \cdot (Q(s,a) - b(s))]$

This is still unbiased because $\mathbb{E}[\nabla \log \pi(a|s) \cdot b(s)] = b(s) \cdot \mathbb{E}[\nabla \log \pi(a|s)] = b(s) \cdot \nabla \mathbb{E}[1] = 0$

The optimal baseline is $V(s)$ — the expected return from state s. Using A(s,a) = Q(s,a) - V(s) (the advantage) gives the lowest variance because it centers the gradient signal around zero.

Intuition: $Q(s,a) - V(s)$ measures "how much better/worse was this action compared to average?" This removes the variance from state-dependent return magnitudes. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: In actor-critic, why does the critic also need to be trained?

The critic estimates V(s), which is used to compute the advantage A(s,a) = r + γV(s') - V(s).

If the critic is inaccurate, the advantage estimate is wrong, and the actor updates in suboptimal directions. Training the critic with the MSE loss $(r + \gamma V(s') - V(s))^2$ ensures it learns better value estimates over time.

This creates a co-adaptation process:

1. Critic learns better value estimates
2. Better values → better advantage estimates
3. Better advantages → better policy updates
4. Better policy → different state distribution → critic needs to learn more

Both actor and critic improve together, which is why actor-critic is more stable than either pure policy gradients or pure value-based methods. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: Compare REINFORCE (Monte Carlo policy gradient) with actor-critic (TD policy gradient).

REINFORCE trades variance for unbiasedness — it uses actual returns but they're noisy. Actor-critic trades bias for lower variance — TD targets are smoother but systematically biased if the critic is wrong.

Best of both: Generalized Advantage Estimation (GAE) smoothly interpolates between MC returns and TD targets using λ parameter.

</details> * * * ## 🔗 Cross-References - **Next**: [PPO](/notes/04-degree-electives-bsda5007-reinforcement-learning-week10-10-ppo) - **Previous**: [DQN](/notes/04-degree-electives-bsda5007-reinforcement-learning-week07-07-dqn) - **Video**: BSDA5007 Week 9-10 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Deep Q-Networks (DQN)**](/notes/04-degree-electives-bsda5007-reinforcement-learning-week07-07-dqn)[Next**Actor-Critic Methods**](/notes/04-degree-electives-bsda5007-reinforcement-learning-week09-09-actor-critic)