Neural Sync Active

Policy Gradients: REINFORCE and Actor-Critic

Registry Synced

Policy Gradients: REINFORCE and Actor-Critic

979 words

5 min read

Reading compass

Now · 🎯 Learning Objectives

Policy Gradients: REINFORCE and Actor-Critic

🎯 Learning Objectives

Understand why policy gradients are needed beyond value-based methods
Derive the policy gradient theorem
Implement REINFORCE with baseline
Explain advantage estimation in actor-critic methods

📋 Prerequisites

Neural network basics
Q-learning concepts
Probability and expectation

1. 📖 Core Content

1.1 Why Policy Gradients?

Value-based methods (Q-learning, DQN): Learn Q-values, derive policy implicitly (

\pi(s) = \arg\max_a Q(s,a)

) Policy gradient methods: Directly learn the policy

\pi_\theta(a|s)

using gradient ascent on expected return. Advantages:

Naturally handles continuous action spaces
Can learn stochastic policies
Better convergence properties (follows true gradient)
Can incorporate domain knowledge via policy architecture

1.2 The Policy Gradient Theorem

Objective:

J(\theta) = \mathbb{E}_{\pi_\theta}[G_0]

(expected return) Gradient:

\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}[\nabla_\theta \log \pi_\theta(a|s) \cdot Q^{\pi_\theta}(s,a)]

The gradient is the expected product of:

The score function $\nabla_\theta \log \pi_\theta(a|s)$ (direction to increase action probability)
The action value $Q^{\pi_\theta}(s,a)$ (how good was this action?)

1.3 REINFORCE


python
def reinforce(env, policy, episodes=1000, gamma=0.99, lr=1e-3):
    """REINFORCE (Monte Carlo Policy Gradient)"""
    optimizer = torch.optim.Adam(policy.parameters(), lr=lr)
    for episode in range(episodes):
        states, actions, rewards = [], [], []
        # Collect one episode
        state = env.reset()
        done = False
        while not done:
            action = policy.sample_action(state)
            next_state, reward, done = env.step(action)
            states.append(state)
            actions.append(action)
            rewards.append(reward)
            state = next_state
        # Compute discounted returns
        returns = []
        G = 0
        for r in reversed(rewards):
            G = r + gamma * G
            returns.insert(0, G)
        returns = torch.tensor(returns)
        # Normalize returns (reduce variance)
        returns = (returns - returns.mean()) / (returns.std() + 1e-8)
        # Compute loss and update
        loss = 0
        for t in range(len(states)):
            log_prob = policy.get_log_prob(states[t], actions[t])
            loss += -log_prob * returns[t]  # Negative for gradient ascent
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return policy

1.4 Actor-Critic

Actor-critic combines policy gradients (actor) with value function learning (critic):

Component	Role	Output	Loss
Actor $\pi_\theta(a	s)$	Policy	Action distribution
Critic $V_\phi(s)$	Value function	State value	MSE: $(G_t - V_\phi(s))^2$

Advantage:

A(s,a) = Q(s,a) - V(s) = r + \gamma V(s') - V(s)

(TD error)


python
class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=256):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU()
        )
        self.actor = nn.Linear(hidden_dim, action_dim)  # Policy logits
        self.critic = nn.Linear(hidden_dim, 1)          # Value
    def forward(self, state):
        features = self.fc(state)
        action_logits = self.actor(features)
        value = self.critic(features)
        return action_logits, value

📝 Practice Questions

Q1

Q1

Q1

Q1: Derive the policy gradient for a Gaussian policy $\pi_\theta(a|s) = \mathcal{N}(\mu_\theta(s), \sigma^2)$ .

The log probability: $\log \pi_\theta(a|s) = -\frac{1}{2}\log(2\pi\sigma^2) - \frac{(a-\mu_\theta(s))^2}{2\sigma^2}$

Gradient w.r.t. $\mu$ : $\nabla_\mu \log \pi_\theta(a|s) = \frac{a - \mu_\theta(s)}{\sigma^2}$

The policy gradient: $\nabla_\theta J = \mathbb{E}[\frac{a - \mu_\theta(s)}{\sigma^2} \cdot Q(s,a)]$

This means:

If $Q(s,a) > 0$ and $a > \mu$ (action better than average, higher than mean): gradient increases $\mu$

If $Q(s,a) < 0$ and $a > \mu$ (action worse than average, higher than mean): gradient decreases $\mu$

The Gaussian policy naturally balances exploration (via $\sigma$ ) and exploitation (via $\mu$ ). Q2

Q2

Q2

Q2

Q2

Q2: Why does adding a baseline (like V(s)) reduce variance in policy gradients?

The REINFORCE gradient: $\nabla J = \mathbb{E}[\nabla \log \pi(a|s) \cdot Q(s,a)]$

If we subtract a baseline b(s) that doesn't depend on a: $\nabla J = \mathbb{E}[\nabla \log \pi(a|s) \cdot (Q(s,a) - b(s))]$

This is still unbiased because $\mathbb{E}[\nabla \log \pi(a|s) \cdot b(s)] = b(s) \cdot \mathbb{E}[\nabla \log \pi(a|s)] = b(s) \cdot \nabla \mathbb{E}[1] = 0$

The optimal baseline is $V(s)$ — the expected return from state s. Using A(s,a) = Q(s,a) - V(s) (the advantage) gives the lowest variance because it centers the gradient signal around zero.

Intuition: $Q(s,a) - V(s)$ measures "how much better/worse was this action compared to average?" This removes the variance from state-dependent return magnitudes. Q3

Q3

Q3

Q3

Q3

Q3

Q3

Q3

Q3: In actor-critic, why does the critic also need to be trained?

The critic estimates V(s), which is used to compute the advantage A(s,a) = r + γV(s') - V(s).

If the critic is inaccurate, the advantage estimate is wrong, and the actor updates in suboptimal directions. Training the critic with the MSE loss $(r + \gamma V(s') - V(s))^2$ ensures it learns better value estimates over time.

This creates a co-adaptation process:

Critic learns better value estimates

Better values → better advantage estimates

Better advantages → better policy updates

Better policy → different state distribution → critic needs to learn more

Both actor and critic improve together, which is why actor-critic is more stable than either pure policy gradients or pure value-based methods. Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4: Compare REINFORCE (Monte Carlo policy gradient) with actor-critic (TD policy gradient).

Aspect	REINFORCE	Actor-Critic
Update frequency	End of episode	Every step
Returns	Full episode return (unbiased, high variance)	TD target (biased, low variance)
Variance	High (wait for full return)	Low (immediate TD error)
Bias	Unbiased	Biased (if critic is inaccurate)
Learning speed	Slow (wait for episode end)	Fast (online updates)
Eligibility for continuing tasks	No (needs episodes)	Yes

REINFORCE trades variance for unbiasedness — it uses actual returns but they're noisy. Actor-critic trades bias for lower variance — TD targets are smoother but systematically biased if the critic is wrong.

Best of both: Generalized Advantage Estimation (GAE) smoothly interpolates between MC returns and TD targets using λ parameter.

</details> * * * ## 🔗 Cross-References - **Next**: [PPO](/notes/04-degree-electives-bsda5007-reinforcement-learning-week10-10-ppo) - **Previous**: [DQN](/notes/04-degree-electives-bsda5007-reinforcement-learning-week07-07-dqn) - **Video**: BSDA5007 Week 9-10 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Deep Q-Networks (DQN)**](/notes/04-degree-electives-bsda5007-reinforcement-learning-week07-07-dqn)[Next**Actor-Critic Methods**](/notes/04-degree-electives-bsda5007-reinforcement-learning-week09-09-actor-critic)