Deep Q-Networks: Experience Replay, Target Networks, Rainbow

🎯 Learning Objectives

• Understand why function approximation is needed for large state spaces
• Implement DQN with experience replay and target networks
• Explain Double DQN and Dueling DQN improvements
• Understand the Rainbow DQN architecture

📋 Prerequisites

• Q-learning fundamentals
• Neural network basics
• Experience replay concept

---

1. 📖 Core Content

1.1 From Tabular to Deep RL

1.2 DQN Architecture

python
Run Inline
import torch.nn as nn
class DQN(nn.Module):
    """Deep Q-Network for Atari games"""
    def __init__(self, n_actions):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(4, 32, 8, stride=4),  # 84→20
            nn.ReLU(),
            nn.Conv2d(32, 64, 4, stride=2), # 20→9
            nn.ReLU(),
            nn.Conv2d(64, 64, 3, stride=1), # 9→7
            nn.ReLU(),
            nn.Flatten()
        )
        self.fc = nn.Sequential(
            nn.Linear(64 * 7 * 7, 512),
            nn.ReLU(),
            nn.Linear(512, n_actions)
        )
    def forward(self, x):
        features = self.conv(x)
        return self.fc(features)

1.3 Key DQN Innovations

Experience Replay

python
Run Inline
class ReplayBuffer:
    def __init__(self, capacity=100000):
        self.buffer = deque(maxlen=capacity)
    def push(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))
    def sample(self, batch_size):
        batch = random.sample(self.buffer, batch_size)
        states, actions, rewards, next_states, dones = zip(*batch)
        return (torch.tensor(np.array(states)),
                torch.tensor(actions),
                torch.tensor(rewards, dtype=torch.float32),
                torch.tensor(np.array(next_states)),
                torch.tensor(dones, dtype=torch.float32))

Target Network

python
Run Inline
# Initialize
online_net = DQN(n_actions)
target_net = DQN(n_actions)
target_net.load_state_dict(online_net.state_dict())
# Training step
states, actions, rewards, next_states, dones = buffer.sample(batch_size)
# Q-values from online network
q_values = online_net(states).gather(1, actions.unsqueeze(1))
# TD targets from target network (no gradients through target!)
with torch.no_grad():
    max_q_next = target_net(next_states).max(1)[0]
    targets = rewards + gamma * max_q_next * (1 - dones)
loss = F.mse_loss(q_values.squeeze(), targets)
# Periodically update target network
if step % target_update == 0:
    target_net.load_state_dict(online_net.state_dict())

1.4 DQN Improvements

---

📝 Practice Questions

Q1

<strong>Q1

<strong>Q1

<strong>Q1</strong>: In DQN, why does the target network stabilize training?

The TD target: $r + \gamma \max_{a'} Q(s', a'; \theta)$ depends on the same parameters $\theta$ as the Q-network being updated. This creates a moving target — every update changes both the prediction AND the target, causing oscillations and divergence.

The target network uses frozen parameters $\theta^-$: $r + \gamma \max_{a'} Q(s', a'; \theta^-)$

This decouples the target from the online network. The target changes slowly (only when we copy weights every C steps), providing a stable learning signal.

Without a target network, DQN training often diverges, especially in environments with complex dynamics. Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2</strong>: Double DQN computes $a^* = \arg\max_a Q_{online}(s', a)$ and then uses $Q_{target}(s', a^*)$. Why does this help?

Standard Q-learning: $r + \gamma \max_a Q(s', a)$ — uses max, which overestimates Q-values because max of noisy estimates is higher than true max.

Double DQN:

1. Select action using online net: $a^* = \arg\max_a Q_{online}(s', a)$
2. Evaluate using target net: $r + \gamma Q_{target}(s', a^*)$

By decoupling selection and evaluation, Double DQN reduces overestimation. The online network might select a suboptimal action, but the target network evaluates it fairly, producing a lower (more accurate) target.

In Atari games, Double DQN finds better policies than standard DQN, especially later in training when overestimation bias accumulates. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: How does experience replay break the correlation between consecutive training samples?

In online RL, transitions are highly correlated: (s_t, a_t, r_t, s_{t+1}) and (s_{t+1}, a_{t+1}, r_{t+1}, s_{t+2}) are almost identical. Training on consecutive samples creates a biased, non-stationary training distribution.

Experience replay stores many transitions and samples uniformly at random from the buffer. This:

1. Breaks temporal correlation (samples are from different episodes, different times)
2. Increases data efficiency (each experience used multiple times)
3. Smoothes the training distribution (reduces variance)

Without replay, neural networks quickly overfit to recent experiences, causing catastrophic forgetting and unstable learning. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4

<strong>Q4

<strong>Q4</strong>: Dueling DQN computes Q(s,a) = V(s) + A(s,a) - mean(A(s,a)). Why split into V and A?

In many states, the choice of action doesn't matter much (e.g., driving on an empty highway — all actions keep the car on the road). The Q-value is dominated by state value V(s), not action advantage A(s,a).

The dueling architecture explicitly separates:

• V(s): Value of being in state s (how good is this state)
• A(s,a): Advantage of taking action a in state s (how much better is this action than average)

Q(s,a) = V(s) + (A(s,a) - mean(A))

Advantage: The V stream learns the state's value even when actions have similar outcomes. This gives the network a richer learning signal and better policy evaluation.

The dueling architecture significantly improves performance on tasks where actions don't affect the environment in many states.

</details> * * * ## 🔗 Cross-References - **Next**: [Policy Gradients](/notes/04-degree-electives-bsda5007-reinforcement-learning-week08-08-policy-gradients) - **Previous**: [Q-Learning & SARSA](/notes/04-degree-electives-bsda5007-reinforcement-learning-week06-06-q-learning-sarsa) - **Video**: BSDA5007 Week 7-8 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Q-Learning & SARSA**](/notes/04-degree-electives-bsda5007-reinforcement-learning-week06-06-q-learning-sarsa)[Next**Policy Gradients**](/notes/04-degree-electives-bsda5007-reinforcement-learning-week08-08-policy-gradients)