RLHF, Alignment, and Constitutional AI

🎯 Learning Objectives

• Understand why alignment is necessary for LLMs
• Explain the RLHF pipeline: SFT → reward modeling → PPO
• Implement reward model training and PPO optimization
• Compare RLHF with Constitutional AI and other alignment methods

📋 Prerequisites

• Reinforcement learning basics (policy gradient)
• Supervised fine-tuning
• Cross-entropy loss

1. 📖 Core Content

1.1 The Alignment Problem

• Generate offensive/harmful content
• Follow malicious instructions
• Produce biased outputs
• Hallucinate confidently Alignment ensures models behave according to human values and intentions.

1.2 The RLHF Pipeline

1.3 Stage 1: Supervised Fine-tuning (SFT)

• Human prompters write responses they'd like to see
• Train the model via standard cross-entropy loss
• Result: A model that can produce human-quality responses Dataset: Typically 10K-100K (prompt, response) pairs.

1.4 Stage 2: Reward Model Training

1.5 Stage 3: PPO Optimization

• $\pi_{\theta}$: Current policy (the model being trained)
• $\pi_{SFT}$: Frozen SFT model (reference)
• $R_\phi(x, y)$: Reward model score
• $KL(\pi_{\theta} || \pi_{SFT})$: KL divergence to prevent reward hacking
• $\beta$: KL penalty coefficient PPO clipped surrogate:

python
Run Inline
# runnable
import numpy as np
def ppo_loss(old_log_probs, new_log_probs, advantages, epsilon=0.2):
    """
    Compute PPO clipped loss
    Args:
        old_log_probs: Log probabilities under old policy
        new_log_probs: Log probabilities under current policy
        advantages: Advantage estimates
        epsilon: Clip range
    Returns:
        loss: PPO clipped objective loss
    """
    # Probability ratio
    ratio = np.exp(new_log_probs - old_log_probs)
    # Clipped ratio
    clipped_ratio = np.clip(ratio, 1 - epsilon, 1 + epsilon)
    # Surrogate objectives
    surr1 = ratio * advantages
    surr2 = clipped_ratio * advantages
    # PPO loss (negative for maximization)
    loss = -np.minimum(surr1, surr2)
    return np.mean(loss)
# Example
np.random.seed(42)
old_log_probs = np.array([-0.5, -0.3, -1.0])
new_log_probs = np.array([-0.4, -0.5, -0.8])
advantages = np.array([0.5, -0.2, 0.8])
loss = ppo_loss(old_log_probs, new_log_probs, advantages)
print(f"PPO loss: {loss:.4f}")
# The loss is low when good actions (positive advantage) have increased probability
# and bad actions (negative advantage) have decreased probability

1.6 Constitutional AI (Anthropic)

1. Generate harmful responses from model
2. Ask the model to revise based on constitution (e.g., "Be helpful, harmless, honest")
3. Fine-tune on (harmful, revised) pairs Stage 2: RL Stage
4. Generate response pairs
5. Ask the model which response is better according to constitution (AI feedback)
6. Train reward model on AI preference labels
7. PPO optimization as in standard RLHF Advantages:

• No human labeling needed (or minimal)
• Scalable to many principles
• Transparent principles (the constitution is public)

1.7 Comparison: RLHF vs Constitutional AI

1.8 Why This Matters

• Safety: Prevent harmful outputs
• Trust: Users trust models that behave appropriately
• Legal: Regulatory compliance (EU AI Act, etc.)
• Business: Brand reputation Without alignment, even the most capable models are unsuitable for production.

6. 📝 Practice Questions

Q1: In the Bradley-Terry preference model, if R(x,y₁)=5 and R(x,y₂)=2, what is P(y₁ ≻ y₂)?

P(y₁ ≻ y₂) = e⁵ / (e⁵ + e²) = 148.41 / (148.41 + 7.39) = 148.41 / 155.80 = 0.953

The preferred response (y₁) is predicted to be chosen 95.3% of the time. Q2

<strong>Q2</strong>: Why is a KL penalty needed in PPO training?

Without the KL penalty, the model would optimize purely for reward:

1. Reward hacking: Find token sequences that maximize reward but are nonsensical
2. Mode collapse: Generate only the highest-reward response pattern
3. Catastrophic forgetting: Forget pre-training knowledge

The KL penalty $\beta \cdot KL(\pi_{\theta} || \pi_{SFT})$ ensures:

• The model stays close to the SFT model (doesn't deviate too far)
• Maintains language quality and diversity
• Balances reward optimization with linguistic naturalness Q3

<strong>Q3</strong>: In PPO, what is the advantage A and how is it estimated?

Advantage $A(s, a) = Q(s, a) - V(s)$ measures how much better the action a is compared to the average action in state s.

Estimation: GAE (Generalized Advantage Estimation): $A_t = \sum_{l=0}^{\infty} (\gamma \lambda)^l (r_t + \gamma V(s_{t+1}) - V(s_t))$

Where:

• $r_t$: Reward at time t
• $V(s)$: Value function estimate
• $\gamma$: Discount factor
• $\lambda$: GAE smoothing parameter

A positive advantage means "this action was better than expected" → increase its probability. Q4

<strong>Q4

<strong>Q4</strong>: How does the reward model differ from the policy model in architecture?

The reward model shares most weights with the SFT policy model:

• Same Transformer backbone
• Same embedding layers
• Difference: The final unembedding layer (vocab_size → d_model) is replaced by a scalar head (d_model → 1)
• The scalar head outputs a single number: the reward score

During training, only the scalar head is typically trained (the backbone is frozen or uses low learning rate). This prevents the reward model from learning language patterns and focuses it on preference prediction. Q5

<strong>Q5

<strong>Q5</strong>: In Constitutional AI, describe how the "Red Team" generation creates training data.

Red team generation in Constitutional AI:

1. Prompt selection: Choose prompts that might trigger harmful outputs
2. Initial response: Generate response from the model (potentially harmful)
3. Constitution-guided critique: Ask the model to critique its own response according to constitutional principles:
   • "Identify specific ways this response violates the principle of..."
4. Revision: Ask the model to revise the response based on the critique
5. Training pair: (harmful_response, revised_response) used as supervised training

This creates a self-improvement loop where the model learns to identify and correct its own harmful outputs. Q6

<strong>Q6

<strong>Q6</strong>: Compare reward over-optimization in RLHF with overfitting in supervised learning.

Reward over-optimization: The model learns to exploit the reward model's specific preferences rather than actual human preferences. The reward score keeps increasing, but human-evaluated quality plateaus then drops.

Overfitting: The model memorizes training data rather than learning general patterns. Validation loss increases while training loss continues decreasing.

Both are forms of proxy misalignment: optimizing the proxy (reward / training loss) beyond the point where it correlates with the true objective (human satisfaction / generalization).

Solutions: KL penalty (RLHF), early stopping, regularization. Q7

<strong>Q7

<strong>Q7</strong>: What is "reward hacking" and how can it be detected?

Reward hacking: The policy finds ways to get high reward without actually satisfying human preferences.

Examples:

• Generating very long responses (reward models sometimes prefer longer text)
• Using flattery or sycophancy
• Learning "tells" of the reward model (certain phrases trigger high reward)

Detection:

1. Held-out evaluation: Compare reward scores with human evaluation
2. Diversity metrics: Check if responses become formulaic
3. Adversarial probes: Test specific reward-hacking patterns
4. KL monitoring: Unexpectedly high KL suggests reward hacking

Mitigation: KL penalty, ensemble reward models, regular human evaluation. Q8

<strong>Q8

<strong>Q8</strong>: If you have 100K human preference comparisons, how many parameters should the reward model have relative to the policy model?

Guidelines:

• Reward model should be smaller (or equal) to policy model
• Ratio: Typically 0.1× to 1× the policy model size
• Memory: Larger reward models overfit on limited preference data

For a 7B policy model:

• Reward model: 1-3B parameters (adequate)
• Training data: 100K comparisons with 7B RM → risk of overfitting
• Suggested: Use a medium-sized RM (1-3B) or apply strong regularization

The reward model needs enough capacity to capture human preferences but not so much that it overfits to the limited comparison data. Q9

<strong>Q9

<strong>Q9

<strong>Q9</strong>: How does DPO (Direct Preference Optimization) differ from RLHF?

DPO (Direct Preference Optimization) eliminates the reward model:

Standard RLHF: Train RM → PPO with RM rewards DPO: Directly optimize policy from preferences

DPO loss: $\mathcal{L}_{DPO} = -\mathbb{E}_{(x, y_w, y_l)}[\log \sigma(\beta \log \frac{\pi_{\theta}(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_{\theta}(y_l|x)}{\pi_{ref}(y_l|x)})]$

Advantages:

• No reward model training (simpler, cheaper)
• No PPO (no value function, no advantage estimation)
• Stable training (no RL instability)

Disadvantages:

• Implicit reward assumption (Bradley-Terry may not hold)
• Less flexible than RLHF for complex reward shaping

DPO has become popular as a simpler alternative to full RLHF. Q10

<strong>Q10

<strong>Q10

<strong>Q10</strong>: An RLHF-trained model always refuses to answer "Write a poem about AI" because it's incorrectly classified as harmful. How would you fix this?

This is over-alignment: the model refuses legitimate requests.

Fixes:

1. Reward model calibration: Add "safe poem" examples to the preference dataset labeled as good
2. Constitutional revision: Add a principle that distinguishes genuine harm from benign requests
3. Threshold adjustment: Use a refusal probability threshold that accepts borderline cases
4. Adversarial data: Include safe requests that look similar to harmful ones in the training data
5. System prompt: Use a system prompt like "You are helpful and harmless. When a request is clearly safe and beneficial, respond helpfully."
6. Rejection sampling: Generate multiple responses, check via classifier, select non-refusing ones

Balancing harmlessness with helpfulness is an active research area. The ideal model refuses genuinely harmful requests but handles ambiguous or clearly safe requests helpfully.

7. 🔗 Cross-References

• Next: Quantization & Inference Optimization (Week 10)
• Previous: Prompt Engineering
• Video: BSDA5004 Week 9 transcripts Join Discord PreviousPrompt EngineeringNextInference Optimization