Fine-tuning Methods: LoRA, Adapters, and PEFT

🎯 Learning Objectives

• Distinguish full fine-tuning from parameter-efficient fine-tuning (PEFT)
• Implement LoRA (Low-Rank Adaptation) and understand its mathematical basis
• Understand adapter layers, prefix tuning, and prompt tuning
• Choose the right fine-tuning method based on task requirements

📋 Prerequisites

• Transformer architecture
• Matrix factorization (SVD, low-rank decomposition)
• Pre-training concepts

1. 📖 Core Content

1.1 The Fine-tuning Problem

• Storing a full copy of gradients (O(params) memory)
• Multiple GPUs/TPUs for distributed training
• Full copy per task (impractical for many tasks) Solution: Parameter-Efficient Fine-Tuning (PEFT) — update only a tiny fraction of parameters while keeping the base model frozen. (Diagram)

1.2 LoRA: Low-Rank Adaptation

Intuition

Formal Definition

• $r$: Rank (typically 1-64)
• $A$: Random initialized, $B$: Zero initialized
• Only $A$ and $B$ are trained
• $W_0$ is frozen Parameter savings: From $d \times k$ to $r \times (d + k)$ For a 4096×4096 weight matrix with r=8:
• Full update: 16.8M parameters
• LoRA: 8 × (4096 + 4096) = 65,536 parameters
• 256× reduction!

python
Run Inline
# runnable
import numpy as np
class LoRALayer:
    """
    LoRA adaptation layer
    Args:
        d: Input dimension
        k: Output dimension
        r: Rank (low intrinsic rank)
    """
    def __init__(self, d, k, r=8):
        self.d = d
        self.k = k
        self.r = r
        # Original weight (frozen)
        np.random.seed(42)
        self.W = np.random.randn(d, k) * 0.1
        # LoRA matrices (trainable)
        # A: random Gaussian initialization
        self.A = np.random.randn(r, k) * 0.01
        # B: zero initialization (so ΔW=0 at start)
        self.B = np.zeros((d, r))
    def forward(self, x):
        """Forward pass with LoRA"""
        # Original path (frozen)
        original = x @ self.W
        # LoRA path (trainable)
        lora = x @ self.B @ self.A
        return original + lora
    def parameter_count(self):
        """Count trainable parameters"""
        lora_params = self.d * self.r + self.r * self.k
        full_params = self.d * self.k
        return lora_params, full_params
# Example
layer = LoRALayer(d=4096, k=4096, r=8)
lora_params, full_params = layer.parameter_count()
print(f"LoRA parameters: {lora_params:,}")
print(f"Full parameters: {full_params:,}")
print(f"Reduction: {full_params / lora_params:.0f}×")

1.3 Adapter Layers

• $W_{down} \in \mathbb{R}^{d \times m}$: Down-projection (m << d)
• $W_{up} \in \mathbb{R}^{m \times d}$: Up-projection
• Typically m = d/4 to d/64 Position: After attention or FFN sublayer (before residual)

1.4 Prefix Tuning & Prompt Tuning

1.5 Why This Matters

• Fine-tune GPT-3 (175B) on a single GPU: LoRA reduces memory from ~350GB to ~20GB
• Deploy multiple tasks: Store one base model + small adapters per task
• Quick iteration: Minutes instead of days for fine-tuning
• No catastrophic forgetting: Base model remains unchanged

6. 📝 Practice Questions

Q1: For GPT-3 (d_model=12288), what rank r gives a 1000× parameter reduction with LoRA?

Full W^Q: d_model² = 12288² = 151M params per matrix With LoRA: r × (d + k) = r × 2 × 12288 = r × 24576

For 1000× reduction: 151M / (r × 24576) = 1000 r = 151M / (24576 × 1000) = 6.14

So r ≈ 6 gives ~1000× parameter reduction. Q2

<strong>Q2</strong>: Why is B initialized to zero and A to random? What happens if both are random?

If B and A are both random non-zero, the initial forward pass would be: h = W₀x + BAx

The BA term would add random noise to the output, shifting the model's behavior from its pre-trained state. Training would need to "correct" this shift.

By initializing B=0, the initial output is exactly the pre-trained output (h = W₀x). The LoRA update starts from zero and smoothly learns the task-specific adaptation. This preserves the pre-trained model's behavior at initialization. Q3

<strong>Q3

<strong>Q3</strong>: A 7B LLaMA model is fine-tuned with LoRA r=16 on all attention projection matrices. How many trainable parameters?

LLaMA 7B: d_model=4096, 32 layers, 4 attention projections per layer (Q, K, V, O).

Per matrix: LoRA params = r × (d + k) = 16 × (4096 + 4096) = 131,072 All attention: 32 × 4 × 131,072 = 16,777,216 ≈ 16.8M

Total model: 7B parameters Trainable: 16.8M / 7B = 0.24% of all parameters

This is why LoRA can fine-tune on a single GPU — only 0.24% of parameters need gradients and optimizer states. Q4

<strong>Q4</strong>: Compare the memory requirements during training for full fine-tuning vs LoRA (assume 7B model, FP16, Adam optimizer).

Full fine-tuning memory:

• Model weights: 7B × 2 bytes (FP16) = 14 GB
• Gradients: 7B × 2 bytes = 14 GB
• Optimizer states (Adam): 7B × 2 × 4 bytes = 56 GB (2 copies of momentum + variance in FP32)
• Activations: ~10-20 GB (batch size dependent)
• Total: ~94-104 GB

LoRA fine-tuning (r=16, 16.8M trainable):

• Model weights (frozen): 7B × 2 = 14 GB
• LoRA weights: 16.8M × 2 = 0.034 GB
• Gradients (LoRA only): 16.8M × 2 = 0.034 GB
• Optimizer states (LoRA only): 16.8M × 8 = 0.134 GB
• Activations: ~10-20 GB
• Total: ~24-34 GB

LoRA requires ~1/3 the memory, making single-GPU fine-tuning feasible. Q5

<strong>Q5</strong>: What is the "intrinsic dimension" hypothesis that LoRA relies on?

The intrinsic dimension hypothesis states that while neural network weights are high-dimensional, the space of effective adaptations (meaningful changes through fine-tuning) has much lower intrinsic dimension. In other words, you don't need to move in all d² directions to adapt a model — you can achieve good performance by moving in a low-dimensional subspace of rank r.

LoRA exploits this by parameterizing the update ΔW as a rank-r matrix BA. The hypothesis holds empirically: r=1 or r=2 can work for many tasks, and r=8 often matches full fine-tuning performance. Q6

<strong>Q6</strong>: How does adapter-based fine-tuning differ from LoRA in terms of inference latency?

Adapters: Add extra layers sequentially. The forward pass goes: Attention → Adapter → Residual → FFN → Adapter → Residual. This adds 2 extra matrix multiplications per layer, increasing latency by 5-10%.

LoRA: The update BA can be merged into W₀ after training: W' = W₀ + BA

During inference, W' is a single matrix (same size as W₀). There's zero additional latency compared to the original model.

This makes LoRA preferable for deployment scenarios where latency matters. Q7

<strong>Q7

<strong>Q7</strong>: In what scenarios would you prefer full fine-tuning over LoRA?

Full fine-tuning might be preferred when:

1. Maximum performance is critical: LoRA typically achieves 90-99% of full FT performance. For state-of-the-art results, full FT may edge ahead.
2. Sufficient compute is available: If you have the GPUs, full FT is simpler (no rank to tune).
3. The task requires significant domain shift: If the target domain is very different from pre-training (e.g., medical text → code), larger capacity may help.
4. You can afford per-task storage: If deploying only 1-2 models.

But for most practical scenarios, LoRA matches full FT performance at a fraction of the cost. Q8

<strong>Q8</strong>: How does QLoRA extend LoRA for even more memory-efficient fine-tuning?

QLoRA (Quantized LoRA) adds:

1. 4-bit NormalFloat quantization of the base model (instead of FP16)
2. Double quantization: Quantize the quantization constants too
3. Paged optimizers: Use CPU memory for optimizer states

Memory savings:

• Base model: 7B × 0.5 bytes (NF4) = 3.5 GB (vs 14 GB for FP16)
• Same LoRA parameters
• Total: ~10 GB for 7B model fine-tuning

QLoRA makes it possible to fine-tune 65B models on a single 48GB GPU. Q9

<strong>Q9

<strong>Q9</strong>: Why might applying LoRA to all weight matrices not be optimal?

Applying LoRA everywhere:

1. Increases total trainable parameters (diminishing the benefit)
2. May overfit on small datasets
3. Not all matrices benefit equally from adaptation

Research shows:

• Attention weights (W_Q, W_K, W_V) benefit most from LoRA
• Output projection (W_O) benefits moderately
• FFN weights benefit least
• Optimal strategy: apply LoRA only to selected matrices

For most tasks, applying LoRA to W_Q and W_V (or all attention projections) with r=8-64 is sufficient. Q10: If you're deploying a model for 100 different tasks, compare storage requirements for full fine-tuning vs LoRA.

Assume: 7B model, LoRA adds 16.8M params per task, FP16 storage.

Full fine-tuning: 100 × 14 GB = 1,400 GB (one full copy per task)

LoRA: 1 × 14 GB (base model) + 100 × 0.034 GB (adapters) = 17.4 GB

LoRA saves 1,382 GB (~98.8% reduction). Moreover, LoRA adapters can be loaded/unloaded dynamically, so only the base model + one adapter need to be in GPU memory at a time.

7. 🔗 Cross-References

• Next: Prompt Engineering (Week 8)
• Previous: Tokenization
• Video: BSDA5004 Week 7 transcripts Join Discord PreviousTokenization MethodsNextPrompt Engineering