Neural Sync Active

Fine-tuning Methods: LoRA, Adapters, and PEFT

Registry Synced

Fine-tuning Methods: LoRA, Adapters, and PEFT

1748 words

9 min read

Reading compass

Now · 🎯 Learning Objectives

Fine-tuning Methods: LoRA, Adapters, and PEFT

🎯 Learning Objectives

Distinguish full fine-tuning from parameter-efficient fine-tuning (PEFT)
Implement LoRA (Low-Rank Adaptation) and understand its mathematical basis
Understand adapter layers, prefix tuning, and prompt tuning
Choose the right fine-tuning method based on task requirements

📋 Prerequisites

Transformer architecture
Matrix factorization (SVD, low-rank decomposition)
Pre-training concepts

1. 📖 Core Content

1.1 The Fine-tuning Problem

Full fine-tuning: Update all model parameters (175B for GPT-3). This requires:

Storing a full copy of gradients (O(params) memory)
Multiple GPUs/TPUs for distributed training
Full copy per task (impractical for many tasks) Solution: Parameter-Efficient Fine-Tuning (PEFT) — update only a tiny fraction of parameters while keeping the base model frozen. (Diagram)

1.2 LoRA: Low-Rank Adaptation

Intuition

LoRA is based on the observation that weight updates during fine-tuning have low intrinsic rank. Instead of updating a full weight matrix

W \in \mathbb{R}^{d \times k}

, we learn a low-rank decomposition:

W' = W + \Delta W = W + BA

Where

B \in \mathbb{R}^{d \times r}

A \in \mathbb{R}^{r \times k}

, and

r \ll \min(d, k)

Formal Definition

For a pre-trained weight matrix

W_0 \in \mathbb{R}^{d \times k}

, LoRA constrains its update:

h = W_0 x + \Delta W x = W_0 x + BAx

Where:

$r$ : Rank (typically 1-64)
$A$ : Random initialized, $B$ : Zero initialized
Only $A$ and $B$ are trained
$W_0$ is frozen Parameter savings: From $d \times k$ to $r \times (d + k)$ For a 4096×4096 weight matrix with r=8:
Full update: 16.8M parameters
LoRA: 8 × (4096 + 4096) = 65,536 parameters
256× reduction!


python
# runnable
import numpy as np
class LoRALayer:
    """
    LoRA adaptation layer
    Args:
        d: Input dimension
        k: Output dimension
        r: Rank (low intrinsic rank)
    """
    def __init__(self, d, k, r=8):
        self.d = d
        self.k = k
        self.r = r
        # Original weight (frozen)
        np.random.seed(42)
        self.W = np.random.randn(d, k) * 0.1
        # LoRA matrices (trainable)
        # A: random Gaussian initialization
        self.A = np.random.randn(r, k) * 0.01
        # B: zero initialization (so ΔW=0 at start)
        self.B = np.zeros((d, r))
    def forward(self, x):
        """Forward pass with LoRA"""
        # Original path (frozen)
        original = x @ self.W
        # LoRA path (trainable)
        lora = x @ self.B @ self.A
        return original + lora
    def parameter_count(self):
        """Count trainable parameters"""
        lora_params = self.d * self.r + self.r * self.k
        full_params = self.d * self.k
        return lora_params, full_params
# Example
layer = LoRALayer(d=4096, k=4096, r=8)
lora_params, full_params = layer.parameter_count()
print(f"LoRA parameters: {lora_params:,}")
print(f"Full parameters: {full_params:,}")
print(f"Reduction: {full_params / lora_params:.0f}×")

1.3 Adapter Layers

Adapters insert small bottleneck layers between Transformer sublayers:

\text{Adapter}(h) = \text{ReLU}(h W_{down} + b_{down}) W_{up} + b_{up}

Where:

$W_{down} \in \mathbb{R}^{d \times m}$ : Down-projection (m << d)
$W_{up} \in \mathbb{R}^{m \times d}$ : Up-projection
Typically m = d/4 to d/64 Position: After attention or FFN sublayer (before residual)

1.4 Prefix Tuning & Prompt Tuning

Prefix Tuning: Prepends learnable "virtual tokens" to the input at each layer.

h = [h_{prefix}; h_{input}]

Where

h_{prefix} \in \mathbb{R}^{l \times d}

is learned (l = prefix length, ~10-100 tokens). Prompt Tuning: Similar but only adds virtual tokens to the input embedding (not every layer).

Method	Trainable Params	Performance	Complexity
Full Fine-tuning	100%	Best baseline	High
LoRA	0.1-1%	≈ Full FT	Low
Adapters	1-5%	≈ Full FT	Medium
Prefix Tuning	0.01-0.1%	Slightly below	Low
Prompt Tuning	0.001-0.01%	Task-dependent	Minimal

1.5 Why This Matters

PEFT methods make LLM fine-tuning accessible:

Fine-tune GPT-3 (175B) on a single GPU: LoRA reduces memory from ~350GB to ~20GB
Deploy multiple tasks: Store one base model + small adapters per task
Quick iteration: Minutes instead of days for fine-tuning
No catastrophic forgetting: Base model remains unchanged

6. 📝 Practice Questions

Q1: For GPT-3 (d_model=12288), what rank r gives a 1000× parameter reduction with LoRA?

Full W^Q: d_model² = 12288² = 151M params per matrix With LoRA: r × (d + k) = r × 2 × 12288 = r × 24576

For 1000× reduction: 151M / (r × 24576) = 1000 r = 151M / (24576 × 1000) = 6.14

So r ≈ 6 gives ~1000× parameter reduction. Q2

Q2: Why is B initialized to zero and A to random? What happens if both are random?

If B and A are both random non-zero, the initial forward pass would be: h = W₀x + BAx

The BA term would add random noise to the output, shifting the model's behavior from its pre-trained state. Training would need to "correct" this shift.

By initializing B=0, the initial output is exactly the pre-trained output (h = W₀x). The LoRA update starts from zero and smoothly learns the task-specific adaptation. This preserves the pre-trained model's behavior at initialization. Q3

Q3

Q3: A 7B LLaMA model is fine-tuned with LoRA r=16 on all attention projection matrices. How many trainable parameters?

LLaMA 7B: d_model=4096, 32 layers, 4 attention projections per layer (Q, K, V, O).

Per matrix: LoRA params = r × (d + k) = 16 × (4096 + 4096) = 131,072 All attention: 32 × 4 × 131,072 = 16,777,216 ≈ 16.8M

Total model: 7B parameters Trainable: 16.8M / 7B = 0.24% of all parameters

This is why LoRA can fine-tune on a single GPU — only 0.24% of parameters need gradients and optimizer states. Q4

Q4: Compare the memory requirements during training for full fine-tuning vs LoRA (assume 7B model, FP16, Adam optimizer).

Full fine-tuning memory:

Model weights: 7B × 2 bytes (FP16) = 14 GB

Gradients: 7B × 2 bytes = 14 GB

Optimizer states (Adam): 7B × 2 × 4 bytes = 56 GB (2 copies of momentum + variance in FP32)

Activations: ~10-20 GB (batch size dependent)

Total: ~94-104 GB

LoRA fine-tuning (r=16, 16.8M trainable):

Model weights (frozen): 7B × 2 = 14 GB

LoRA weights: 16.8M × 2 = 0.034 GB

Gradients (LoRA only): 16.8M × 2 = 0.034 GB

Optimizer states (LoRA only): 16.8M × 8 = 0.134 GB

Activations: ~10-20 GB

Total: ~24-34 GB

LoRA requires ~1/3 the memory, making single-GPU fine-tuning feasible. Q5

Q5: What is the "intrinsic dimension" hypothesis that LoRA relies on?

The intrinsic dimension hypothesis states that while neural network weights are high-dimensional, the space of effective adaptations (meaningful changes through fine-tuning) has much lower intrinsic dimension. In other words, you don't need to move in all d² directions to adapt a model — you can achieve good performance by moving in a low-dimensional subspace of rank r.

LoRA exploits this by parameterizing the update ΔW as a rank-r matrix BA. The hypothesis holds empirically: r=1 or r=2 can work for many tasks, and r=8 often matches full fine-tuning performance. Q6

Q6: How does adapter-based fine-tuning differ from LoRA in terms of inference latency?

Adapters: Add extra layers sequentially. The forward pass goes: Attention → Adapter → Residual → FFN → Adapter → Residual. This adds 2 extra matrix multiplications per layer, increasing latency by 5-10%.

LoRA: The update BA can be merged into W₀ after training: W' = W₀ + BA

During inference, W' is a single matrix (same size as W₀). There's zero additional latency compared to the original model.

This makes LoRA preferable for deployment scenarios where latency matters. Q7

Q7

Q7: In what scenarios would you prefer full fine-tuning over LoRA?

Full fine-tuning might be preferred when:

Maximum performance is critical: LoRA typically achieves 90-99% of full FT performance. For state-of-the-art results, full FT may edge ahead.

Sufficient compute is available: If you have the GPUs, full FT is simpler (no rank to tune).

The task requires significant domain shift: If the target domain is very different from pre-training (e.g., medical text → code), larger capacity may help.

You can afford per-task storage: If deploying only 1-2 models.

But for most practical scenarios, LoRA matches full FT performance at a fraction of the cost. Q8

Q8: How does QLoRA extend LoRA for even more memory-efficient fine-tuning?

QLoRA (Quantized LoRA) adds:

4-bit NormalFloat quantization of the base model (instead of FP16)

Double quantization: Quantize the quantization constants too

Paged optimizers: Use CPU memory for optimizer states

Memory savings:

Base model: 7B × 0.5 bytes (NF4) = 3.5 GB (vs 14 GB for FP16)

Same LoRA parameters

Total: ~10 GB for 7B model fine-tuning

QLoRA makes it possible to fine-tune 65B models on a single 48GB GPU. Q9

Q9

Q9: Why might applying LoRA to all weight matrices not be optimal?

Applying LoRA everywhere:

Increases total trainable parameters (diminishing the benefit)

May overfit on small datasets

Not all matrices benefit equally from adaptation

Research shows:

Attention weights (W_Q, W_K, W_V) benefit most from LoRA

Output projection (W_O) benefits moderately

FFN weights benefit least

Optimal strategy: apply LoRA only to selected matrices

For most tasks, applying LoRA to W_Q and W_V (or all attention projections) with r=8-64 is sufficient. Q10: If you're deploying a model for 100 different tasks, compare storage requirements for full fine-tuning vs LoRA.

Assume: 7B model, LoRA adds 16.8M params per task, FP16 storage.

Full fine-tuning: 100 × 14 GB = 1,400 GB (one full copy per task)

LoRA: 1 × 14 GB (base model) + 100 × 0.034 GB (adapters) = 17.4 GB

LoRA saves 1,382 GB (~98.8% reduction). Moreover, LoRA adapters can be loaded/unloaded dynamically, so only the base model + one adapter need to be in GPU memory at a time.

7. 🔗 Cross-References

Next: Prompt Engineering (Week 8)
Previous: Tokenization
Video: BSDA5004 Week 7 transcripts Join Discord PreviousTokenization MethodsNextPrompt Engineering