Neural Sync Active
🏗️ Transformer Architecture Deep Dive
Registry Synced
🏗️ Transformer Architecture Deep Dive
285 words
1 min read
Reading compass
Now · 1. 🎯 Learning Objectives
🏗️ Transformer Architecture Deep Dive
1. 🎯 Learning Objectives
- Compute attention scores step-by-step
- Implement multi-head attention
- Explain masking in decoder
- Describe positional encoding formula
2. 📖 Core Content
3.1 Scaled Dot-Product Attention: Numerical Example
Given Q, K, V matrices (2 words, d_k=3):
Q = [[1, 0, 1], [0, 1, 0]] K = [[0, 1, 0], [1, 0, 1]]
V = [[1, 0], [0, 1]] (2 words, d_v=2)
Step 1: Q × K^T = [[1×0+0×1+1×0, 1×1+0×0+1×1], [0×0+1×1+0×0, 0×1+1×0+0×1]] = [[0, 2], [1, 0]]
Step 2: Scale by √d_k = √3 ≈ 1.732 = [[0, 1.155], [0.577, 0]]
Step 3: Softmax per row Row 0: softmax([0, 1.155]) = [0.240, 0.760] Row 1: softmax([0.577, 0]) = [0.640, 0.360]
Step 4: Attention × V = [[0.240×1+0.760×0, 0.240×0+0.760×1], [0.640×1+0.360×0, 0.640×0+0.360×1]] = [[0.240, 0.760], [0.640, 0.360]]
3.2 Multi-Head Attention
Instead of one attention, use h heads (h=8 in original transformer):
- Each head has its own Q, K, V projections
- Concatenate outputs, project back to d_model Benefits:
- Different heads learn different relationships (syntax, semantics)
- Increased representational capacity
- More stable training
3.3 Positional Encoding
Position 0: PE(0,0)=sin(0)=0, PE(0,1)=cos(0)=1 Position 1: PE(1,0)=sin(1/10000^{0/512})=sin(1)=0.84, PE(1,1)=cos(1)=0.54
Each position gets a unique encoding that allows the model to use order information.
3.4 Masking
In decoder: Mask future positions (set attention scores to -inf before softmax) so position i can only attend to positions ≤ i. This preserves autoregressive property.
4. 📝 Practice Questions
Q1: For d_k=2, Q=[1,0], K=[[1,1],[0,1]]. Compute attention weights.Answer: QK^T = [1×1+0×1, 1×0+0×1] = [1, 0]. Scale: [0.707, 0]. Softmax: [0.668, 0.332]. These are the weights for V rows. Join Discord PreviousSeq2Seq & AttentionNextOptimizers Comparison