Neural Sync Active

🏗️ Transformer Architecture Deep Dive

Registry Synced

🏗️ Transformer Architecture Deep Dive

285 words

1 min read

Reading compass

Now · 1. 🎯 Learning Objectives

🏗️ Transformer Architecture Deep Dive

1. 🎯 Learning Objectives

Compute attention scores step-by-step
Implement multi-head attention
Explain masking in decoder
Describe positional encoding formula

2. 📖 Core Content

3.1 Scaled Dot-Product Attention: Numerical Example

Given Q, K, V matrices (2 words, d_k=3): Q = [[1, 0, 1], [0, 1, 0]] K = [[0, 1, 0], [1, 0, 1]] V = [[1, 0], [0, 1]] (2 words, d_v=2) Step 1: Q × K^T = [[1×0+0×1+1×0, 1×1+0×0+1×1], [0×0+1×1+0×0, 0×1+1×0+0×1]] = [[0, 2], [1, 0]] Step 2: Scale by √d_k = √3 ≈ 1.732 = [[0, 1.155], [0.577, 0]] Step 3: Softmax per row Row 0: softmax([0, 1.155]) = [0.240, 0.760] Row 1: softmax([0.577, 0]) = [0.640, 0.360] Step 4: Attention × V = [[0.240×1+0.760×0, 0.240×0+0.760×1], [0.640×1+0.360×0, 0.640×0+0.360×1]] = [[0.240, 0.760], [0.640, 0.360]]

3.2 Multi-Head Attention

Instead of one attention, use h heads (h=8 in original transformer):

Each head has its own Q, K, V projections
Concatenate outputs, project back to d_model Benefits:

Different heads learn different relationships (syntax, semantics)
Increased representational capacity
More stable training

3.3 Positional Encoding

Position 0: PE(0,0)=sin(0)=0, PE(0,1)=cos(0)=1 Position 1: PE(1,0)=sin(1/10000^{0/512})=sin(1)=0.84, PE(1,1)=cos(1)=0.54 Each position gets a unique encoding that allows the model to use order information.

3.4 Masking

In decoder: Mask future positions (set attention scores to -inf before softmax) so position i can only attend to positions ≤ i. This preserves autoregressive property.

4. 📝 Practice Questions

Q1: For d_k=2, Q=[1,0], K=[[1,1],[0,1]]. Compute attention weights.

Answer: QK^T = [1×1+0×1, 1×0+0×1] = [1, 0]. Scale: [0.707, 0]. Softmax: [0.668, 0.332]. These are the weights for V rows. Join Discord PreviousSeq2Seq & AttentionNextOptimizers Comparison