Positional Encoding — Sinusoidal and Learned Embeddings

🎯 Learning Objectives

• Explain why Transformers need positional encoding
• Derive and implement sinusoidal positional encodings
• Compare sinusoidal vs learned positional embeddings
• Understand how frequency encoding captures position
• Analyze the linearity property of sinusoidal encodings

📋 Prerequisites

• Transformer Architecture
• Basic trigonometry (sine, cosine functions)
• Word embeddings concept

1. 📖 Core Content

1.1 Intuition: Why Position Matters

1. "The dog bit the man"
2. "The man bit the dog" They contain the exact same words but mean completely opposite things. Position determines meaning. RNNs process words sequentially, so position is inherent. Transformers process all words in parallel — they need explicit position information. Without position information, the Transformer sees a bag of tokens — "The dog bit the man" is indistinguishable from "man the bit dog The."

1.2 Sinusoidal Positional Encoding

• Each position gets a unique encoding vector
• Different dimensions have different frequencies (geometric progression)
• Low dimensions (small i): high frequency → distinguish nearby positions
• High dimensions (large i): low frequency → encode absolute position range

1.3 Frequency Analysis

• i=0: f₀ = 1 (period = 2π)
• i=128: f₁₂₈ = 1/10000^{0.5} = 1/100 ≈ 0.01 (period ≈ 628)
• i=256: f₂₅₆ = 1/10000^{1} = 0.0001 (period ≈ 62832)

python
Run Inline
# runnable
import numpy as np
def compute_frequencies(d_model=512):
    """Compute the frequency at each dimension index"""
    frequencies = []
    for i in range(0, d_model, 2):
        f = 1.0 / (10000 ** (2 * i / d_model))
        frequencies.append(f)
    return frequencies
freqs = compute_frequencies(512)
print(f"Frequency at i=0 (dim 0-1): {freqs[0]:.6f}")
print(f"Frequency at i=128 (dim 256-257): {freqs[128]:.6f}")
print(f"Frequency at i=256 (dim 512-513): {freqs[256]:.6f}")

1.4 Worked Example: Complete Encoding for Short Sentence

• i=0 (even): sin(0/1) = sin(0) = 0
• i=0 (odd): cos(0/1) = cos(0) = 1
• i=1 (even): sin(0/10000^{2/6}) = sin(0/10000^{0.333}) = sin(0) = 0
• i=1 (odd): cos(0/10000^{0.333}) = cos(0) = 1
• i=2 (even): sin(0/10000^{4/6}) = sin(0/10000^{0.667}) = sin(0) = 0
• i=2 (odd): cos(0/10000^{0.667}) = cos(0) = 1 PE(0) = [0, 1, 0, 1, 0, 1] Position 1:
• i=0 (even): sin(1) = 0.841
• i=0 (odd): cos(1) = 0.540
• i=1 (even): sin(1/10000^{0.333}) = sin(1/21.54) = sin(0.0464) = 0.0464
• i=1 (odd): cos(0.0464) = 0.999
• i=2 (even): sin(1/10000^{0.667}) = sin(1/464.2) = sin(0.00215) = 0.00215
• i=2 (odd): cos(0.00215) = 1.000 PE(1) ≈ [0.841, 0.540, 0.046, 0.999, 0.002, 1.000] Position 2:
• i=0 (even): sin(2) = 0.909
• i=0 (odd): cos(2) = -0.416
• i=1 (even): sin(2/21.54) = sin(0.0929) = 0.0927
• i=1 (odd): cos(0.0929) = 0.996
• i=2 (even): sin(2/464.2) = sin(0.00431) = 0.00431
• i=2 (odd): cos(0.00431) = 1.000 PE(2) ≈ [0.909, -0.416, 0.093, 0.996, 0.004, 1.000] Position 3:
• i=0 (even): sin(3) = 0.141
• i=0 (odd): cos(3) = -0.990
• i=1 (even): sin(3/21.54) = sin(0.139) = 0.139
• i=1 (odd): cos(0.139) = 0.990
• i=2 (even): sin(3/464.2) = sin(0.00646) = 0.00646
• i=2 (odd): cos(0.00646) = 1.000 PE(3) ≈ [0.141, -0.990, 0.139, 0.990, 0.006, 1.000]

python
Run Inline
# runnable
import numpy as np
def sinusoidal_encoding(max_len, d_model):
    """Compute full positional encoding matrix"""
    PE = np.zeros((max_len, d_model))
    for pos in range(max_len):
        for i in range(0, d_model, 2):
            angle = pos / (10000 ** (2 * i / d_model))
            PE[pos, i] = np.sin(angle)
            if i + 1 < d_model:
                PE[pos, i + 1] = np.cos(angle)
    return PE
pe = sinusoidal_encoding(10, 6)
print("Positional Encoding Matrix (10 positions, 6 dims):")
print(np.round(pe, 4))

1.5 Sinusoidal vs Learned Positional Embeddings

1.6 The Key Property: Linear Relationship

1.7 Adding Positional Encoding to Embeddings

python
Run Inline
# runnable
import numpy as np
# Example: Adding PE to word embeddings
vocab_size = 1000
d_model = 512
max_len = 100
# Learned token embeddings (randomly initialized)
token_embeddings = np.random.randn(vocab_size, d_model) * 0.1
# Positional encoding
pe = sinusoidal_encoding(max_len, d_model)
# Example: sentence "I love transformers" (token IDs 42, 87, 256)
sentence = [42, 87, 256]
seq_len = len(sentence)
# Get token embeddings
token_vecs = token_embeddings[sentence]  # (3, d_model)
# Add positional encodings
position_vecs = pe[:seq_len]  # (3, d_model)
final_input = token_vecs + position_vecs
print("Token embedding shape:", token_vecs.shape)
print("Position encoding shape:", position_vecs.shape)
print("Final input shape:", final_input.shape)
print("Position encoding added to token: position {:.2f}% of signal magnitude".format(
    np.linalg.norm(position_vecs) / np.linalg.norm(final_input) * 100
))

1.8 Edge Cases and Limitations

1. Very long sequences: Sinusoidal encodings work for any length, but very high dimensions become nearly constant (frequency → 0), providing little position information.
2. Position 0 bias: Position 0 always has sin(0)=0 and cos(0)=1 for all dimensions, creating a distinct "first position" signature. This might cause a bias toward the first token.
3. Wavelength exceeds sequence length: For high dimensions, the wavelength ($2\pi \cdot 10000^{2i/d_{model}}$) can be much larger than the sequence length, making those dimensions act like a constant offset rather than position-dependent signal.

4. 📐 Key Formulas / Concepts

5. ⚠️ Common Pitfalls

Pitfall 1: Confusing positional encoding with positional embedding

• Positional encoding: Fixed sinusoidal functions (no parameters)
• Positional embedding: Learned vectors (trainable parameters) The original Transformer used encodings; BERT and GPT use learned embeddings.

Pitfall 2: Thinking higher dimensions encode more position information

Pitfall 3: Forgetting that PE is added, not concatenated

6. 📝 Practice Questions

Q1: For d_model=512, what is the frequency at i=256 (dimension 512)?

At i=256: 2i/d_model = 512/512 = 1. So: Frequency = 1/10000¹ = 0.0001 Wavelength = 2π/0.0001 ≈ 62,832 positions

This means the sine wave completes one full cycle every ~62,832 positions — essentially constant over typical sequence lengths. Q2: Why can't learned positional embeddings handle sequences longer than the training max length?

Learned embeddings are stored in a lookup table of shape (max_len, d_model). Position 1024 doesn't have an embedding if the model was only trained up to length 512. There's no way to "extrapolate" because the embeddings are discrete vectors learned per position. This is why some models use sinusoidal encodings or relative position encodings (like RoPE) that can extrapolate. Q3: Compute PE for position 4, dimension pairs 0-1, with d_model=6

i=0: pos/10000^{0} = 4/1 = 4 PE(4, 0) = sin(4) = -0.757 PE(4, 1) = cos(4) = -0.653

i=1: pos/10000^{2/6} = 4/10000^{0.333} = 4/21.54 = 0.186 PE(4, 2) = sin(0.186) = 0.185 PE(4, 3) = cos(0.186) = 0.983

i=2: pos/10000^{4/6} = 4/10000^{0.667} = 4/464.2 = 0.00862 PE(4, 4) = sin(0.00862) = 0.00862 PE(4, 5) = cos(0.00862) = 1.000

PE(4) = [-0.757, -0.653, 0.185, 0.983, 0.009, 1.000] Q4: If a Transformer with learned positional embeddings is trained on sequences up to length 512, what happens when it receives a sequence of length 600 during inference?

It will crash or produce incorrect results. The embedding lookup table only has 512 entries. Position 512 (0-indexed: the 513th token) has no embedding. Most implementations either:

1. Truncate the sequence to 512
2. Throw an out-of-bounds error
3. Use position 511's embedding as a fallback (which gives wrong position signal)

This is a key limitation addressed by relative position encodings (RoPE, ALiBi). Q5: What is the approximate wavelength for dimension i when d_model=512?

Wavelength = 2π × 10000^{2i/512}

For i=0: 2π × 1 ≈ 6.28 For i=64: 2π × 10000^{128/512} = 2π × 10000^{0.25} = 2π × 10 = 62.8 For i=128: 2π × 10000^{256/512} = 2π × 10000^{0.5} = 2π × 100 = 628 For i=192: 2π × 10000^{384/512} = 2π × 10000^{0.75} = 2π × 1000 = 6283 For i=256: 2π × 10000 = 62832 Q6: What would happen if we used only sine (no cosine) in positional encoding?

Without cosine, the encoding would use only sine functions: PE(pos, i) = sin(pos/10000^{i/d_model}) for ALL dimensions

This loses the linear transformation property. A sine-only encoding at position pos+k cannot be expressed as a linear function of the encoding at position pos. The model would struggle to learn relative position patterns.

The sine/cosine pair creates a "phase-amplitude" representation that enables the rotation property. Q7: How does the choice of base 10000 affect positional encoding?

The base (10000) controls the frequency range:

• Larger base (e.g., 100000): Slower frequency decay → more dimensions have high frequency → finer position discrimination over longer ranges
• Smaller base (e.g., 100): Faster frequency decay → fewer high-frequency dimensions → coarser position encoding

The original choice 10000 was empirically determined. Some models use different bases (e.g., 10000 in original Transformer, smaller values in some variants). Q8: In the expression PE(pos+k) = T(k)·PE(pos), what is T(k) for a single dimension pair (2i, 2i+1)?

For a single frequency ω = 1/10000^{2i/d_model}:

This is a rotation matrix by angle kω. So adding offset k corresponds to rotating the (sin, cos) pair by angle kω.

Q9: A Transformer with sinusoidal encoding processes position 0 and position 1000. How does the encoding differ at high dimensions vs low dimensions?

At low dimensions (small i): The frequency is high. Position 0 and position 1000 will have very different values (many cycles of difference). These dimensions clearly distinguish the two positions.

At high dimensions (large i): The frequency is extremely low. The wavelength might be much larger than 1000, so position 0 and position 1000 have nearly the same value. These dimensions provide little position distinction but encode the overall position range. Q10: If we multiply positional encoding by a learned weight w before adding to token embeddings, is this still "sinusoidal encoding"?

No, this becomes a hybrid approach. The encoding is based on sinusoidal functions (structure) but with learnable scaling (amplitude). This retains the relative position properties while allowing the model to adjust how much position information to use.

$\text{input} = \text{embedding} + \text{PE} \cdot w$

Where w is either a scalar or a per-dimension learned weight. This is used in some Transformer variants.

7. 🔗 Cross-References

• Next: Encoder & Decoder Layers (Week 2)
• Advanced: RoPE and ALiBi (Week 11)
• Video: BSDA5004 Week 1-2 transcripts Join Discord PreviousMulti-Head AttentionNextDecoder & Cross-Attention