🏗️ Transformers & Attention
207 words
1 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# 🏗️ Transformers & Attention ## 1. 🎯 Learning Objectives - Compute scaled dot-product attention - Explain multi-head attention - Understand positional encoding - Describe the Transformer encoder-decoder architecture ## 2.
🏗️ Transformers & Attention
1. 🎯 Learning Objectives
- Compute scaled dot-product attention
- Explain multi-head attention
- Understand positional encoding
- Describe the Transformer encoder-decoder architecture
2. 📖 Core Content
3.1 Scaled Dot-Product Attention
- Q (Query): What am I looking for?
- K (Key): What do I contain?
- V (Value): My actual content
- : Dimension of keys (scaling factor prevents softmax saturation)
3.2 Multi-Head Attention
Instead of one attention function, use h heads in parallel:
Each head can focus on different relationships (syntax, semantics, position).
3.3 Positional Encoding
Since self-attention is permutation-invariant (no inherent order), we add positional encodings:
3.4 Transformer Architecture
(Diagram)
Each sublayer has residual connection and layer normalization: .
4. 📝 Practice Questions
Q1: Why is scaling by √d_k needed in attention?Answer: Without scaling, for large d_k, the dot products grow large, pushing softmax into regions with extremely small gradients (saturating). Scaling ensures more moderate values and better gradient flow. Join Discord PreviousLSTM & GRU DetailsNextSeq2Seq & Attention