๐ Seq2Seq & Attention Mechanisms
127 words
1 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# ๐ Seq2Seq & Attention Mechanisms ## 1. ๐ฏ Learning Objectives - Explain encoder-decoder for sequence tasks - Compute attention weights for alignment - Describe attention variants: additive, multiplicative, self-attention ## 2.
๐ Seq2Seq & Attention Mechanisms
1. ๐ฏ Learning Objectives
- Explain encoder-decoder for sequence tasks
- Compute attention weights for alignment
- Describe attention variants: additive, multiplicative, self-attention
2. ๐ Core Content
3.1 Encoder-Decoder
Encoder RNN: reads input sequence, produces context vector c. Decoder RNN: generates output sequence from c. Problem: Information bottleneck โ c must capture entire input sequence.
3.2 Attention Mechanism
Compute attention weights: ฮฑโแตข = softmax(score(hโ, sแตข)) Context vector: cโ = ฮฃแตข ฮฑโแตข hแตข Score functions:
- Additive (Bahdanau): vแต tanh(Wโhโ + Wโsแตข)
- Dot product (Luong): hโแต sแตข
- Scaled dot: hโแต sแตข / โd
3.3 Benefits
- No information bottleneck
- Alignment learned end-to-end
- Interpretability (visualize attention weights) Join Discord PreviousTransformers & AttentionNextTransformer Deep Dive