Many NLP tasks require mapping an input sequence to an output sequence of different length:

The attention mechanism allows the decoder to focus on relevant parts of the input at each step: \begin{cases} h_t^T W \bar{h}s & \text{Luong (multiplicative)} \\ v_a^T \tanh(W[h_t; \bar{h}s]) & \text{Bahdanau (additive)} \end{cases}$$ \alpha{ts} = \frac{\exp(\text{score}(h_t, \bar{h}s))}{\sum{s'} \exp(\text{score}(h_t, \bar{h}{s'}))} c_t = \sum_s \alpha_{ts} \bar{h}_s ```python # runnable import numpy as np class BahdanauAttention: """Additive attention (Bahdanau et al., 2015)""" def init(self, hidden_dim): self.W = np.random.randn(hidden_dim, hidden_dim) * 0.01 self.V = np.random.randn(hidden_dim) * 0.01 def score(self, h_t, h_s): """Compute attention score between decoder state h_t and encoder state h_s""" # h_t: (hidden_dim,), h_s: (hidden_dim,) score = self.V @ np.tanh(self.W @ (h_t + h_s)) return score def forward(self, h_t, encoder_states): """Compute context vector and attention weights""" n_steps = len(encoder_states) scores = np.array([self.score(h_t, h_s) for h_s in encoder_states]) # Softmax exp_scores = np.exp(scores - np.max(scores)) alpha = exp_scores / np.sum(exp_scores) # Context vector context = np.sum([alpha[i] * encoder_states[i] for i in range(n_steps)], axis=0) return context, alpha # Example: translating "I love cats" (3 source tokens) hidden_dim = 256 attn = BahdanauAttention(hidden_dim) # Simulated encoder states encoder_states = np.random.randn(3, hidden_dim) # Simulated decoder state at step 1 h_t = np.random.randn(hidden_dim) context, weights = attn.forward(h_t, encoder_states) print(f"Context vector shape: {context.shape}") print(f"Attention weights: {np.round(weights, 3)}") print(f"Weights sum to: {np.sum(weights):.3f}") ``` ### 1.4 Teacher Forcing During training, instead of feeding the model's prediction as input to the next step, we feed the ground truth token. This is called teacher forcing. ### 1.5 Beam Search At inference, instead of greedy decoding (pick argmax at each step), maintain k hypotheses: ```python def beam_search(model, src, k=3, max_len=50): """Simplified beam search for seq2seq""" # Encode source encoder_states = model.encode(src) # Initialize beams: [(sequence, score)] beams = [([BOS], 0.0)] completed = [] for step in range(max_len): candidates = [] for seq, score in beams: if seq[-1] == EOS: completed.append((seq, score)) continue # Get next token probs dec_state = model.decode(seq[-1], encoder_states) probs = model.output_probs(dec_state) # Top-k tokens top_k = np.argsort(probs)[-k:] for token in top_k: new_seq = seq + [token] new_score = score + np.log(probs[token]) candidates.append((new_seq, new_score)) # Keep top-k candidates.sort(key=lambda x: x[1], reverse=True) beams = candidates[:k] completed.extend(beams) completed.sort(key=lambda x: x[1], reverse=True) return completed[0][0] ``` --- ## 📝 Practice Questions > Q1 > > <strong>Q1</strong>: In Bahdanau attention, why is the alignment score computed as v_a^T tanh(W[h_t; h_s]) rather than a simpler dot product? > > The additive formulation (Bahdanau) is more flexible than dot product: > 1. Different dimensions: Decoder state and encoder state may have different dimensions (not required) > 2. Non-linearity: tanh adds non-linearity, allowing more complex alignments > 3. Learned parameters: v_a and W are learned, allowing the model to discover task-specific alignment patterns > > Luong's multiplicative attention (h_t^T W h_s) is simpler and often faster but less expressive. In practice, both work well; Luong is more common in newer implementations. > Q2 > > <strong>Q2 > > <strong>Q2</strong>: For a source sentence of 10 words and target of 8 words, what's the shape of the attention matrix? > > The attention matrix has shape (target_len, source_len) = (8, 10). > > Each row corresponds to a target token and its attention distribution over source tokens. Row sums to 1 (softmax over source positions). > > Column sums can be anything — some source words may receive attention from many target words (important words), while others may receive little attention. > Q3 > > <strong>Q3 > > <strong>Q3 > > <strong>Q3</strong>: Why does beam search with k=5 use more memory than greedy decoding? > > Greedy decoding: Maintains 1 hypothesis, 1 forward pass per step > Beam search (k=5): Maintains 5 hypotheses, k × vocab_size candidate scores per step > > Memory differs because: > - Beam search stores 5 complete sequences simultaneously > - At each step, it evaluates 5 × |V| candidates (vs 1 × |V| for greedy) > - Backtracking requires storing the full history of all beams > - Model must process 5× the tokens (softmax over 5 decoder states) > > This 5× memory and compute cost is the trade-off for better quality. > Q4 > > <strong>Q4 > > <strong>Q4 > > <strong>Q4 > > <strong>Q4</strong>: A seq2seq model trained with teacher forcing works well but produces repetitive text during inference. Why? > > This is exposure bias: during training (teacher forcing), the model always sees correct previous tokens. During inference, it sees its own predictions which may be slightly off, leading to error accumulation. > > Repetition occurs because: > 1. Once a word is generated, the decoder sees it as input, making the same word more likely again > 2. The training never included recovery from its own mistakes > > Fixes: > - Scheduled sampling: Gradually mix teacher forcing with model predictions during training > - Beam search / diversity penalties: Prevent repetition > - Training with noise: Add noise to training inputs to make model robust > - Reinforcement learning: Treat generation as a sequential decision problem </details> --- ## 🔗 Cross-References - Next: [Sentiment Analysis](../week09/09-sentiment-analysis.md) - Previous: [RNNs & LSTMs](../week07/07-rnn-lstm-nlp.md) - Video: BSDA5005 Week 7-8 transcripts Join Discord PreviousText ClassificationNextEvaluation Metrics