🔬 LSTM & GRU Detailed Analysis
434 words
2 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# 🔬 LSTM & GRU Detailed Analysis ## 1. 🎯 Learning Objectives - Trace LSTM forward pass with numerical example - Explain gradient flow through cell state - Compare LSTM vs GRU: parameter count, performance - Implement LSTM from scratch ## 2.
🔬 LSTM & GRU Detailed Analysis
1. 🎯 Learning Objectives
- Trace LSTM forward pass with numerical example
- Explain gradient flow through cell state
- Compare LSTM vs GRU: parameter count, performance
- Implement LSTM from scratch
2. 📖 Core Content
3.1 LSTM Forward Pass Numerical Example
Let's trace a single LSTM cell with input x_t = [0.5, -0.3] and previous hidden h_{t-1} = [0.2, 0.1], cell state C_{t-1} = [0.0, 0.0].
Step 1: Forget gate f_t = σ(W_f · [h_{t-1}, x_t] + b_f) Assuming W_f · [h_{t-1}, x_t] = [0.5, -0.2], b_f = [0.1, 0.0] f_t = σ([0.6, -0.2]) = [0.646, 0.450]
Step 2: Input gate i_t = σ(W_i · [h_{t-1}, x_t] + b_i) = σ([0.3, 0.4]) = [0.574, 0.599]
Step 3: Candidate cell state C̃_t = tanh(W_C · [h_{t-1}, x_t] + b_C) = tanh([0.8, -0.5]) = [0.664, -0.462]
Step 4: Update cell state C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t C_t = [0.646×0 + 0.574×0.664, 0.450×0 + 0.599×(-0.462)] C_t = [0.381, -0.277]
Step 5: Output gate o_t = σ(W_o · [h_{t-1}, x_t] + b_o) = σ([0.2, 0.6]) = [0.550, 0.646]
Step 6: Hidden state h_t = o_t ⊙ tanh(C_t) = [0.550×tanh(0.381), 0.646×tanh(-0.277)] h_t = [0.550×0.364, 0.646×(-0.270)] h_t = [0.200, -0.174]
3.2 Gradient Flow
The cell state recurrence: C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t
The gradient of C_t w.r.t. C_{t-1} is f_t (not a matrix multiplication). If f_t ≈ 1, gradient flows back almost unchanged — this is why LSTM avoids vanishing gradients.
3.3 LSTM vs GRU Comparison
| Aspect | LSTM | GRU |
|---|---|---|
| Gates | 4 (forget, input, candidate, output) | 2 (reset, update) |
| Cell state | Yes (C_t) | No (only hidden h_t) |
| Parameters | More (4× W matrices) | Fewer (3× W matrices) |
| Performance | Comparable | Comparable (often slightly worse) |
| Training speed | Slower (more params) | Faster (fewer params) |
| When to use | Complex sequence tasks | Simpler tasks, less data |
3.4 PyTorch Implementation
pythonclass LSTMCell(nn.Module): def __init__(self, input_size, hidden_size): super().__init__() self.W_f = nn.Linear(input_size + hidden_size, hidden_size) self.W_i = nn.Linear(input_size + hidden_size, hidden_size) self.W_C = nn.Linear(input_size + hidden_size, hidden_size) self.W_o = nn.Linear(input_size + hidden_size, hidden_size) def forward(self, x, h, C): combined = torch.cat([h, x], dim=1) f = torch.sigmoid(self.W_f(combined)) i = torch.sigmoid(self.W_i(combined)) C_tilde = torch.tanh(self.W_C(combined)) o = torch.sigmoid(self.W_o(combined)) C_next = f * C + i * C_tilde h_next = o * torch.tanh(C_next) return h_next, C_next
4. 📝 Practice Questions
Q1: If the forget gate outputs are all close to 1, what happens to gradient flow?Answer: The gradient flows almost unchanged through the cell state. This is how LSTM captures long-term dependencies. The forget gate learns which past information to keep, and when f_t ≈ 1, that information persists. Join Discord PreviousRNN VariantsNextTransformers & Attention