🔬 LSTM & GRU Detailed Analysis

434 words

2 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# 🔬 LSTM & GRU Detailed Analysis ## 1. 🎯 Learning Objectives - Trace LSTM forward pass with numerical example - Explain gradient flow through cell state - Compare LSTM vs GRU: parameter count, performance - Implement LSTM from scratch ## 2.

🔬 LSTM & GRU Detailed Analysis

1. 🎯 Learning Objectives

Trace LSTM forward pass with numerical example
Explain gradient flow through cell state
Compare LSTM vs GRU: parameter count, performance
Implement LSTM from scratch

2. 📖 Core Content

3.1 LSTM Forward Pass Numerical Example

Let's trace a single LSTM cell with input x_t = [0.5, -0.3] and previous hidden h_{t-1} = [0.2, 0.1], cell state C_{t-1} = [0.0, 0.0]. Step 1: Forget gate f_t = σ(W_f · [h_{t-1}, x_t] + b_f) Assuming W_f · [h_{t-1}, x_t] = [0.5, -0.2], b_f = [0.1, 0.0] f_t = σ([0.6, -0.2]) = [0.646, 0.450] Step 2: Input gate i_t = σ(W_i · [h_{t-1}, x_t] + b_i) = σ([0.3, 0.4]) = [0.574, 0.599] Step 3: Candidate cell state C̃_t = tanh(W_C · [h_{t-1}, x_t] + b_C) = tanh([0.8, -0.5]) = [0.664, -0.462] Step 4: Update cell state C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t C_t = [0.646×0 + 0.574×0.664, 0.450×0 + 0.599×(-0.462)] C_t = [0.381, -0.277] Step 5: Output gate o_t = σ(W_o · [h_{t-1}, x_t] + b_o) = σ([0.2, 0.6]) = [0.550, 0.646] Step 6: Hidden state h_t = o_t ⊙ tanh(C_t) = [0.550×tanh(0.381), 0.646×tanh(-0.277)] h_t = [0.550×0.364, 0.646×(-0.270)] h_t = [0.200, -0.174]

3.2 Gradient Flow

The cell state recurrence: C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t The gradient of C_t w.r.t. C_{t-1} is f_t (not a matrix multiplication). If f_t ≈ 1, gradient flows back almost unchanged — this is why LSTM avoids vanishing gradients.

3.3 LSTM vs GRU Comparison

Aspect	LSTM	GRU
Gates	4 (forget, input, candidate, output)	2 (reset, update)
Cell state	Yes (C_t)	No (only hidden h_t)
Parameters	More (4× W matrices)	Fewer (3× W matrices)
Performance	Comparable	Comparable (often slightly worse)
Training speed	Slower (more params)	Faster (fewer params)
When to use	Complex sequence tasks	Simpler tasks, less data

3.4 PyTorch Implementation


python
class LSTMCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.W_f = nn.Linear(input_size + hidden_size, hidden_size)
        self.W_i = nn.Linear(input_size + hidden_size, hidden_size)
        self.W_C = nn.Linear(input_size + hidden_size, hidden_size)
        self.W_o = nn.Linear(input_size + hidden_size, hidden_size)
    def forward(self, x, h, C):
        combined = torch.cat([h, x], dim=1)
        f = torch.sigmoid(self.W_f(combined))
        i = torch.sigmoid(self.W_i(combined))
        C_tilde = torch.tanh(self.W_C(combined))
        o = torch.sigmoid(self.W_o(combined))
        C_next = f * C + i * C_tilde
        h_next = o * torch.tanh(C_next)
        return h_next, C_next

4. 📝 Practice Questions

Q1: If the forget gate outputs are all close to 1, what happens to gradient flow?

Answer: The gradient flows almost unchanged through the cell state. This is how LSTM captures long-term dependencies. The forget gate learns which past information to keep, and when f_t ≈ 1, that information persists. Join Discord PreviousRNN VariantsNextTransformers & Attention