🔄 Recurrent Neural Networks & LSTMs

311 words

2 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# 🔄 Recurrent Neural Networks & LSTMs ## 1. 🎯 Learning Objectives - Explain the RNN hidden state recurrence - Trace gradient flow and identify vanishing gradient in RNNs - Describe LSTM gates: forget, input, output, cell state - Compare LSTM, GRU, and vanilla RNN ## 2.

🔄 Recurrent Neural Networks & LSTMs

1. 🎯 Learning Objectives

Explain the RNN hidden state recurrence
Trace gradient flow and identify vanishing gradient in RNNs
Describe LSTM gates: forget, input, output, cell state
Compare LSTM, GRU, and vanilla RNN

2. 📖 Core Content

3.1 RNN Formulation

h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h)

y_t = W_{hy}h_t + b_y

The same weight matrices are used at every time step. This is parameter sharing.

3.2 Vanishing Gradient in RNNs

During BPTT (Backpropagation Through Time), gradients are multiplied by

W_{hh}

at each time step. If

||W_{hh}|| < 1

, gradients vanish exponentially. If

||W_{hh}|| > 1

, gradients explode. Result: Vanilla RNNs struggle to learn long-range dependencies.

3.3 LSTM

Cell state

C_t

flows through time with minimal linear operations, preserving gradients. Gates:

Gate	Formula	Purpose
Forget	$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$	What to discard from cell state
Input	$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$	What new info to store
Candidate	$\tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C)$	New candidate values
Output	$o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$	What to output

Cell update:

C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t

h_t = o_t \odot \tanh(C_t)

The additive cell update (instead of multiplicative in vanilla RNN) helps gradients flow.

3.4 GRU (Gated Recurrent Unit)

Simplified LSTM with 2 gates (reset and update), no separate cell state.

4. 📝 Practice Questions

Q1: Why does the LSTM's cell state help with vanishing gradients?

Answer: The cell state C_t = f_t ⊙ C_{t-1} + i_t ⊙ Ĉ_t involves ADDITION (not multiplication) across time steps. During backpropagation, the gradient flows through the forget gate additive connection, avoiding the repeated matrix multiplication that causes vanishing gradients in vanilla RNNs. Join Discord PreviousBatch Norm DetailsNextAutoencoders & GANs

What this note is really saying

🔄 Recurrent Neural Networks & LSTMs

1. 🎯 Learning Objectives

2. 📖 Core Content

3.1 RNN Formulation

3.2 Vanishing Gradient in RNNs

3.3 LSTM

3.4 GRU (Gated Recurrent Unit)

4. 📝 Practice Questions

Intelligence Hub