⚡ Activation Functions

792 words

4 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# ⚡ Activation Functions ## 1. 🎯 Learning Objectives By the end of this topic, you will be able to: - Compute the output and derivative of sigmoid, tanh, ReLU, and their variants - Explain the vanishing gradient problem and how ReLU mitigates it - Choose the appropriate activation function for a given layer/network...

⚡ Activation Functions

1. 🎯 Learning Objectives

By the end of this topic, you will be able to:

Compute the output and derivative of sigmoid, tanh, ReLU, and their variants
Explain the vanishing gradient problem and how ReLU mitigates it
Choose the appropriate activation function for a given layer/network
Implement activation functions in Python with NumPy
Analyze the computational and numerical properties of each function

2. 📋 Prerequisites

Prerequisite	Course	Why It Matters
Derivatives	BSMA1002	Need derivatives for backpropagation
Perceptron	W1 T1	Understanding why activation matters
Chain rule	BSMA1002	For gradient computation

3. 📖 Core Content

3.1 Intuition: Why Do We Need Activation Functions?

Without activation functions, a neural network would just be a linear transformation of inputs — no matter how many layers, the entire network would collapse into a single linear transformation (because composition of linear functions is linear). Activation functions introduce nonlinearity, allowing networks to learn complex patterns. Each activation function has:

A forward pass: $a = f(z)$ — transforms the weighted sum $z$ into the activation $a$
A backward pass: $\frac{\partial a}{\partial z} = f'(z)$ — derivative used for gradient computation

3.2 Sigmoid (Logistic)

\sigma(z) = \frac{1}{1 + e^{-z}}

\sigma'(z) = \sigma(z)(1 - \sigma(z))

Range: (0, 1) Use: Output layer for binary classification, hidden layers (historically) Derivation of derivative:

\sigma'(z) = \frac{e^{-z}}{(1+e^{-z})^2} = \frac{1}{1+e^{-z}} \cdot \frac{e^{-z}}{1+e^{-z}} = \sigma(z)(1-\sigma(z))

Problems:

Vanishing gradient: For $|z| > 5$ , $\sigma(z) \approx 0$ or $1$ , so $\sigma'(z) \approx 0$ . Gradients "vanish" — no learning.
Not zero-centered: Outputs are always positive, causing zigzagging in optimization.
Expensive: Exponential computation.

3.3 Tanh (Hyperbolic Tangent)

\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}

\tanh'(z) = 1 - \tanh^2(z)

Range: (-1, 1) Use: Hidden layers (zero-centered advantage over sigmoid) Properties:

Zero-centered: outputs in [-1, 1], which helps optimization
Still suffers from vanishing gradients for large $|z|$

3.4 ReLU (Rectified Linear Unit)

\text{ReLU}(z) = \max(0, z)

\text{ReLU}'(z) = \begin{cases} 1 & z > 0 \\ 0 & z \leq 0 \end{cases}

Range: [0, ∞) Use: Default for hidden layers in most modern networks Advantages:

No vanishing gradient for $z > 0$ (gradient = 1)
Sparse activation (many neurons are 0)
Computationally cheap (just max(0, z))
Empirically accelerates convergence Disadvantages:
Dying ReLU: If a neuron's weights push all inputs to negative, gradient is 0 and the neuron never recovers
Not zero-centered

3.5 Leaky ReLU and Variants

Leaky ReLU:

\text{LReLU}(z) = \max(\alpha z, z)

Typically

\alpha = 0.01

ELU (Exponential Linear Unit):

\text{ELU}(z) = \begin{cases} z & z > 0 \\ \alpha(e^z - 1) & z \leq 0 \end{cases}

Swish (Self-Gated):

\text{Swish}(z) = z \cdot \sigma(z)

3.6 Comparison Table

Function	Range	Derivative	Vanishing?	Zero-Centered?	Computational Cost
Sigmoid	(0, 1)	σ(1-σ)	Severe	No	Medium
Tanh	(-1, 1)	1-tanh²	Moderate	Yes	Medium
ReLU	[0, ∞)	1 or 0	No (z>0)	No	Low
Leaky ReLU	(-∞, ∞)	1 or α	No	No	Low
ELU	(-α, ∞)	1 or αe^z	No	Near-zero	Medium
Swish	(-0.28, ∞)	σ + z·σ(1-σ)	No	No	High

4. 📐 Key Formulas

Function	Formula	Derivative
Sigmoid	$\sigma(z) = 1/(1+e^{-z})$	$\sigma(1-\sigma)$
Tanh	$\tanh(z) = (e^z-e^{-z})/(e^z+e^{-z})$	$1-\tanh^2$
ReLU	$\max(0, z)$	$1_{z>0}$
Leaky ReLU	$\max(\alpha z, z)$	$1_{z>0} + \alpha \cdot 1_{z\leq 0}$

5. ⚠️ Common Pitfalls

Pitfall 1: Using Sigmoid in Hidden Layers

The mistake: Using sigmoid activation in deep hidden layers. Why: Vanishing gradients prevent learning in early layers. Correct approach: Use ReLU (or Leaky ReLU) in hidden layers, sigmoid only for binary classification output.

Pitfall 2: Dying ReLU

The mistake: Using ReLU with too high learning rate. Why: Large updates can push weights such that a neuron always outputs negative, gradient becomes 0, neuron "dies." Solution: Use Leaky ReLU, or smaller learning rates, or batch normalization.

Pitfall 3: Forgetting the Derivative in Backprop

The mistake: Computing the gradient without multiplying by the activation derivative. Correct approach:

\frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \cdot f'(z)

6. 📝 Practice Questions

Q1: Compute sigmoid(0), sigmoid(1), sigmoid(-1). What are the derivatives?

Answer: σ(0) = 0.5, σ(1) = 1/(1+e⁻¹) ≈ 0.731, σ(-1) = 1/(1+e¹) ≈ 0.269. Derivatives: σ'(0) = 0.5×0.5 = 0.25, σ'(1) = 0.731×0.269 ≈ 0.197, σ'(-1) = 0.269×0.731 ≈ 0.197. Q2: For z = [-5, 0, 5], compute ReLU outputs and derivatives.

Answer: ReLU(-5) = 0, derivative = 0. ReLU(0) = 0 (edge case, typically derivative = 0 or 0.5 in implementations). ReLU(5) = 5, derivative = 1. Q3: Why is tanh preferred over sigmoid for hidden layers despite both being S-shaped?

Answer: Tanh is zero-centered (range -1 to 1), which means gradients can be both positive and negative. This helps with optimization because the gradients don't all go in the same direction. Sigmoid outputs are always positive, which can cause zigzagging gradients. Join Discord PreviousDL History & PerceptronNextPerceptron to MLP

What this note is really saying

⚡ Activation Functions

1. 🎯 Learning Objectives

2. 📋 Prerequisites

3. 📖 Core Content

3.1 Intuition: Why Do We Need Activation Functions?

3.2 Sigmoid (Logistic)

3.3 Tanh (Hyperbolic Tangent)

3.4 ReLU (Rectified Linear Unit)

3.5 Leaky ReLU and Variants

3.6 Comparison Table

4. 📐 Key Formulas

5. ⚠️ Common Pitfalls

Pitfall 1: Using Sigmoid in Hidden Layers

Pitfall 2: Dying ReLU

Pitfall 3: Forgetting the Derivative in Backprop

6. 📝 Practice Questions

Intelligence Hub