Deep Learning Foundations
486 words
2 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# Deep Learning Foundations ## 🎯 Learning Objectives - Understand deep learning architectures (CNNs, RNNs, Transformers) - Apply modern optimization (Adam, learning rate schedules) - Use regularization techniques (dropout, batch norm, data augmentation) - Apply transfer learning with pre-trained models ## 📖 Core C...
Deep Learning Foundations
🎯 Learning Objectives
- Understand deep learning architectures (CNNs, RNNs, Transformers)
- Apply modern optimization (Adam, learning rate schedules)
- Use regularization techniques (dropout, batch norm, data augmentation)
- Apply transfer learning with pre-trained models
📖 Core Content
10.1 Why Deep Learning?
Deep learning uses many layers of neural networks to learn hierarchical representations. Early layers learn simple patterns (edges), middle layers learn parts (eyes, wheels), and later layers learn complex concepts (faces, cars).
10.2 Key Architectures
| Architecture | Key Innovation | Best For |
|---|---|---|
| CNN | Convolution + Pooling | Images, spatial data |
| RNN/LSTM | Recurrent connections | Sequences, time series |
| Transformer | Self-attention mechanism | NLP, sequences (parallelizable) |
| Autoencoder | Encoder-decoder | Unsupervised, compression |
| GAN | Generator + Discriminator | Synthetic data generation |
10.3 Modern Optimizers
| Optimizer | Update Rule | Key Feature |
|---|---|---|
| SGD | Simple, needs LR tuning | |
| Momentum | Accelerates, smooths | |
| AdaGrad | Adaptive per-parameter LR | Good for sparse features |
| RMSProp | Running average of squared gradients | Fixes AdaGrad's decay |
| Adam | Momentum + RMSProp | Default choice |
10.4 Modern Regularization
| Technique | How It Works | When to Use |
|---|---|---|
| Dropout | Randomly drop neurons during training | Large networks, prevent co-adaptation |
| Batch Norm | Normalize layer outputs | Deep networks, faster convergence |
| Data Augmentation | Create modified training examples | Limited data (images, text) |
| Label Smoothing | Soften target labels | Overconfident models |
| Early Stopping | Stop when validation loss plateaus | Always use |
10.5 Transfer Learning
python# runnable # Concept code — requires tensorflow/pytorch # from tensorflow.keras.applications import ResNet50 # from tensorflow.keras.layers import Dense, GlobalAveragePooling2D # from tensorflow.keras.models import Model # # # Load pre-trained model (ImageNet weights) # base_model = ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3)) # base_model.trainable = False # Freeze base layers # # # Add custom classification head # x = base_model.output # x = GlobalAveragePooling2D()(x) # x = Dense(128, activation='relu')(x) # predictions = Dense(10, activation='softmax')(x) # # model = Model(inputs=base_model.input, outputs=predictions) # model.compile(optimizer='adam', loss='categorical_crossentropy')
📝 Practice Questions
Q1: Why is Adam the default optimizer?Adam combines Momentum (accelerates, smooths gradients) and RMSProp (adaptive per-parameter learning rates). It works well with: (1) default hyperparameters (LR=0.001), (2) noisy gradients, (3) sparse gradients, (4) non-stationary objectives. It requires less tuning than SGD. Q2: What problem does batch normalization solve?Internal covariate shift: as layers update, the distribution of inputs to each layer changes. This forces later layers to continuously adapt to shifting distributions, slowing training. Batch norm normalizes each layer's outputs (mean=0, std=1), then learns scale and shift parameters. This: (1) allows higher learning rates, (2) reduces sensitivity to initialization, (3) provides slight regularization. Q3: How does transfer learning work with pre-trained models?
- Take a model trained on a large dataset (ImageNet: 14M images, 1000 classes)
- Remove the final classification layer
- Add new layers for your task (e.g., 10 classes instead of 1000)
- Freeze the pre-trained layers (trainable=False)
- Train only the new layers (fast, needs little data)
- Optionally fine-tune: unfreeze some pre-trained layers with very small learning rate
This works because early layers learn general features (edges, textures) that transfer across tasks. Join Discord PreviousML System DesignNextImbalanced Classification