Learning Objectives

• Understand DNN-HMM hybrid systems
• Apply end-to-end models (CTC, attention)
• Compare DNN architectures for speech
• Neural network basics
• Week 6: ASR fundamentals

1. DNN-HMM Hybrid Systems

• Input: Stacked MFCC frames (context window of 11 frames: +-5)
• Output: Probability over HMM states (senones, typically thousands)
• Training: Cross-entropy, then sequence discriminative training (MMI, sMBR) Advantages over GMM: DNNs model complex, non-linear relationships, use correlated features, handle high-dimensional input.

2. End-to-End Models

• Input: Acoustic features -> Output: Character/phoneme sequence
• No explicit alignment needed
• Blank label handles variable-length output
• Loss: Sum over all possible alignments
• Decoding: Greedy or beam search with LM Listen, Attend, Spell (LAS):
• Encoder (Listener): Bi-directional RNN/CNN processes acoustic features
• Decoder (Speller): Auto-regressive RNN generates output characters
• Attention: Aligns encoder outputs to decoder steps

3. Transformer for Speech

• Conformer: Combines convolution + self-attention (best for ASR)
• Whisper: Large transformer trained on 680k hours of multilingual data

Q1: How do DNNs improve over GMMs for ASR?

DNNs can model complex, non-linear relationships between features and states. They work with correlated, high-dimensional features (no decorrelation needed). Better at generalizing across speakers and conditions. Q2: How does CTC handle variable-length input/output?

CTC allows repetition of labels and blank label (-). Output length = input length (after downsampling). Collapse repeated labels, remove blanks. Loss sums over all possible alignments (dynamic programming). Q3: What is the difference between CTC and attention-based models?

CTC assumes conditional independence (output frames independent given input). Attention models (LAS) are autoregressive (each output depends on previous outputs). CTC is simpler, attention handles context better but slower for decoding. Q4: What is the Conformer architecture?

Hybrid of convolution (captures local patterns like formants) and self-attention (captures long-range dependencies like syntactic context). State-of-the-art for ASR since 2020. Q5: What is sequence discriminative training?

Train to optimize WER directly rather than frame-level error. Criteria: MMI (Maximum Mutual Information), sMBR (state-level Minimum Bayes Risk). Small but consistent improvement over cross-entropy. Q6: Write a simple inference script with Whisper:

python
Run Inline
import whisper
model = whisper.load_model("base")
result = model.transcribe("audio.mp3")
print(result["text"])
# Output: transcribed text with timestamps