NLP Projects: Tokenization, Fine-Tuning, Text Classification, Seq2Seq

🎯 Learning Objectives

• Implement text tokenization and preprocessing pipelines
• Fine-tune pretrained transformer models (BERT, GPT) for downstream tasks
• Build text classification and sequence-to-sequence models
• Use Hugging Face Transformers for efficient model development

📋 Prerequisites

• PyTorch basics (Week 1)
• Data Pipelines (Week 2)

1. 📖 Core Content

1.1 Text Tokenization

python
Run Inline
# runnable
from transformers import AutoTokenizer
# Load pretrained tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# Basic tokenization
text = "IIT Madras BS Degree is amazing!"
tokens = tokenizer.tokenize(text)
print(f"Tokens: {tokens}")
# Full encoding
encoded = tokenizer(
    text,
    padding='max_length',
    truncation=True,
    max_length=20,
    return_tensors='pt'
)
print(f"Input IDs: {encoded['input_ids'][0]}")
print(f"Attention Mask: {encoded['attention_mask'][0]}")
# Decoding
decoded = tokenizer.decode(encoded['input_ids'][0])
print(f"Decoded: {decoded}")

1.2 Fine-Tuning BERT for Text Classification

python
Run Inline
# runnable
from transformers import AutoModelForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset
import numpy as np
# Load model and dataset
model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2
)
# Example with IMDB sentiment
# dataset = load_dataset("imdb", split="train[:100]")
# Training arguments
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=64,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir="./logs",
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
)
# Trainer handles training, evaluation, checkpointing
# trainer = Trainer(
#     model=model,
#     args=training_args,
#     train_dataset=dataset,
#     eval_dataset=dataset,
# )
# trainer.train()
print("Training pipeline configured. Ready for fine-tuning.")

1.3 Sequence-to-Sequence with T5

python
Run Inline
# runnable
from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")
# Translation example
input_text = "translate English to French: The weather is nice today."
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_length=50,
    num_beams=4,
    early_stopping=True
)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Input: {input_text}")
print(f"Output: {result}")

1.4 Practical Project Pipeline

python
Run Inline
# runnable
def nlp_project_pipeline():
    """End-to-end NLP project template."""
    # 1. Load and explore data
    df = pd.read_csv('reviews.csv')
    print(f"Classes: {df['sentiment'].value_counts()}")
    # 2. Preprocess text
    def clean_text(text):
        text = text.lower()
        text = re.sub(r'[^a-zA-Z\s]', '', text)  # Remove special chars
        text = re.sub(r'\s+', ' ', text).strip()   # Remove extra spaces
        return text
    df['clean_text'] = df['text'].apply(clean_text)
    # 3. Tokenize
    encodings = tokenizer(
        list(df['clean_text']),
        truncation=True,
        padding=True,
        max_length=128
    )
    # 4. Create Dataset
    class TextDataset(torch.utils.data.Dataset):
        def __init__(self, encodings, labels):
            self.encodings = encodings
            self.labels = labels
        def __getitem__(self, idx):
            item = {k: torch.tensor(v[idx]) for k, v in self.encodings.items()}
            item['labels'] = torch.tensor(self.labels[idx])
            return item
        def __len__(self):
            return len(self.labels)
    # 5. Train using Hugging Face Trainer
    # trainer = Trainer(model=model, args=training_args,
    #                   train_dataset=train_dataset, eval_dataset=val_dataset)
    # trainer.train()
    print("NLP pipeline ready for production data")

1.5 Why This Matters

• Sentiment analysis: Monitor brand perception
• Chatbots: Customer service automation
• Translation: Global content delivery
• Summarization: Information extraction from documents The Hugging Face ecosystem (Transformers, Datasets, Tokenizers) has made fine-tuning large language models accessible to everyone.

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Not Setting Padding Correctly

Pitfall 2: Training All Layers When Fine-Tuning

4. 📝 Practice Questions

Q1: You fine-tune BERT on 500 movie reviews for sentiment analysis. Training loss decreases, but validation accuracy stays at 50% (same as random). Diagnose the problem.

Likely causes:

1. Dataset too small: 500 reviews aren't enough to fine-tune 110M parameters. Try data augmentation or use a smaller model (DistilBERT).
2. Learning rate too high: BERT fine-tuning needs a small LR (2e-5 to 5e-5). Higher LR causes catastrophic forgetting.
3. Training epochs too few: BERT typically needs 3-5 epochs. With 500 samples, try 5-10 epochs.
4. Class imbalance: If 450/500 reviews are positive, the model learns to predict "positive" for everything. Check class distribution.
5. Wrong tokenizer: Using bert-base-uncased tokenizer with a case-sensitive task. Check that preprocessing matches the pretrained model's training data.

Fix: Increase data to 2000+ examples, use LR=2e-5, train for 5 epochs, check class balance, use appropriate tokenizer.

5. 🔗 Cross-References

• Previous: Experiment Tracking (Week 9)
• Related: DL-CV Cross-course
• External: Hugging Face Course (huggingface.co/learn) Join Discord PreviousExperiment TrackingNextBSDA5013 — Deep Learning Practice