Regular Expressions: Pattern Matching for Data Cleaning
305 words
2 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# Regular Expressions: Pattern Matching for Data Cleaning ## 🎯 Learning Objectives - Write regex patterns to match text patterns - Extract structured data from unstructured text - Clean and normalize text data using regex - Use regex in pandas for column operations ## 📖 Core Content ### 1.1 Why Regex for Data Scie...
Regular Expressions: Pattern Matching for Data Cleaning
🎯 Learning Objectives
- Write regex patterns to match text patterns
- Extract structured data from unstructured text
- Clean and normalize text data using regex
- Use regex in pandas for column operations
📖 Core Content
1.1 Why Regex for Data Science?
Data comes as messy text: phone numbers in various formats, messy addresses, log files, social media posts. Regex is the most precise tool for extracting structured information from unstructured text.
1.2 Key Patterns
| Pattern | Matches | Example |
|---|---|---|
\d | Any digit | \d{10} → phone numbers |
\w | Word character (letter, digit, _) | \w+ → words |
\s | Whitespace | \s+ → spaces/tabs |
. | Any character (except newline) | .* → everything |
[a-z] | Range | [A-Z][a-z]+ → Capitalized words |
^ | Start of string | ^Error → lines starting with Error |
$ | End of string | \.$ → lines ending with period |
1.3 Practical Examples
python# runnable import re import pandas as pd # Extract email addresses text = "Contact: [email protected], or [email protected]" emails = re.findall(r'[\w.+-]+@[\w-]+\.[\w.]+', text) print(f"Emails: {emails}") # Validate phone numbers phone = "+91-9876543210" pattern = r'^\+?91?[-.\s]?[6-9]\d{9}$' print(f"Valid Indian phone: {bool(re.match(pattern, phone))}") # Pandas: Extract year from dates df = pd.DataFrame({'date': ['2024-01-15', '2023-12-01', '2022-06-30']}) df['year'] = df['date'].str.extract(r'(\d{4})') print(df)
1.4 Why This Matters
Regex is the most universally useful text processing skill for data scientists. It's available in Python, R, SQL, bash, and every programming language. One regex pattern can replace 50 lines of manual string processing.
2. 📝 Practice Questions
Q1: Write a regex that matches Indian PIN codes (6 digits) from text. Example: "Bangalore 560001" should match 560001.Pattern:\b[1-9]\d{5}\b
\b: Word boundary (prevents matching within longer numbers)[1-9]: First digit 1-9 (PIN codes start with non-zero)\d{5}: Exactly 5 more digits\b: Word boundaryTest:re.findall(r'\b[1-9]\d{5}\b', "Delhi 110001, Mumbai 400001, 123456789")→['110001', '400001']Join Discord PreviousCloud ComputingNextCLI Tools