Text Processing — The Unix Power Tools

🎯 Learning Objectives

• View and navigate files with cat, head, tail, less
• Search text with grep (regex, recursive, inverted)
• Edit text streams with sed (substitute, delete)
• Process structured text with awk
• Sort, count, and transform text with sort, uniq, wc, cut, tr

1. Viewing Files

bash
# Concatenate and display
cat file.txt               # Print entire file
cat -n file.txt            # Print with line numbers
cat file1.txt file2.txt    # Concatenate multiple files
# Head and tail
head file.txt              # First 10 lines
head -n 20 file.txt        # First 20 lines
tail file.txt              # Last 10 lines
tail -n 50 file.txt        # Last 50 lines
tail -f log.txt            # Follow (watch) file for new lines — great for logs!
# Paginated viewing
less file.txt              # Scrollable viewer (q to quit, / to search)
more file.txt              # Simpler pager (space to advance)
# View in reverse order
tac file.txt               # Print lines in reverse order

2. Searching — grep

bash
# Basic search
grep "pattern" file.txt             # Search for pattern
grep -i "pattern" file.txt          # Case-insensitive
grep -v "pattern" file.txt          # Invert match (show lines WITHOUT pattern)
grep -n "pattern" file.txt          # Show line numbers
grep -c "pattern" file.txt          # Count matching lines
# Recursive
grep -r "TODO" src/                  # Search recursively in directory
grep -R "error" /var/log/           # Search all log files
grep -rn "class" *.py               # Recursive with line numbers
# Regular expressions
grep "^Start" file.txt              # Lines starting with "Start"
grep "end$" file.txt                # Lines ending with "end"
grep "[0-9]" file.txt               # Lines containing digits
grep "^$" file.txt                  # Empty lines
grep -E "error|failed" log.txt      # Extended regex (alternation)
# Context
grep -A 2 "pattern" file.txt        # Show 2 lines After match
grep -B 2 "pattern" file.txt        # Show 2 lines Before match
grep -C 3 "pattern" file.txt        # Show 3 lines of Context

3. Stream Editor — sed

bash
# Substitute
sed 's/old/new/' file.txt           # Replace first occurrence on each line
sed 's/old/new/g' file.txt          # Replace all occurrences (global)
sed 's/old/new/2' file.txt          # Replace second occurrence on each line
sed 's/old/new/gi' file.txt         # Global + case-insensitive
sed 's/old/new/' file.txt > new.txt # Save to file
sed -i 's/old/new/g' file.txt       # Edit file in-place
# Line-specific
sed '3s/old/new/' file.txt          # Replace only on line 3
sed '1,5s/old/new/' file.txt        # Replace on lines 1-5
sed '/pattern/s/old/new/' file.txt  # Replace only on lines matching pattern
# Delete lines
sed '3d' file.txt                   # Delete line 3
sed '1,5d' file.txt                 # Delete lines 1-5
sed '/^$/d' file.txt                # Delete empty lines
sed '/pattern/d' file.txt           # Delete lines matching pattern
# Print lines
sed -n '5,10p' file.txt             # Print lines 5-10 only
sed -n '/error/p' log.txt           # Print lines matching "error"

4. Text Processing — awk

bash
# Print fields
awk '{print $1}' file.txt           # Print first field (column)
awk '{print $1, $3}' file.txt       # Print fields 1 and 3
awk '{print $NF}' file.txt          # Print last field
awk '{print NR, $0}' file.txt       # Print line number and entire line
# With conditions
awk '/error/' log.txt               # Print lines matching "error"
awk '$3 > 100' data.txt             # Print if field 3 > 100
awk '$1 == "Alice"' data.txt        # Print if field 1 equals "Alice"
# Field separator
awk -F: '{print $1, $3}' /etc/passwd  # Fields separated by colon
awk -F',' '{print $1}' data.csv        # Fields separated by comma
# BEGIN and END blocks
awk 'BEGIN {sum=0} {sum+=$1} END {print "Total:", sum}' numbers.txt

5. Sorting and Counting

bash
# Sort
sort file.txt                    # Alphabetical sort
sort -n file.txt                 # Numerical sort
sort -r file.txt                 # Reverse sort
sort -k2 file.txt                # Sort by second field
sort -t, -k3 -n data.csv         # Sort CSV by 3rd column numerically
# Unique lines (usually used after sort)
sort file.txt | uniq             # Remove consecutive duplicates
sort file.txt | uniq -c          # Count occurrences
sort file.txt | uniq -d          # Show only duplicates
sort file.txt | uniq -u          # Show only unique lines
# Count
wc file.txt                      # Lines, words, characters
wc -l file.txt                   # Line count only
wc -w file.txt                   # Word count only

6. Column Extraction and Transformation

bash
# Cut (extract columns)
cut -f1,3 file.txt               # Extract fields 1 and 3 (tab-separated)
cut -d, -f1 data.csv             # Comma-separated, first field
cut -c1-10 file.txt              # Extract characters 1-10
# Translate (character replacement)
tr '[:lower:]' '[:upper:]' < file.txt  # Convert to uppercase
tr ' ' '\n' < file.txt                 # Replace spaces with newlines
tr -d ' ' < file.txt                   # Delete all spaces
# Paste (merge files line by line)
paste file1.txt file2.txt        # Merge columns
# Join (on common field)
join file1.txt file2.txt         # Join on first column

7. Practical Pipelines

bash
# Count unique IPs in web server log
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -10
# Find top 5 largest files
find . -type f -exec du -h {} + | sort -rh | head -5
# Find most common words
tr ' ' '\n' < file.txt | sort | uniq -c | sort -nr | head -10
# Get process with highest memory usage
ps aux --sort=-%mem | head -5
# Extract and sort URLs from log
grep -o 'http[s]://[^ ]*' access.log | sort -u

8. Practice Questions

Q1: How do you find all lines containing "ERROR" in log files?

Answer: grep "ERROR" log.txt (basic), grep -r "ERROR" /var/log/ (recursive), grep -ri "error" *.log (case-insensitive). Q2: How do you replace all occurrences of "foo" with "bar" in a file?

Answer: sed -i 's/foo/bar/g' file.txt. Without -i, it prints to stdout. The g flag replaces all occurrences on each line. Q3: How do you print the first 10 lines of a file?

Answer: head -n 10 file.txt or simply head file.txt (default 10 lines). Q4: How do you watch a file for new lines being added?

Answer: tail -f filename.txt. Shows new lines as they're appended. Press Ctrl+C to stop. Great for monitoring log files. Q5: How do you count the number of lines, words, and characters in a file?

Answer: wc file.txt shows all three. wc -l for lines only, wc -w for words, wc -c for characters. Q6: How do you sort a CSV file by the second column numerically?

Answer: sort -t, -k2 -n data.csv. -t, sets comma as field separator, -k2 sorts by second field, -n for numerical sort. Q7: How do you extract the first field from /etc/passwd?

Answer: cut -d: -f1 /etc/passwd or awk -F: '{print $1}' /etc/passwd. Both split by colon and print the first field (username). Q8: Write a pipeline to find the 5 most frequent words in a file.

bash
tr ' ' '\n' < file.txt | sort | uniq -c | sort -nr | head -5

This: converts spaces to newlines, sorts alphabetically, counts unique, sorts by count descending, shows top 5.

📐 Key Concepts

🔗 Cross-References

• Next: Pipes & Redirection Join Discord Previous3.1 File PermissionsNext5.1 Pipes & Redirection