Pipes & Redirection — The Unix Philosophy

🎯 Learning Objectives

• Redirect stdin, stdout, and stderr
• Chain commands with pipes
• Use tee to split output
• Use xargs to build and execute command lines
• Use here documents for multi-line input

1. Standard Streams

2. Output Redirection

bash
# Redirect stdout to file (overwrite)
ls -la > output.txt          # Write listing to file
echo "Hello" > file.txt      # Create file with content
cat file1.txt > file2.txt    # Copy file1 to file2
# Redirect stdout to file (append)
echo "More data" >> log.txt   # Append to file
ls >> listing.txt             # Append listing
# Redirect stderr
ls nonexistent 2> error.txt   # Redirect error to file
ls nonexistent 2>> error.log  # Append error
# Redirect both stdout and stderr
command > output.txt 2>&1     # Both to output.txt (old syntax)
command &> output.txt          # Both to output.txt (bash syntax)
command > output.txt 2> error.txt  # Separate files
# Discard output
command > /dev/null           # Discard stdout
command 2> /dev/null          # Discard stderr
command &> /dev/null          # Discard both

3. Input Redirection

bash
# Read input from file
sort < unsorted.txt           # Sort contents of file
wc -l < file.txt              # Count lines (only content, not filename)
# Here document (multi-line input)
cat << EOF
This is a
multi-line
input
EOF
# Here string
grep "pattern" <<< "search this string"

4. Pipes — The Unix Superpower

bash
# Basic pipe
ls -la | grep ".txt"           # List files, filter .txt
cat file.txt | wc -l           # Count lines in file
# Multi-stage pipeline
ls -la | grep ".txt" | sort -k5 -n | head -5
# 1. List files
# 2. Keep only .txt files
# 3. Sort by 5th column (size) numerically
# 4. Show top 5 smallest
# Classic Unix pipelines
ps aux | grep apache           # Find apache processes
history | grep git             # Find git commands in history
dmesg | tail -20               # Last 20 kernel messages
cat access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -10
# Count unique IPs in web server log, show top 10

5. tee — Split Output

bash
# Save output to file while also displaying it
ls -la | tee listing.txt
ls -la | tee -a listing.txt   # Append mode
# In a pipeline
cat log.txt | grep "ERROR" | tee errors.txt | wc -l
# Saves ERROR lines to errors.txt AND counts them
# Logging while monitoring
./long_running_script.sh | tee output.log

6. xargs — Build and Execute Commands

bash
# Basic usage
ls *.txt | xargs rm             # Remove all .txt files
find . -name "*.tmp" | xargs rm # Remove all .tmp files
# With options
find . -type f -name "*.py" | xargs wc -l  # Count lines in all Python files
cat urls.txt | xargs curl -O               # Download all URLs in file
# Interactive (prompt before each)
find . -name "*.log" | xargs -p rm         # Ask before deleting
# Limit arguments per command
echo {1..100} | xargs -n 10 echo          # Echo 10 numbers per line
# Parallel execution
cat urls.txt | xargs -P 4 curl -O         # Download 4 files at once

7. Combining Redirection and Pipes

bash
# Redirect errors but pipe stdout
command 2> error.log | grep "pattern"
# Pipe to multiple commands (using tee)
command | tee file.txt | wc -l
# Redirect both stdout and stderr through pipe
command 2>&1 | grep "error"
# Useful combination: find + grep
find . -type f -name "*.txt" -exec grep -l "TODO" {} \;
# Equivalent with xargs:
find . -type f -name "*.txt" | xargs grep -l "TODO"

8. Practice Questions

Q1: What's the difference between > and >>?

Answer: > overwrites the output file. >> appends to the output file. echo "a" > f.txt then echo "b" > f.txt → f.txt contains "b". echo "a" >> f.txt then echo "b" >> f.txt → f.txt contains "a\nb". Q2: How do you redirect both stdout and stderr to the same file?

Answer: command > file 2>&1 (redirect stderr to where stdout is going) or command &> file (bash shorthand, redirects both). The order matters: 2>&1 must come after >. Q3: What does the pipe operator | do?

Answer: Connects the stdout of the left command to the stdin of the right command. ls | grep txt sends the output of ls into grep as input. This allows chaining simple commands to build complex operations. Q4: How do you discard all output (stdout and stderr)?

Answer: command &> /dev/null. /dev/null is a special file that discards everything written to it. Q5: What does tee do?

Answer: Splits output: sends it to both a file and stdout. Like a T-pipe in plumbing that branches the flow. command | tee file.txt displays output on screen AND saves it to file.txt. Q6: What is xargs used for?

Answer: Builds and executes command lines from stdin. Useful when a command doesn't accept piped input. Example: find . -name "*.txt" | xargs grep "pattern" — this is equivalent to grep "pattern" file1.txt file2.txt .... Q7: What does > do when used inside a pipeline?

Answer: Typically redirection applies before the pipe. command > file.txt | something — the > redirects command's stdout to file.txt (nothing goes through the pipe). To pipe AND save, use tee. Q8: Write a pipeline to find the 10 largest files in /var/log.

bash
find /var/log -type f -exec du -h {} + | sort -rh | head -10
# Or with xargs:
find /var/log -type f | xargs du -h | sort -rh | head -10

📐 Key Concepts

🔗 Cross-References

• Next: Process Management Join Discord Previous4.1 Text ProcessingNext6.1 Process Management