CLI Tools: Linux Command Line for Data Science

🎯 Learning Objectives

• Navigate the filesystem and manage files from the command line
• Process text data using grep, sed, awk, and pipes
• Automate data processing tasks with shell scripts
• Use CLI tools for quick data exploration

📖 Core Content

1.1 Essential Commands

bash
# File operations
ls -lh          # List files with sizes
head -n 20 data.csv   # First 20 rows
tail -n 10 data.csv   # Last 10 rows
wc -l data.csv        # Count lines
# Text processing
grep "error" logfile.txt     # Find lines containing "error"
cut -d',' -f1,3 data.csv     # Extract columns 1 and 3
sort -t',' -k2 -n data.csv   # Sort by column 2 numerically
uniq -c data.csv             # Count unique values
# Pipes (combine commands)
cat data.csv | wc -l                           # Count lines
cat data.csv | cut -d',' -f2 | sort | uniq -c  # Frequency of column 2 values
cat large.csv | head -1000 > sample.csv         # Create a sample

1.2 One-Liner Data Exploration

bash
# Quick data profiling
head -1 data.csv                    # Header
cat data.csv | awk -F',' '{print NF}' | sort | uniq    # Column count
cat data.csv | awk -F',' '{print $1}' | sort | uniq -c | sort -rn | head  # Most common values in col 1
cat data.csv | awk -F',' '{print NR, length($0)}' | sort -k2 -rn | head  # Longest rows
# Check for missing values
cat data.csv | grep -c ",,"        # Count rows with consecutive commas
cat data.csv | awk -F',' '{for(i=1;i<=NF;i++) if($i=="") count++} END{print count}'  # Total missing values

1.3 Why This Matters

2. 📝 Practice Questions

Q1: You have a 5GB CSV file with 50M rows. You need to sample 10K random rows. Write a bash one-liner to do this without loading the entire file into memory.

bash
# Using shuf (random sampling)
cat data.csv | shuf -n 10000 > sample.csv

# If the file has a header you want to preserve:
head -1 data.csv > sample.csv && tail -n +2 data.csv | shuf -n 9999 >> sample.csv

This streams the file line by line, never loading more than necessary into memory. Python's pd.read_csv() would need to load the 5GB file completely (or use chunks, which is more code). Join Discord PreviousRegular ExpressionsNextDVC: Data Version Control