Week 10: CSV Processing & Exception Handling
4561 words
23 min read
2026-09-12T00:00:00.000Z
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
Parsing tabular files, Python csv module, DictReader/DictWriter, try-except-finally blocks, and input sanitization. # Programming in Python — Week 10: CSV Processing & Exception Handling The official IITM syllabus lists **Week 10: File Operations (Continued)**.

Visual strip
Relevant recall pieces for this note
Programming in Python — Week 10: CSV Processing & Exception Handling
The official IITM syllabus lists Week 10: File Operations (Continued). The syllabus file does not provide finer subtopics, so the material below is a standard continuation of Python file handling and file-processing workflows, not a claim about exact lecture ordering.
TL;DR
Week 9 introduced:
textopen read write append parse close
Week 10 is best thought of as:
textfile ↓ parse records ↓ validate ↓ transform ↓ aggregate ↓ write structured output
The main skills are:
- processing files record-by-record
- maintaining running summaries
- multiple input/output files
- structured records
- headers
- malformed data
- copying/filtering/transforming files
- file position
- read/write mode details
- exception-safe file handling
- turning file data into Python collections and back
1. File processing is usually a pipeline
A realistic file task often looks like:
textINPUT FILE ↓ read one record ↓ parse fields ↓ convert datatypes ↓ validate ↓ process ↓ update result ↓ OUTPUT FILE
This is the same collection-processing logic from Weeks 6–8, except data now arrives from persistent storage.
2. Record-by-record processing
Suppose file:
textA,80 B,90 C,70
Each line is one record:
textname,mark
A clean loop:
pythonwith open("marks.txt", "r") as f: for line in f: name, mark = line.strip().split(",") mark = int(mark) # process record
3. Keep parsing separate from processing
Conceptually separate:
textraw line ↓ parsed fields ↓ typed values ↓ algorithm
For example:
pythonline = "A,80\n" name, mark_text = line.strip().split(",") mark = int(mark_text)
Now:
textname = "A" mark = 80
Only after this should numerical logic begin.
4. Header handling
Suppose:
textname,mark A,80 B,90
The first row is a header.
One simple pattern:
pythonwith open("marks.txt", "r") as f: header = f.readline() for line in f: ...
This consumes the header once.
5. Why header handling matters
If you attempt:
pythonname, mark = "name,mark".split(",") mark = int(mark)
then Python tries:
pythonint("mark")
which fails.
So metadata rows must be treated differently from data rows.
6. Skip blank lines
A robust processing loop may do:
pythonwith open("data.txt", "r") as f: for line in f: line = line.strip() if line == "": continue # parse real data
This avoids accidental conversion of empty strings.
7. Validate field count
Suppose expected:
textname,mark
Then:
pythonparts = line.strip().split(",")
should have:
fields.
You can conceptually check:
pythonif len(parts) != 2: ...
before unpacking.
8. Why validation matters
Malformed line:
textA,80,Delhi
contains three fields.
This fails:
pythonname, mark = line.split(",")
because two variables cannot receive three values.
9. Type validation
Even if field count is correct:
textA,eighty
still fails when converted:
pythonint("eighty")
So file validity has multiple layers:
textcorrect structure + correct datatypes + correct value ranges
10. Exception handling intuition
Python can handle predictable failures using:
pythontry: ... except: ...
Example:
pythontry: mark = int(mark_text) except ValueError: print("Invalid mark")
This is useful when external file data may be malformed.
11. Prefer specific exceptions
Instead of:
pythonexcept:
prefer when possible:
pythonexcept ValueError:
This says exactly what kind of failure you intend to handle.
It avoids accidentally hiding unrelated bugs.
12. Parsing safely
Example:
pythonwith open("marks.txt", "r") as f: for line in f: parts = line.strip().split(",") if len(parts) != 2: continue name, mark_text = parts try: mark = int(mark_text) except ValueError: continue
Now only valid two-field numerical records proceed.
13. Range validation after conversion
Suppose valid marks satisfy:
After parsing:
pythonif not 0 <= mark <= 100: continue
So validation order is:
textstructure ↓ datatype ↓ value range
14. Count valid vs invalid records
pythonvalid = 0 invalid = 0
Then for each row:
pythonif bad_record: invalid += 1 else: valid += 1
This is useful when processing imperfect datasets.
15. Aggregate while reading
You do not always need to store all records.
Example:
pythontotal = 0 count = 0 with open("marks.txt", "r") as f: for line in f: mark = int(line.strip()) total += mark count += 1
This computes summaries in one pass.
16. Streaming average
At the end:
pythonif count > 0: average = total / count
Memory usage does not grow with number of records because we keep only:
texttotal count
rather than the entire dataset.
17. Streaming maximum
pythonmaximum = None with open("numbers.txt", "r") as f: for line in f: x = float(line) if maximum is None or x > maximum: maximum = x
Only one running state variable is needed.
18. Preserve associated record
Suppose file:
textA,80 B,95 C,70
To find best student:
pythonbest_name = None best_mark = None with open("marks.txt", "r") as f: for line in f: name, mark_text = line.strip().split(",") mark = int(mark_text) if best_mark is None or mark > best_mark: best_name = name best_mark = mark
Keep both the maximum value and associated identity.
19. Frequency analysis from files
Suppose file:
textred blue red green red blue
Use:
pythonfreq = {} with open("colors.txt", "r") as f: for line in f: color = line.strip() freq[color] = freq.get(color, 0) + 1
Result:
python{ "red": 3, "blue": 2, "green": 1 }
20. Grouping records from files
File:
textA,10 B,5 A,7 B,2
Grouped totals:
pythontotals = {} with open("data.txt", "r") as f: for line in f: name, value = line.strip().split(",") value = int(value) totals[name] = totals.get(name, 0) + value
Final:
python{ "A": 17, "B": 7 }
21. Group into lists
If you need all values:
pythongroups = {} with open("data.txt", "r") as f: for line in f: name, value = line.strip().split(",") value = int(value) if name not in groups: groups[name] = [] groups[name].append(value)
Now:
python{ "A": [10, 7], "B": [5, 2] }
22. Multiple-pass processing
Sometimes one pass is not enough.
Example:
- compute average
- count how many values exceed average
You may:
- read file twice
- store values in memory
- or reopen the file
23. Reopen for second pass
pythontotal = 0 count = 0 with open("numbers.txt", "r") as f: for line in f: total += float(line) count += 1 average = total / count above = 0 with open("numbers.txt", "r") as f: for line in f: if float(line) > average: above += 1
This uses two passes without storing every value.
24. Why reopening works
At the end of the first pass, file cursor is at EOF.
Opening the file again starts a fresh file object at the beginning.
25. Alternative: store data once
pythonvalues = [] with open("numbers.txt", "r") as f: for line in f: values.append(float(line))
Then:
pythonaverage = sum(values) / len(values) above = sum(1 for x in values if x > average)
This uses more memory but avoids a second disk read.
26. Trade-off: memory vs rereading
Two-pass file:
textless memory + more I/O
Store-all approach:
textmore memory + easier repeated analysis
Algorithm design often involves such trade-offs.
27. Read/write mode "r+"
Python supports combined modes such as:
python"r+"
which permit reading and writing.
However, mixing reading and writing around one file cursor requires more care.
For introductory work, separate input and output files are often clearer.
28. "w+" intuition
"w+" allows reading and writing but also truncates the file when opened.So it can erase previous contents just like
"w".Do not use it casually.
29. "a+" intuition
"a+" allows appending plus reading.Writes go to the end.
Again, mixed read/write modes are more subtle than simple
"r", "w", "a".30. File cursor after writing
If you write to a file and then want to read from it using the same object, cursor position matters.
Example:
pythonf.write("hello")
leaves position after the written text.
A subsequent read begins from that current position unless repositioned.
31. tell()
Python:
pythonf.tell()
reports the current file position.
At a conceptual level:
where is the cursor now?
This helps explain why reads can return nothing after earlier operations.
32. seek()
pythonf.seek(0)
moves the cursor back to the beginning.
Useful in combined read/write workflows.
For basic text processing, reopening the file is often conceptually simpler.
33. Input file + output file
A clean transform pattern:
pythonwith open("input.txt", "r") as src: with open("output.txt", "w") as dst: for line in src: ... dst.write(...)
This separates responsibilities clearly.
34. Filter file
Input:
text5 -2 7 0 -1
Write only positives:
pythonwith open("input.txt", "r") as src: with open("positive.txt", "w") as dst: for line in src: x = int(line) if x > 0: dst.write(f"{x}\n")
Output:
text5 7
35. Transform file
Input:
text1 2 3
Output squares:
pythonwith open("input.txt", "r") as src: with open("squares.txt", "w") as dst: for line in src: x = int(line) dst.write(f"{x ** 2}\n")
Result:
text1 4 9
36. Normalize text file
pythonwith open("input.txt", "r") as src: with open("output.txt", "w") as dst: for line in src: cleaned = line.strip().lower() if cleaned != "": dst.write(cleaned + "\n")
This:
- trims whitespace
- converts case
- removes blank lines
37. Merge files
Suppose:
textpart1.txt part2.txt
Conceptually:
pythonwith open("combined.txt", "w") as dst: with open("part1.txt", "r") as f: for line in f: dst.write(line) with open("part2.txt", "r") as f: for line in f: dst.write(line)
Output contains first file followed by second.
38. Compare two files conceptually
You might:
- read both
- compare their contents
Simple:
pythonwith open("a.txt", "r") as fa: a = fa.read() with open("b.txt", "r") as fb: b = fb.read() print(a == b)
This tests exact textual equality.
39. Exact equality includes whitespace
These are different:
texthello
and:
texthello
because one contains an extra space.
Likewise final newline differences can matter.
Normalization may be needed if formatting differences should be ignored.
40. Copy while changing selected records
Suppose:
textA,80 B,90
and we want to add 5 marks:
pythonwith open("input.txt", "r") as src: with open("output.txt", "w") as dst: for line in src: name, mark = line.strip().split(",") mark = int(mark) + 5 dst.write(f"{name},{mark}\n")
41. Do not overwrite input prematurely
A dangerous idea:
pythonopen("data.txt", "w")
before you finish reading old
data.txt.That truncates the source.
Safer approach:
textread source → write separate output → optionally replace later
42. Temporary-file pattern conceptually
For updating a file safely:
textoriginal file ↓ read ↓ write modified data to temporary file ↓ replace original only after success
This avoids destroying source data halfway through processing.
43. Append logs
Append mode is ideal for event history:
pythonwith open("log.txt", "a") as f: f.write("Task completed\n")
Repeated executions accumulate history.
44. Structured logging
Instead of:
textsomething happened
write predictable fields:
text2026-01-01,INFO,Task completed
Structured records are easier to parse later.
45. Writing dictionaries
Suppose:
pythonfreq = { "cat": 3, "dog": 2 }
Write:
pythonwith open("freq.txt", "w") as f: for word, count in freq.items(): f.write(f"{word},{count}\n")
46. Deterministic output ordering
Dictionary iteration follows Python's mapping order semantics, but if a problem explicitly requires sorted output, sort it:
pythonfor word in sorted(freq): f.write(f"{word},{freq[word]}\n")
Now output is sorted by key.
47. Sort by frequency
pythonitems = sorted( freq.items(), key=lambda pair: pair[1], reverse=True ) with open("freq.txt", "w") as f: for word, count in items: f.write(f"{word},{count}\n")
This connects Week 8 sorting with file output.
48. Parse key-value configuration
Suppose:
texttheme=dark volume=80 language=en
You can parse:
pythonconfig = {} with open("config.txt", "r") as f: for line in f: key, value = line.strip().split("=", 1) config[key] = value
Result:
python{ "theme": "dark", "volume": "80", "language": "en" }
49. Why split("=", 1)?
Without the second argument, every
= is treated as a separator.Using:
pythonsplit("=", 1)
means:
split at most once.
Useful when the value might itself contain the delimiter.
50. Parsing is format-dependent
Different files need different parsers.
Examples:
textone value per line comma-separated tab-separated key=value free-form text
There is no universal
.split() pattern for all files.First understand the data format.
51. Tabs
Tab character:
python"\t"
A tab-separated line can be parsed with:
pythonline.strip().split("\t")
Same idea as comma-separated text.
52. Whitespace-separated data
If fields are separated by arbitrary whitespace:
pythonparts = line.split()
is useful because it handles runs of spaces/tabs.
Example:
textA 80 90
can become:
python["A", "80", "90"]
53. strip().split() vs split()
Often:
pythonline.split()
already ignores surrounding whitespace and splits on whitespace.
So:
pythonline.strip().split()
may be redundant in simple whitespace-delimited cases.
Understand what each method contributes.
54. Exceptions from file opening
This:
pythonopen("missing.txt", "r")
can raise:
textFileNotFoundError
A specific handler:
pythontry: with open("missing.txt", "r") as f: ... except FileNotFoundError: print("File missing")
55. Permission failures
A file may exist but still not be readable/writable due to permissions.
So external resources can fail for reasons unrelated to algorithm logic.
This is a general lesson:
56. with still matters during exceptions
If an exception occurs inside:
pythonwith open(...) as f:
Python still performs file cleanup when leaving the context.
This is one reason
with is preferred.57. File operation vs parsing error
Distinguish:
File error
Could not open/read file.
Parsing error
File opened, but content format is wrong.
Logical error
File and format are valid, but algorithm computes wrong result.
These are separate debugging layers.
58. Example layered debugging
Suppose average calculation fails.
Ask:
- Did file open?
- Were lines actually read?
- Did conversion succeed?
- Was header skipped?
- Were blank lines handled?
- Were totals/counts updated correctly?
- Was division done after loop?
This isolates the failure source.
59. Read-once iterables intuition
File iteration naturally moves forward.
pythonfor line in f: ...
consumes lines.
After completion, another loop over the same file object sees EOF unless cursor is reset or file reopened.
60. This differs from a list
A list:
pythonvalues = [1, 2, 3]
can easily be looped repeatedly:
pythonfor x in values: ... for x in values: ...
The list itself is still present in memory.
A file object has mutable cursor state.
61. Load into list when repeated traversal is useful
pythonwith open("data.txt", "r") as f: lines = f.readlines()
Now
lines is a list and can be traversed repeatedly.Trade-off:
textconvenience ↔ memory usage
62. Generator-style processing intuition
Line-by-line file processing is an example of lazy/streaming behavior:
textget next item process it get next item ...
You need not materialize the entire dataset first.
63. Huge-file reasoning
If a file contains millions of records:
pythonf.read()
may consume substantial memory.
Instead:
pythonfor line in f:
can process incrementally.
This is one reason iteration abstractions matter.
64. Count unique records
pythonseen = set() with open("data.txt", "r") as f: for line in f: value = line.strip() seen.add(value) print(len(seen))
This combines files with sets.
65. Detect duplicates
pythonseen = set() duplicates = set() with open("data.txt", "r") as f: for line in f: value = line.strip() if value in seen: duplicates.add(value) else: seen.add(value)
At end:
pythonduplicates
contains repeated values.
66. Preserve duplicate order
If order matters:
pythonseen = set() duplicates = [] for value in data: if value in seen and value not in duplicates: duplicates.append(value) else: seen.add(value)
Collection choice changes semantics.
67. Index line numbers with enumerate()
pythonwith open("data.txt", "r") as f: for line_number, line in enumerate(f, start=1): print(line_number, line)
This is useful when reporting malformed lines.
68. Example error reporting
pythonwith open("data.txt", "r") as f: for line_number, line in enumerate(f, start=1): parts = line.strip().split(",") if len(parts) != 2: print("Invalid line:", line_number)
Now diagnostics identify where the issue occurred.
69. Multiple input files
Suppose one file contains student names and another contains marks.
You might read both into collections and combine them.
But parallel files can create alignment problems similar to parallel lists.
Whenever possible, related fields are often safer in the same record.
70. One record per line is a strong pattern
Example:
textstudent_id,name,mark
keeps related information together.
This reduces risk of losing synchronization across separate files.
71. File record invariant
If your algorithm assumes:
texteach valid line has exactly 3 fields
that is an invariant/format contract.
The parser should either:
- trust guaranteed input, or
- validate before processing
depending on problem assumptions.
72. Reformat data
Input:
textA 80 B 90
Output:
textA,80 B,90
Program:
pythonwith open("input.txt", "r") as src: with open("output.txt", "w") as dst: for line in src: name, mark = line.split() dst.write(f"{name},{mark}\n")
This changes representation without changing underlying information.
73. Selection while copying
Input file can be treated like a stream of records.
Example:
pythonif mark >= 40: dst.write(...)
This is exactly the filter pattern from collections.
74. Transformation while copying
pythonmark = min(mark + 5, 100)
then write.
This is exactly the transform pattern.
75. Grouping while reading
pythongroups[key].append(value)
This is the group pattern.
So file processing adds no new algorithmic category; it connects old categories to external data.
76. Reduce while reading
Examples:
textsum count min max frequency
These are reductions/aggregations.
You can calculate them without storing every row.
77. File operation pattern library
Count rows
pythoncount += 1
Sum field
pythontotal += value
Frequency
pythonfreq[x] = freq.get(x, 0) + 1
Group
pythongroups[key].append(value)
Filter to output
pythonif condition: dst.write(...)
Transform to output
pythondst.write(format(transform(record)))
Common mistake: forgetting header changes counts
If file has:
textname,mark A,80 B,90
then raw line count is:
But number of student records is:
Understand whether metadata counts as data.
Common mistake: count blank lines
A blank line may not represent a record.
If required, check:
pythonif line.strip() == "": continue
before incrementing count.
Common mistake: strip destroys meaningful delimiter spaces
Suppose fixed-width file relies on exact spaces.
Then blindly using:
pythonstrip()
can alter record structure.
Parsing must match format.
Common mistake: mutate dictionary during iteration
If reading file builds a dictionary and later you iterate over it, avoid adding/removing keys during that same direct iteration unless you specifically understand the behavior.
Safer:
pythonresult = {}
or iterate over a snapshot.
Common mistake: overwrite duplicate dictionary key unintentionally
Input:
textA,80 A,90
This:
pythonmarks[name] = mark
leaves:
python{"A": 90}
If both values matter, use:
pythongroups[name].append(mark)
or aggregate them.
Common mistake: assume order-independent output
If a grader expects sorted output, explicitly sort.
Example:
pythonfor key in sorted(d):
Do not assume arbitrary collection traversal matches required output order.
Common mistake: file already exhausted
pythonfor line in f: ... for line in f: ...
Second loop may execute zero times.
Either:
pythonf.seek(0)
or reopen the file.
Common mistake: forgot newline while writing records
Without:
python"\n"
records can run together:
textA,80B,90C,70
Formatting separators are part of the data format.
Common mistake: extra newline
If
line already includes newline:pythondst.write(line + "\n")
can add an extra blank line.
Know whether your string already contains the separator.
Common mistake: write before validation
Safer processing order:
textread → parse → validate → transform → write
not:
textread → immediately write → discover later it was invalid
Common mistake: open output inside loop
Wrong:
pythonfor item in values: with open("out.txt", "w") as f: f.write(...)
Every iteration reopens with
"w" and truncates previous output.Likely only the last record survives.
89. Correct output loop
pythonwith open("out.txt", "w") as f: for item in values: f.write(...)
Open once.
Write many times.
Close once.
Common mistake: reopening append unintentionally
Opening repeatedly with:
python"a"
may be valid, but it is unnecessary overhead if one open handle can remain active for the whole operation.
Prefer clear scope.
91. Trace multi-record aggregation
File:
textA,3 B,4 A,5 C,2 B,1
Code:
pythontotals = {} for line in f: key, value = line.strip().split(",") value = int(value) totals[key] = totals.get(key, 0) + value
Trace:
text{} A,3 → {"A":3} B,4 → {"A":3,"B":4} A,5 → {"A":8,"B":4} C,2 → {"A":8,"B":4,"C":2} B,1 → {"A":8,"B":5,"C":2}
92. Trace filter/write
Input:
text5 -1 8 0
Program writes values satisfying:
pythonx > 0
Output:
text5 8
The output file itself is the accumulated result.
93. File processing as state machine
At any point, program state may include:
textcurrent file position current line parsed fields running counters running dictionary output written so far
Tracing file programs means tracking both:
94. File state includes cursor position
For read questions, note:
textwhich lines have already been consumed?
For write questions, note:
textwhat has already been written?
This avoids confusion in multi-step I/O traces.
95. A complete processing example
Input:
textname,mark A,80 B,35 C,91 D,50
Goal:
- skip header
- keep passing marks ≥40
- write
name,mark - calculate passing average
pythontotal = 0 count = 0 with open("marks.txt", "r") as src: with open("passed.txt", "w") as dst: src.readline() for line in src: name, mark_text = line.strip().split(",") mark = int(mark_text) if mark >= 40: dst.write(f"{name},{mark}\n") total += mark count += 1 if count > 0: print(total / count)
96. Result of example
Passing students:
textA,80 C,91 D,50
Average:
One file traversal performed:
- parsing
- filtering
- output
- aggregation
97. File pipelines prepare you for Pandas
Manual Week 9–10 code often looks like:
textopen loop through records parse filter aggregate
Later libraries such as Pandas provide higher-level abstractions for the same broad data-processing tasks.
Understanding manual processing first makes those abstractions easier to reason about.
98. Compact Week 10 cheat sheet
python## parse record name, value = line.strip().split(",") value = int(value)
python## skip bad/blank record if line.strip() == "": continue
python## two-pass reading with open(path) as f: ... with open(path) as f: ...
python## copy/transform with open(src_path, "r") as src: with open(dst_path, "w") as dst: for line in src: ...
python## report line numbers for line_no, line in enumerate(f, start=1): ...
python## error handling try: x = int(text) except ValueError: ...
99. File-problem solving checklist
For any file problem, determine:
- What is the input file format?
- Is there a header?
- Can blank lines occur?
- What delimiter is used?
- Which fields need datatype conversion?
- What constitutes a valid record?
- Should invalid records be skipped or reported?
- Do you need all data in memory?
- Can you process one row at a time?
- Is one pass enough?
- What state must persist across records?
- Is output fresh or appended?
- What exact output format is required?
- Does ordering matter?
100. Week 9 → Week 10 progression
Week 9:
textHow do I read and write files?
Week 10:
textHow do I build reliable algorithms around file data?
The conceptual progression is:
101. What you should actually master
By the end of Python Week 10, you should be able to:
- Parse structured file records.
- Handle headers.
- Skip blank lines.
- Validate field counts.
- Convert fields to appropriate types.
- Understand basic exception handling for bad input.
- Aggregate file data in one pass.
- Find min/max while streaming.
- Preserve associated record information.
- Build frequency dictionaries from files.
- Group file records.
- Perform multiple-pass processing.
- Understand memory-vs-rereading trade-offs.
- Work with input and output files simultaneously.
- Filter records into a new file.
- Transform records into a new file.
- Merge/copy files conceptually.
- Understand file cursor state.
- Use
seek()/reopening conceptually when rereading. - Report line numbers with
enumerate(). - Recognize malformed-data problems.
- Avoid destructive
"w"mistakes. - Avoid opening output inside a loop.
- Keep exact output formatting correct.
- Trace both program state and file state.
If these are comfortable, Programming in Python Week 10 is under control.
Quick Recall & Flashcards
Test your active recall before proceeding to practice problems. Format:
Question :: Answer.- Week 10 Diagnostic: In Python's
csvmodule, what doescsv.DictReaderreturn for each row? :: Answer: B —csv.DictReadermaps each row into a dictionary using the header row as keys. - Week 10 Diagnostic: Which block in a
try-except-finallystatement is ALWAYS executed regardless of whether an exception occurred? :: Answer: D — Thefinallyblock always executes, commonly used for cleanup actions like releasing resources.
Practice & Self-Check
Attempt these diagnostic questions to verify your operational understanding.
1
Question 1
In Python's
csv module, what does csv.DictReader return for each row?MCQ
3 Unit Assessment
A. A list of strings
B. A dictionary mapping column header names to field values
C. A tuple of integers
D. A Pandas DataFrame
2
Question 2
Which block in a
try-except-finally statement is ALWAYS executed regardless of whether an exception occurred?MCQ
3 Unit Assessment
A. try
B. except
C. else
D. finally