Lexical Analysis — Tokens, Regex, NFA→DFA, Flex

🎯 Learning Objectives

• Define tokens, lexemes, and patterns
• Construct NFA from regular expressions (Thompson's construction)
• Convert NFA to DFA (subset construction)
• Write Flex lex specifications

1. Introduction to Lexical Analysis

1.1 Intuition

1.2 Lexer Role in Compiler

1.3 Token Types

2. Regular Expressions to NFA

2.1 Thompson's Construction

3. NFA to DFA (Subset Construction)

3.1 Algorithm

pseudo
1. Start state of DFA = ε-closure(start state of NFA)
2. For each DFA state (set of NFA states):
   a. For each input symbol a:
      - Let next = ε-closure(move(state_set, a))
      - If next is non-empty and new → add to DFA states
      - Add transition state_set --a--> next
3. Accepting DFA states = any set containing an NFA accept state

3.2 Worked Example

3.3 DFA Minimization

1. Partition into accept and non-accept states
2. Iteratively split groups that have different transitions for the same input
3. Continue until no changes

4. Flex (Scanner Generator)

4.1 Flex Specification Structure

flex
%{
/* C declarations — included in generated scanner */
#include "tokens.h"
int line_count = 1;
%}
/* Regular expression definitions */
DIGIT    [0-9]
ID       [a-zA-Z_][a-zA-Z0-9_]*
NUMBER   {DIGIT}+
WS       [ \t\n]+
%%
/* Rules: pattern → action */
{WS}        { /* skip whitespace */ }
"int"       { return INT; }
"if"        { return IF; }
"while"     { return WHILE; }
{ID}        { yylval.id = strdup(yytext); return ID; }
{NUMBER}    { yylval.num = atoi(yytext); return NUM; }
"+"         { return PLUS; }
"-"         { return MINUS; }
"="         { return ASSIGN; }
";"         { return SEMI; }
.           { printf("Unknown token: %s (line %d)\n", yytext, line_count); }
%%
/* C code — user functions */
int main() {
    yylex();  // Run the scanner
    return 0;
}

4.2 Generated Scanner

5. Common Pitfalls

Pitfall 1: Regex too greedy (overlapping patterns)

Pitfall 2: Forgetting to handle whitespace and comments

Pitfall 3: ε-closure includes the state itself

6. 📐 Key Formulas / Concepts

7. 📝 Practice Questions

Q1: Write a regular expression for identifiers in C (letters, digits, underscores, must start with letter or underscore).

Answer: [a-zA-Z_][a-zA-Z0-9_]* Q2: Build an NFA for regex (a|b)*abb using Thompson's construction.

Answer: The NFA has: (1) NFA for a|b (union of a and b), (2) Kleene star of that, (3) concatenation with a, then b, then b. The resulting NFA has 11 states (starting from ε-NFA for star and concatenation). Q3: What is the purpose of the ε-closure in subset construction?

Answer: The ε-closure accounts for all states reachable via ε-transitions without consuming input. Since NFA can take ε-transitions "for free," the DFA state must track all possible NFA states reachable before processing the next input symbol. Q4: Why does Flex use the "longest match" rule?

Answer: To correctly handle tokens where one is a prefix of another. For example, <= should be a single token (LESS_EQ), not < followed by =. Longest match ensures multi-character operators are recognized correctly. Q5: Convert the DFA from the worked example above into a minimal DFA.

Answer: Start with partitions: Accepting = {B, C, D} (all contain NFA accepting state 7) vs Non-accepting = {A}. For {B,C,D}: B goes to C on b, D on c; C goes to C on b, D on c; D goes to C on b, D on c. Since B, C, D have identical transition patterns, they can be merged into one state. Final DFA: A--a-->BCD, BCD--b-->BCD, BCD--c-->BCD.

8. 🔗 Cross-References

• Week 3 - Syntax Analysis: Token stream feeds into parser
• BSCS3021 (ToC): Regular languages, NFA, DFA theory
• Week 2 - Compiler Overview: Where lexer fits in compiler phases Join Discord PreviousCompiler OverviewNextSyntax Analysis I