Prompt Engineering: In-Context Learning, Chain-of-Thought

🎯 Learning Objectives

• Understand in-context learning and how it enables zero-shot/few-shot performance
• Design effective prompts using chain-of-thought reasoning
• Apply instruction tuning for aligned model behavior
• Evaluate prompt engineering strategies systematically

📋 Prerequisites

• Basic understanding of LLM inference
• GPT/decoder-only architecture
• Tokenization

1. 📖 Core Content

1.1 What is Prompt Engineering?

1.2 Types of Prompting

Zero-Shot Prompting

pseudo
Translate to French: "Hello, how are you?"
→ "Bonjour, comment allez-vous?"

Few-Shot Prompting (In-Context Learning)

pseudo
Translate to French:
English: "Hello" → French: "Bonjour"
English: "Goodbye" → French: "Au revoir"
English: "Thank you" → French:
→ "Merci"

Chain-of-Thought (CoT) Prompting

pseudo
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls.
Each can has 3 balls. How many tennis balls does he have now?
Let's think step by step:
1. Roger starts with 5 balls
2. He buys 2 cans × 3 balls = 6 balls
3. Total = 5 + 6 = 11
A: 11

1.3 In-Context Learning Mechanics

How does ICL work?

1. Pattern recognition: The model recognizes the pattern from examples
2. Task specification: Examples define the input-output mapping
3. Format induction: Examples set the expected response format
4. Meta-learning: The model "learns" to follow the pattern within the forward pass

Factors Affecting ICL

python
Run Inline
# runnable
def demonstrate_icl_format_importance():
    """Show how format consistency affects ICL"""
    good_prompt = """Sentiment analysis examples:
Text: "I love this movie!" → Sentiment: Positive
Text: "This is terrible" → Sentiment: Negative
Text: "The product is okay" → Sentiment: Neutral
Text: "What a wonderful day!" → Sentiment:"""
    bad_prompt = """Examples:
- "I love this movie!" → Positive
- Text: "This is terrible" → Negative
- "The product is okay" → neutral
Text: "What a wonderful day!" →"""
    print("Good prompt format:")
    print("Consistent: Text → Sentiment: Label")
    print("Each example: same structure, same separator")
    print()
    print("Bad prompt format:")
    print("Inconsistent: different structures, different separators")
    print("Labels have inconsistent capitalization")
demonstrate_icl_format_importance()

1.4 Chain-of-Thought (CoT) Variants

1.5 Instruction Tuning

pseudo
Instruction: "Translate to French: Hello"
Response: "Bonjour"

• FLAN (Fine-tuned LAnguage Net)
• InstructGPT dataset
• OpenAssistant conversations
• Self-Instruct (GPT-generated instructions) Effect: Models become better at following diverse instructions, reducing the need for complex prompt engineering.

1.6 Why This Matters

• Task automation: Convert natural language tasks to reliable LLM outputs
• Reasoning: CoT improves math, logic, and multi-step reasoning
• Tool use: Prompts can make models use calculators, search, code execution
• Safety: Well-designed prompts reduce harmful outputs

6. 📝 Practice Questions

Q1: Why does adding "Let's think step by step" improve math reasoning even without examples?

Zero-shot CoT works because:

1. It triggers the model's internal reasoning sequence learned during pre-training
2. LLMs are trained on data containing step-by-step reasoning (blogs, tutorials, textbooks)
3. The phrase "Let's think step by step" activates the "reasoning" mode rather than "direct answer" mode
4. It shifts the model from pattern-matching to algorithmic processing

Without CoT, the model tries to "guess" the answer directly. With CoT, it simulates the reasoning process it has seen in training data. Q2

<strong>Q2</strong>: For few-shot prompting with 5 examples, why might changing the example order change the output?

LLMs have recent bias or primacy bias depending on architecture:

• Primacy bias: Examples at the start have more influence (common in larger models)
• Recency bias: Examples at the end have more influence (common in smaller models)

This happens because attention weights distribute unevenly across positions. The first and last examples may have higher attention weight due to position bias.

Solution: Try multiple orderings and average results, or use symmetric example ordering. Q3

<strong>Q3</strong>: For a classification task, should you provide balanced or imbalanced examples?

Provide balanced examples matching the real distribution. If a task has 90% positive and 10% negative examples in practice:

• Training examples: Show 5 positive and 5 negative (balanced)
• The model learns the decision boundary, not the label distribution
• The model may still over-predict the majority class due to prior probabilities

For highly imbalanced tasks, you can:

1. Balance the examples (recommended)
2. Explicitly mention the distribution in the prompt
3. Adjust the final prediction probability threshold Q4

<strong>Q4

<strong>Q4</strong>: Design a prompt that makes an LLM extract structured data (JSON) from unstructured text.

sql
Extract the following information from the text and return as JSON:
- person_name: The name of the person mentioned
- organization: The company or organization
- date: Any date mentioned (format: YYYY-MM-DD)
- amount: Any monetary amount (as a number)

Text: "John Smith from Acme Corp announced on March 15, 2024 that they raised $10 million in Series A funding."

Output:
{
  "person_name": "John Smith",
  "organization": "Acme Corp",
  "date": "2024-03-15",
  "amount": 10000000
}

Key elements: clear schema definition, input specification, and an example output showing the exact format. Q5

<strong>Q5

<strong>Q5</strong>: Compare the robustness of zero-shot vs few-shot prompting to prompt variations.

Zero-shot: Highly sensitive to phrasing variations:

• "Translate to French: Hello" → "Bonjour"
• "Say the French word for hello" → "Bonjour"
• "Hello in French is:" → might generate "Hello in French is bonjour" (different format)

Few-shot: More robust because examples anchor the format:

• "English: Hello → French: Bonjour\nEnglish: Goodbye → French: Au revoir\nEnglish: Thank you → French:"
• The model is much more likely to output just "Merci" because the output format is established.

Few-shot reduces format sensitivity by providing concrete examples of the desired output structure. Q6

<strong>Q6

<strong>Q6</strong>: How does self-consistency (sampling multiple CoT paths) improve reasoning accuracy?

Self-consistency:

1. Generate k (typically 5-20) reasoning paths using sampling (temperature > 0)
2. Extract the answer from each path
3. Take the majority vote (or highest confidence answer)

This works because:

• Different reasoning paths may make different errors
• The most common answer across diverse reasoning paths is likely correct
• Random errors cancel out; systematic correct reasoning converges

Accuracy improvements: 5-15% on math reasoning benchmarks. Q7

<strong>Q7

<strong>Q7</strong>: What is the "reversal curse" in LLMs, and how does prompt engineering help?

The reversal curse: LLMs can answer "Who is Tom Cruise's mother?" but not "Who is Mary Lee Pfeiffer's son?" even though the information is identical.

This reveals that LLMs capture directional relationships better than symmetric ones.

Prompt engineering mitigation:

• Explicitly state the relationship bidirectionally: "Tom Cruise's mother is Mary Lee Pfeiffer. Mary Lee Pfeiffer's son is Tom Cruise."
• Use structured knowledge representations
• Ask the model to verify: "Is there any other person related to this?"

The curse is an inherent limitation of causal language modeling that instruction tuning partially addresses. Q8

<strong>Q8

<strong>Q8</strong>: Design a prompt that makes the model generate step-by-step reasoning before giving a final answer, then extract just the final answer programmatically.

pseudo
Solve the following problem step by step, then give the final answer in a format I can parse.

Problem: John has 12 apples. He gives half to Mary. Then he gets 5 more. How many does he have?

Let's solve step by step:
[model's reasoning]

Final Answer: [number]

Programmatic extraction: Parse text after "Final Answer:" using regex.

python
Run Inline
import re
output = "...reasoning...Final Answer: 11"
match = re.search(r"Final Answer:\s*(\d+)", output)
if match:
    answer = int(match.group(1))

This structure enables both human-readable reasoning and machine-parseable answers. Q9

<strong>Q9

<strong>Q9</strong>: A prompt returns English text when you want JSON. Does the issue lie in the prompt, the model, or both?

Usually the prompt needs improvement, not the model:

1. Format specification: Be explicit "Return ONLY valid JSON, no other text"
2. Example: Provide a JSON example in the prompt
3. System message: For API-based models, use the system role to set output format
4. Constraints: "Start with { and end with }. Do not include any text before or after."

If the prompt is well-designed and the model still outputs non-JSON:

• Try a different model (some are better at structured output)
• Use constrained decoding (e.g., grammar-based sampling)
• Adjust temperature (lower is more deterministic)

The failure is rarely "the model can't output JSON" — it's usually "the prompt didn't sufficiently constrain the output." Q10

<strong>Q10

<strong>Q10</strong>: How does instruction-tuned (RLHF) model behavior differ from base model behavior in terms of prompt sensitivity?

Instruction-tuned models (like ChatGPT) differ from base models (like GPT-3 base):

1. Lower prompt sensitivity: RLHF models are trained to follow diverse instructions, so small prompt variations matter less
2. Format compliance: Better at following output format instructions
3. Refusal ability: Can decline harmful or ambiguous requests
4. Concise vs verbose: Base models tend to continue generating; instruction models provide cleaner responses
5. System prompt respect: RLHF models respect system role messages (base models don't have this concept)

This is why instruction-tuned models are preferred for deployment — they're more robust to poor prompt engineering.

7. 🔗 Cross-References

• Next: RLHF & Alignment (Week 9)
• Previous: Fine-tuning Methods
• Video: BSDA5004 Week 8 transcripts Join Discord PreviousFine-tuning & PEFTNextRLHF & Alignment