Followup Router Prompt

The Followup Router uses an LLM with a specialized prompt to classify how user followups should be handled. This document explains the prompt architecture, semantic reasoning approach, and deterministic fallback system.

Purpose

The followup router answers: "How should we handle this followup message?"

Followup Kind	Example	Behavior
`pure_show_more`	"Näita veel"	Paginate same results
`soft_refinement`	"Odavamaid"	Apply mild constraint
`context_addition`	"See on emale"	Add recipient/occasion context
`new_constraint`	"Mitte raamatuid"	Add exclusion constraint
`hard_pivot`	"Do you have books"	Switch product type
`question_about_shown`	"Mis on see toode?"	Ask about shown product

Architecture

Location: app/api/chat/orchestrators/context-orchestrator/followup-router.ts

The Core Problem: Semantic Similarity

Why This Matters

The system initially failed to understand that these queries are semantically equivalent:

Query	Expected	Was Misclassified As
"Do you have books"	`hard_pivot`	❌ `question_about_shown`
"Do you have book recommendations"	`hard_pivot`	✅ `hard_pivot`
"Kas teil on raamatuid?"	`hard_pivot`	❌ `question_about_shown`

Root Cause: The LLM saw "Do you have" as a question pattern and associated it with asking about shown products, rather than requesting a product type change.

Semantic Reasoning Guide

The prompt now includes a reasoning guide that teaches the LLM how to think about classification:

SEMANTIC REASONING GUIDE (apply BEFORE classifying):

1. Does the message reference a SPECIFIC SHOWN product?
   - Look for: ordinals (first, second, esimene, teine), 
               pronouns (this, that, see, selle, seda)
   - If YES → likely question_about_shown
   - If NO → NOT question_about_shown

2. Does the message ask about a PRODUCT TYPE or CATEGORY?
   - Product types: books/raamat, games/mäng, toys, electronics/tehnika
   - Patterns: "do you have X", "show me X", "what about X"
   - If asking about product TYPE without referencing shown product → hard_pivot

3. Key insight: "Do you have books" and "Do you have book recommendations" 
   are SEMANTICALLY EQUIVALENT.
   Both mean: "Switch to showing book products" → hard_pivot

Why Semantic Reasoning Works

Prompt Structure

Followup Kinds with Examples

Followup kinds:
- pure_show_more: user just wants more of same results.
- soft_refinement: mild tweaks (cheaper, more expensive) without big shift.
- context_addition: user adds context info WITHOUT a new searchable product query.
  * "See on emale" (This is for mother) → context_addition
  * "Tema on 55-aastane" (They are 55) → context_addition
  
- new_constraint: adds a constraint but keeps main intent.
- hard_pivot: switches product type/category/intent. Examples:
  * "Do you have books" → hard_pivot
  * "Kas teil on raamatuid?" → hard_pivot (Estonian)
  * "Show me games instead" → hard_pivot
  * "What about toys?" → hard_pivot
  
- question_about_shown: asks about SPECIFIC already shown product(s).
  REQUIRES ordinal or pronoun reference to a SHOWN product.

Critical Clarifications

The prompt includes critical rules to prevent common misclassifications:

CRITICAL: "Do you have X" patterns are NOT question_about_shown!
- "Do you have books" → hard_pivot (asks for a PRODUCT TYPE)
- "Kas teil on raamatuid?" → hard_pivot (same in Estonian)
- question_about_shown REQUIRES: ordinal (first/second) OR pronoun (this/that/see)
- If NO ordinal/pronoun and asking about PRODUCT TYPE → hard_pivot

Output Structure

The LLM outputs a JSON response with reasoning:

{
  "reasoning": "User asks 'do you have books' - no pronouns or ordinals referencing shown products. 'Books' is a product type request, not asking about a specific shown product.",
  "followup_kind": "hard_pivot",
  "preserve": {
    "context": true,
    "excludes": true,
    "categoryHints": false,
    "productType": false
  },
  "needs_reextract": true,
  "context_switch": true,
  "context_switch_reason": "Product type change from games to books",
  "confidence": 0.92,
  "productInquiryMatch": { "productId": null, "productName": null }
}

Preservation Flags

Flag	When `true`	When `false`
`preserve.context`	Keep recipient/occasion	Reset all context
`preserve.excludes`	Keep exclude list	Clear excludes
`preserve.categoryHints`	Keep category hints	Clear hints
`preserve.productType`	Keep product type	Allow type change

Deterministic Corrections

Even with improved prompts, LLMs can misclassify. The system includes deterministic pattern matching as a safety net:

All 12 Product Types Covered

const PRODUCT_TYPE_KEYWORDS = [
  // Raamat (Books)
  '(?:book|books|raamat|raamatuid|novel)',
  // Mängud (Games)
  '(?:game|games|mäng|mängud|toy|puzzle)',
  // Kinkekaart (Gift Cards)
  '(?:gift\\s*card|kinkekaart|voucher)',
  // Kingitused (Gifts)
  '(?:gift|gifts|kingitus|present)',
  // Kodu ja aed (Home & Garden)
  '(?:home|garden|kodu|aed|candle|küünal)',
  // Kontorikaup (Office Supplies)
  '(?:office|stationery|kontori|notebook|märkmik)',
  // Ilu ja stiil (Beauty & Style)
  '(?:beauty|cosmetic|kosmeetika|parfüüm|jewelry)',
  // Tehnika (Electronics)
  '(?:electronic|electronics|tehnika|gadget|e-reader)',
  // Film (Movies)
  '(?:movie|movies|film|dvd)',
  // Muusika (Music)
  '(?:music|muusika|album|vinyl)',
  // Joodav ja söödav (Food & Drink)
  '(?:food|drink|toit|chocolate|tea|coffee)',
  // Sport ja harrastused (Sports & Hobbies)
  '(?:sport|sports|fitness|hobby)',
];

Request Patterns

const PRODUCT_TYPE_PIVOT_PATTERNS = [
  // "Do you have X" patterns (English)
  /\b(?:do\s+you\s+have|have\s+you\s+got|got\s+any)\s+PRODUCT_TYPE/i,
  
  // "Show me X" / "I want X" patterns
  /\b(?:show\s+me|i\s+want|looking\s+for)\s+PRODUCT_TYPE/i,
  
  // "What about X" patterns
  /\b(?:what\s+about|how\s+about|instead)\s+PRODUCT_TYPE/i,
  
  // Estonian: "Kas teil on X"
  /\bkas\s+teil\s+on\s+PRODUCT_TYPE/i,
  
  // Estonian: "Näita mulle X" / "Otsin X"
  /\b(?:näita\s+mulle|otsin|tahan)\s+PRODUCT_TYPE/i,
  
  // Direct requests: "Books?" or "Games please"
  /^PRODUCT_TYPE(?:\s+please)?[?!.]?$/i,
];

Correction Logic

Complete Example Flow

Scenario: Gift for 10-year-old → Books

Previously Broken (Now Fixed)

Configuration

Model Selection

const ROUTER_MODEL = 'meta-llama/llama-4-scout-17b-16e-instruct';

Why Llama 4 Scout 17B?

Fast inference (~100-200ms)
Good classification accuracy
Low cost via Groq

Confidence Threshold

export const FOLLOWUP_ACTIONABLE_THRESHOLD = 0.35;

Classifications below this threshold are treated as uncertain and may trigger fallback behavior.

Environment Variables

# Disable followup router entirely
ENABLE_FOLLOWUP_ROUTER=false

# Enable debug logging
CHAT_DEBUG_LOGS=true

Debugging

Log Output

When CHAT_DEBUG_LOGS=true:

🔬 FOLLOWUP_ROUTER: Entry point {
  userMessage: 'Do you have books',
  hasContextToRestore: true,
  shouldRunRouter: true
}

🔬 FOLLOWUP_ROUTER: LLM result {
  status: 'applied',
  followupKind: 'hard_pivot'
}

⚠️ FOLLOWUP ROUTER CORRECTION: Overriding question_about_shown → hard_pivot {
  userMessage: 'Do you have books',
  pattern: 'Do you have [product type] detected'
}

Common Issues

Symptom	Cause	Solution
"I don't have info about this product"	Misclassified as `question_about_shown`	Deterministic patterns correct this
Context lost on followup	`preserve.context: false`	Check LLM reasoning for why
Product type not switching	`preserve.productType: true`	Ensure `hard_pivot` classification

Performance

Latency Breakdown

Component	Time
Prompt building	~2ms
LLM inference	100-200ms
Response parsing	~3ms
Pattern correction	`<1ms`
Total	~150-200ms

Prompt Architecture

System Prompts Overview - Overall prompt strategy
Estonian Prompt - Estonian language rules
Response Validation - Anti-hallucination checks

Conversational Systems

Followup Capabilities - Multi-turn conversation handling
Refinement Detection - Dual-path refinement system
Memory Resolution - Conversation memory

Context Systems

Intent Classification - Dual-path intent detection
Context Preservation - Context building

Key Takeaways

Semantic reasoning in the prompt teaches the LLM how to think, not just what to classify
Explicit examples for each classification type prevent bias toward over-documented categories
Critical clarifications address common misclassification patterns directly
Deterministic corrections catch any remaining LLM errors for known patterns
Defense in depth ensures robust handling even when LLM makes mistakes

Last Updated: November 2025
Version: 2.0 - Semantic Reasoning Enhancement
Status: ✅ Production Ready

Purpose​

Architecture​

The Core Problem: Semantic Similarity​

Why This Matters​

Semantic Reasoning Guide​

Why Semantic Reasoning Works​

Prompt Structure​

Followup Kinds with Examples​

Critical Clarifications​

Output Structure​

Preservation Flags​

Deterministic Corrections​

All 12 Product Types Covered​

Request Patterns​

Correction Logic​

Complete Example Flow​

Scenario: Gift for 10-year-old → Books​

Previously Broken (Now Fixed)​

Configuration​

Model Selection​

Confidence Threshold​

Environment Variables​

Debugging​

Log Output​

Common Issues​

Performance​

Latency Breakdown​

Related Documentation​

Prompt Architecture​

Conversational Systems​

Context Systems​

Key Takeaways​