Context / answers rooted in what you know
Injecting the top retrieved phones as context grounds the model in your data — hallucinated specs disappear. A new failure surfaces immediately: the pipeline forgets everything the moment the turn ends.
Architectural Position
- fn
- The LLM call — rewrite uses prompt_rewrite, respond uses prompt_summarise
- query_user
- Natural language input from the user
- prompt_rewrite
- The system prompt for rewriting — unchanged from Layer 3
- prompt_summarise
- The system prompt for responding — instructs fn to generate a grounded recommendation from the injected context
- query_structured
- Schema-validated rewrite from the first fn call
- context
- Top 3 phone records retrieved and injected into the second fn call
- response
- A grounded completion — rooted in your actual phones.json
Step by Step
# The failure from Layer 3
rewrite working. Moto G84 returned.
User: "Does it have wireless charging?"
Model answers from training weights.
Wrong answer possible.
# Fix: inject phone records into respond
# Model reasons over your data, not memorySchema gave us reliable shape. Context gives us reliable facts — by injecting the actual phone records before respond generates the answer.
Implementation
What you build
rewrite: query rewriting with schema — returnsquery_structuredretrieve: semantic search → top 3 phones fetched ascontextrespond: LLM generates grounded recommendation using the 3 phone records as context
What it fixes
Model answers from your phones.json. Specs are accurate. No hallucination on facts present in context.
Key Concepts
- RAG
- Retrieval-Augmented Generation. A pattern that retrieves relevant documents from a knowledge base and injects them into the prompt before calling the model.
- Grounding
- Constraining the model's output to facts that appear in the injected context rather than drawing from training data.
- Context window
- The maximum number of tokens a model can process in a single call. Injected context, conversation history, and system prompt all compete for this budget.
The Failure
The Haystack LawFilling the window ≠ the model reading it.
Turn 1: "show me phones for my mom" → Moto G84, Samsung A35, Pixel 9a displayed. Turn 2: "what about the second one?" The pipeline runs blind. rewrite treats the query as a fresh search. The Samsung A35 is gone from context. The model returns wrong results. The pipeline is stateless — each turn starts from scratch.
What it reveals
Each turn is stateless. Follow-up questions — "what about the second one?" — fail because the pipeline has no memory of the previous turn.