State / the thread that holds a conversation

Adding conversation history to every call makes follow-up questions work. `state` grows unbounded with every turn — and vanishes completely when the session ends.

Architectural Position

Before

fn(query_user, prompt_rewrite, schema) → query_structured

search(query_structured) → context

fn(query_user, prompt_summarise, context) → response

After

(, prompt_rewrite, schema, ) → query_structured

search(query_structured) →

(, prompt_summarise, , ) →

fn: The LLM call, now receiving conversation history alongside query and context
query_user: The current user message
state: The conversation history — all prior turns in this session
context: Retrieved phone data injected for this turn
response: A contextually coherent response — aware of what was said before

Step by Step

# The failure from Layer 4

Turn 1: "show me phones for my mom"
→ Moto G84, Samsung A35, Pixel 9a

Turn 2: "what about the second one?"
→ Pipeline runs blind
→ Wrong phones returned

# Fix: pass conversation history to rewrite and respond
# fn(query_user, prompt_rewrite, schema, state) → query_structured
# fn(query_user, prompt_summarise, context, state) → response

Without state, each turn is a fresh conversation. Follow-up questions lose all reference to prior results.

1 / 5

Implementation

What you build

A state message array that appends each turn (user + assistant)
rewrite and respond both receive the full state on every call
UI: threaded summary where each turn appends below the previous

What it fixes

Follow-up questions resolve correctly. "What about the second one?" works because the prior turn is in the message array.

Key Concepts

Message history: The accumulated list of prior turns (user messages and assistant responses) in the current session. Passed to the model as part of every prompt.
Session: A bounded conversation — typically tied to a browser session or user login. State lives within a session; Memory persists across them.
Sliding window: Keeping only the last N turns of conversation history, discarding earlier turns to stay within the context budget.
Summarisation buffer: Compressing older turns into a rolling summary before they are dropped. Preserves meaning at the cost of precision.

The Failure

The Stateless Law
Models are stateless by design. Everything they know about you lives in the window.

Turn 1 payload: [system, user_1, assistant_1] — ~800 tokens. Turn 3 payload: [system, user_1, assistant_1, user_2, assistant_2, user_3] — ~1500 tokens. By turn 10 the array is large. By turn 50 it has exceeded the context window or consumed most of the budget. And when the session ends, everything is gone. The user returns the next day and the system has no idea who they are.

What it reveals

state grows unbounded. Token costs increase every turn. Context window limits are hit at long conversations. Session end erases everything.

Implication

Ability LLM can maintain conversational coherence — but only within the context window. Coherence degrades as state grows.

Behaviour State determines what the model knows about the ongoing conversation. Truncating state changes behaviour unpredictably.

Code Message array management. Token budget grows per turn. Sliding window or summarisation logic needed at scale.

Design Threaded UI. Conversation accumulates visibly. Users begin to experience the product as a dialogue, not a search.

Economics Token cost grows with every turn. A 20-turn conversation costs significantly more than a single query. Unbounded without management.