Evals / where quality becomes a number

Assertions and LLM-as-Judge scoring make output quality a number rather than a feeling. Without evals, the hallucination ships — regressions are invisible until a user finds them.

Architectural Position

Before

fn(query_user, prompt_summarise, context, state, memory, tools) → response

After

((..., , ...)) →

eval: The evaluation function — scores outputs against defined criteria
fn: Any fn call in the pipeline — rewrite, respond, and remember can all be evaluated
prompt_evaluate: The system prompt for judging — instructs the evaluator fn to score output on fidelity, relevance, and tone
score: A measurement — assertion pass/fail, or LLM-as-Judge 1–5 rating

Step by Step

# The failure from Layer 7

Query: "best camera phone under ₹50k"
Response: "The Pixel 9a has a 64MP camera,
           making it the top choice..."

Context (phones.json):
  rear_camera_mp: 48

# 64 ≠ 48
# The hallucination was confident
# No assertion caught it

A response that looks correct can be wrong. Without evals, the only way to find this is a user complaint.

1 / 5

Implementation

What you build

Assertions on hard failures: len(results) > 0, price within filter, expected model name in response
LLM-as-Judge scoring respond output on fidelity, relevance, and tone (1–5 scale)
eval runs on every prompt change before shipping

What it fixes

Failures are visible and measurable. Output quality can be tracked over time. Regressions surface before reaching users.

Key Concepts

Eval: A test that scores an LLM output against a criterion. Can be automated (exact match, regex, LLM-as-judge) or human-reviewed.
LLM-as-judge: Using a second LLM call to score the output of the first. Scales cheaply but inherits the biases of the judging model.
Fidelity: Whether the response accurately reflects the injected context — no hallucinated specs, no invented phone features.
Golden dataset: A curated set of representative queries with verified expected outputs. The ground truth against which all prompt changes are measured.

The Failure

The Blindness Law
You can't improve what you can't measure cheaply at scale.

The response looks correct. The model said the Pixel 9a has a 64MP camera. The context says 48MP. No assertion caught it. The tone was confident. The mistake was subtle. Without an LLM-as-Judge fidelity check scoring this 2/5, the hallucination ships to every user who asks about camera specs. Evals are not a final step — they are the mechanism that tells you whether your pipeline can be trusted.

What it reveals

Nothing — this is the last layer. What you build next are product decisions.

Implication

Ability LLM can judge its own output. LLM-as-Judge is itself an fn call. The Blindness Law applies until evals are in place.

Behaviour Eval rubrics define what "good" means. Writing the rubric is a product decision — it encodes your quality standards.

Code Eval pipeline is a separate system, not a test suite. It runs continuously in production, not just at build time.

Design No visible change for the user. But trust in the product increases. Evals are the foundation of confident iteration.

Economics Eval cost is real. LLM-as-Judge = tokens per evaluation. But the cost of shipping broken output is higher.