Evaluating LLM outputs: the evaluation pyramid

Why "it looks good" is not a test

Language models are non-deterministic, sensitive to small prompt changes and prone to failing on inputs nobody tried in the demo. A team that ships on the strength of a dozen hand-checked examples is not testing; it is hoping. The next model release, the next prompt tweak or the next unusual customer will find the gap.

Evaluation is the discipline that replaces hope with numbers. It is also the single biggest difference we see between AI projects that reach production and those that stall in pilot. Teams with an evaluation set can change things safely and argue from evidence. Teams without one argue from anecdotes, and every change is a risk.

The good news is that evaluation does not have to be expensive or academic. Most of it is cheap and automatable. The pyramid below is how we structure it.

The evaluation pyramid

Layer 1: deterministic checks

Code, not models. Does the output parse as the JSON schema you asked for? Is it within length limits? Does it contain banned phrases, personal data patterns or markdown where plain text was required? Did it return within the latency budget? These checks cost nothing, run on every request in production and catch a surprising share of failures. They are also where guardrails begin.

Layer 2: golden sets

A curated set of inputs with known-good outputs or clearly defined acceptance criteria. For classification and extraction, exact or near-exact matching works. For free text, you compare against reference answers using the layers above. Run the full set on every change to prompt, model, retrieval or code, and track the score over time. This is your regression suite.

Layer 3: model-graded rubrics

For qualities that code cannot judge, such as "is this answer faithful to the source" or "is this reply appropriately cautious", use a model as the grader. Give it a written rubric with explicit criteria and a scoring scale, and ask for a short justification alongside the score. Model grading scales to thousands of cases, which makes it the workhorse of the middle of the pyramid.

Layer 4: human review

Domain experts reviewing a sample each week, scoring against the same rubric. Humans are expensive and slow, so their job is not volume. Their job is to catch failure types nobody anticipated and to calibrate the model grader: if the judge and the experts disagree, the rubric or the judge needs work.

Layer 5: production monitoring

Everything above happens before a change ships. Monitoring happens after. Sample live traffic, run it through layers one and three, collect explicit feedback where you can, and watch for drift in scores, topics and refusal rates. New failures found here become new golden set cases. We set this up as part of LLMOps and monitoring.

What to measure

"Quality" is too vague to score. Break it into dimensions, each with a method and an owner.

Report the dimensions separately. A single blended score hides the fact that correctness went up while faithfulness went down, which is exactly what happens when a model starts answering from memory instead of the retrieved context, the root of most production hallucination.

Evaluating retrieval separately

If your system retrieves documents before answering, you have two systems to evaluate, and the first one is usually the problem. For each golden question, record which passages should be retrieved. Measure how often they appear in the top results. Only then measure the answer, given the correct passages.

Separating the two tells you where to spend effort. If retrieval misses the passage, no prompt change will help; fix chunking, embeddings or hybrid search. If retrieval is fine and the answer is still wrong, work on the prompt or the model. RAG explained walks through both halves.

Using a model as a judge, carefully

Model grading is powerful and easy to misuse. A few rules keep it honest.

• Write the rubric down. "Rate helpfulness 1 to 5" produces noise. "Score 5 only if every user question is answered with a cited source" produces a signal.
• Prefer pairwise comparisons for subjective qualities. "Which of these two is better, and why" is more reliable than absolute scores.
• Calibrate against humans. Have experts label a sample, measure agreement with the judge, and fix the rubric until they agree.
• Use a different model from the one being judged where you can, and shuffle the order of candidates to counter position bias.
• Watch for length bias. Judges tend to prefer longer answers. Penalise padding explicitly in the rubric.
• Set temperature low for the judge so scores are repeatable.

Building a golden set that earns its keep

The best cases come from real usage: support tickets, search logs, the questions your pilot users actually asked. Add cases from subject-matter experts for coverage of things users have not asked yet. Include deliberately hard material: ambiguous requests, questions with no answer in the data, inputs that should be refused, and adversarial prompts from your red-teaming work.

Version the set alongside the code. Review it when the product changes. Retire cases that no longer apply and add every production failure as a new case. A golden set that is not maintained becomes a false comfort within months.

Pitfalls we see

Wiring evaluation into delivery

Treat the evaluation suite like a test suite. It runs automatically on every change to prompts, retrieval, model version or orchestration code. Set thresholds per dimension and block releases that fall below them. Keep the run cheap enough that nobody is tempted to skip it; a few hundred cases through layers one to three usually costs far less than one engineer-hour.

In production, sample traffic continuously and feed results back. A weekly review of the lowest-scoring samples with a domain expert is one of the highest-value hours in an AI programme. It is also the point where security and quality meet: the same harness runs the adversarial cases from the LLM security checklist, and the same discipline decides whether you are ready to climb the ladder in prompt engineering vs fine-tuning.

Key takeaways

• Hand-checking a dozen outputs is not evaluation; a maintained set with numbers is.
• Cheap deterministic checks run on everything; expensive human review runs on samples.
• Score correctness, faithfulness, completeness, format, safety and cost separately.
• Evaluate retrieval before generation; most RAG failures are retrieval failures.
• Model judges need a written rubric, human calibration and bias controls.
• Run the suite on every change and turn every production failure into a new case.

Related services