RAG evaluation with RAGAS and a claim-level LLM judge (and why model choice was not the bottleneck)

How do you evaluate a RAG system?

You evaluate a RAG system in layers, because retrieval and generation fail in different ways and a single aggregate score hides which one broke. The harness I run has six parts: a versioned golden set of questions with reference answers and the source chunks that answer them; retrieval metrics that need no LLM at all; RAGAS metrics for faithfulness, answer relevancy and context precision and recall; a claim-level LLM-as-a-judge that splits each answer into atomic claims and verifies every claim against the retrieved sources; grounding filters that reuse that same judge at generation time; and golden-file tests that turn the thresholds into acceptance criteria the build has to pass.

The order matters. Retrieval is measured first because it is cheap, deterministic and upstream of everything else. A generator cannot be faithful to a chunk that was never retrieved, so if context recall is poor there is no point tuning prompts. Only once retrieval is stable do I spend LLM budget on answer-level judgments. This is the same layering I apply on LLM evaluation engagements, and it is the reason the pipeline in the Jozveh-AI case study has an evaluation stage sitting between generation and the final document rather than a one-off benchmark run before launch.

How do you build a golden set that stays honest?

A golden set is a small, versioned collection of questions, each paired with a reference answer and the identifiers of the source chunks that justify it. It lives in the repository next to the tests, and every change to it goes through review like code. Small is deliberate: a set I can read end to end in an afternoon stays honest; a set nobody reads drifts into whatever the pipeline happened to output last month.

Three rules keep it useful. First, questions come from real usage, not from the engineer imagining what users might ask. Logs, support threads and the questions the domain expert was actually asked are the source. Second, the reference answer is written by a person against the sources, never copied from the pipeline. If the model writes its own reference the benchmark measures agreement with yesterday's model, which is the one thing you do not want to reward. Third, the set is stratified on purpose: single-chunk lookups, multi-hop questions that need two or more chunks, questions whose correct answer is "the sources do not say", and questions in the awkward formats the corpus contains (tables, lists, definitions split across pages). The no-answer cases are the ones most RAG systems fail, and they are almost never in a golden set unless someone puts them there.

Anthropic's evaluation guide gives the same advice from a different angle: design evals that "mirror your real-world task distribution", structure them so they can be graded automatically, and prefer more questions with slightly noisier automated grading over a handful of hand-graded ones (see the success criteria and evaluations guide). I agree with the volume point, with one caveat: the retrieval labels (which chunks answer which question) are cheap to produce and must be exact, so I hand-label those and let the LLM-graded metrics absorb the noise elsewhere.

Which retrieval metrics matter?

With chunk identifiers in the golden set, retrieval evaluation is plain information retrieval and needs no LLM. Recall at k tells you whether the answer-bearing chunks are in the top k at all; precision at k tells you how much of the context window is being spent on noise; a rank-aware metric (mean reciprocal rank, or the RAGAS weighted precision described below) tells you whether the right chunk arrives early enough to survive truncation and to influence the generator. I run these on every commit that touches chunking, embeddings, the index, the reranker or the query rewriter, because they are fast and deterministic.

RAGAS offers deterministic variants for exactly this situation. Its context precision and context recall pages document an ID-based mode, where the score is the fraction of retrieved context IDs found in the reference IDs (precision) and the fraction of reference IDs found in the retrieved IDs (recall), and a non-LLM mode that compares text with string distance. I use the ID-based mode in CI and reserve the LLM-based versions for when reference chunk IDs are missing. The retrieval architecture itself is a separate decision; I compare the two main options in GraphRAG versus vector RAG, and the point of a retrieval metric layer is that you can make that decision with numbers from your own corpus rather than from a vendor benchmark.

What do the RAGAS metrics actually measure?

RAGAS is an open-source evaluation library with a set of named metrics for RAG. Its documentation is precise about what each metric computes, and it is worth reading the definitions rather than trusting the metric names, because "faithfulness" and "relevancy" mean specific things here. The four I rely on are these, quoted from the official pages.

• Faithfulness "measures how factually consistent a response is with the retrieved context", from 0 to 1. The score is the number of claims in the response that can be inferred from the retrieved context divided by the total number of claims. That decomposition into claims is the same idea as the custom judge below (faithfulness docs).
• Answer relevancy "measures how relevant a response is to the user input". It is computed by generating a few artificial questions from the response, embedding them, and averaging the cosine similarity between each generated question and the original user input. It punishes incomplete or off-topic answers, not incorrect ones (answer relevancy docs).
• Context precision "evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones". It is a rank-weighted precision over the top K retrieved chunks, available with an LLM judging relevance against a reference answer or against the response, and without an LLM using IDs or string distance (context precision docs).
• Context recall "measures how many of the relevant documents (or pieces of information) were successfully retrieved". The LLM-based variant is the fraction of claims in the reference answer that the retrieved context supports (context recall docs).

The RAGAS metrics overview also states the trade-off plainly: LLM-based metrics are closer to human judgment but "can be somewhat non-deterministic", while non-LLM metrics are deterministic with lower correlation to humans (metrics overview). In practice that means pinning the evaluator model, running it at low temperature, and measuring the run-to-run variance on a subset before you set any threshold, so that a test failure means the pipeline changed and not that the judge had a different day.

Running the RAGAS metrics over a golden set is a short script. The dataset schema uses the field names user_input, retrieved_contexts, response and reference, and evaluate() takes the dataset, the metric instances and a wrapped evaluator LLM, as shown in the RAGAS getting-started guide:

from ragas import EvaluationDataset, evaluate
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import Faithfulness, LLMContextRecall, LLMContextPrecisionWithReference

rows = []
for case in golden_cases:                      # your versioned golden set
    out = pipeline.run(case["question"])       # returns answer + retrieved chunks
    rows.append({
        "user_input": case["question"],
        "retrieved_contexts": [c.text for c in out.chunks],
        "response": out.answer,
        "reference": case["reference_answer"],
    })

dataset = EvaluationDataset.from_list(rows)
evaluator_llm = LangchainLLMWrapper(judge_chat_model)   # not the generator model

result = evaluate(
    dataset=dataset,
    metrics=[Faithfulness(), LLMContextRecall(), LLMContextPrecisionWithReference()],
    llm=evaluator_llm,
)
print(result)   # aggregate scores per metric; log them next to the commit hash and run id

Why a claim-level LLM judge, not a single score?

A holistic 1-to-5 score is easy to produce and almost useless to act on. It does not say which sentence is wrong, it cannot be used as a runtime filter, it makes a poor test failure message, and it is exactly the kind of judgment where known judge biases do the most damage. A claim-level judge fixes all four. The design has three steps: split the answer into atomic, self-contained claims; verify each claim independently against the retrieved sources and return a label with a quoted evidence span; aggregate the labels into a score and a list of offending claims. RAGAS faithfulness is built on the same decomposition, and I still write my own judge, because I need control over claim granularity, over the language of the prompts (Persian in my case), over the evidence-span format that downstream filters consume, and because the same function has to run inside the pipeline, not only in the evaluation script.

from dataclasses import dataclass

LABELS = ("supported", "contradicted", "unsupported")

@dataclass
class Verdict:
    claim: str
    label: str      # one of LABELS
    evidence: str   # verbatim span from the sources, empty unless "supported"

def extract_claims(answer: str) -> list[str]:
    # LLM call. Prompt shape: "Rewrite this answer as a list of atomic factual
    # claims. Each claim must stand alone, name its subject, and contain one fact.
    # Do not add, merge or interpret. Output JSON list of strings."
    ...

def judge_claim(claim: str, sources: list[str]) -> Verdict:
    # LLM call with a different model than the generator. Prompt shape:
    # "Sources: <numbered chunks>. Claim: <claim>. Is the claim supported by the
    # sources, contradicted by them, or not addressed? Quote the exact span that
    # supports or contradicts it. If you cannot quote a span, the label is
    # 'unsupported'. Output JSON: {label, evidence, source_index}."
    ...

def faithfulness(answer: str, sources: list[str]) -> tuple[float, list[Verdict]]:
    claims = extract_claims(answer)
    if not claims:                        # an empty answer is not a faithful answer
        return 0.0, []
    verdicts = [judge_claim(c, sources) for c in claims]
    supported = sum(v.label == "supported" for v in verdicts)
    return supported / len(verdicts), verdicts

Two design choices carry most of the weight. The default label is unsupported, and the judge has to quote an evidence span to escape it; a judge that can say "supported" without pointing at text will say it too often. And the labels are three-way, because a contradicted claim and an unaddressed claim call for different runtime actions: the first is a bug to surface, the second is usually a sign the generator drifted into its own knowledge.

Known judge biases and what I do about them

The MT-Bench paper by Zheng et al. is still the reference for why an LLM judge cannot be trusted blindly. It names position bias, verbosity bias, self-enhancement bias and limited reasoning ability as the main limitations, while also finding that strong judges reach agreement with humans at roughly the level humans reach with each other (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena). The claim-level design sidesteps position bias, because there is no pairwise comparison and each claim is judged alone. Verbosity bias is reduced because the unit is one short claim, not a long answer that reads as thorough. Self-enhancement bias is handled by using a different model as judge than as generator, which Anthropic's evaluation guide also recommends as general practice. Limited reasoning is the one you cannot design away, so I calibrate: a subset of the golden set carries human claim labels, and the judge's agreement with those labels is itself a tracked number that has to hold before the judge is trusted in CI.

Grounding filters at generation time

Once the judge exists as a function, the obvious move is to call it inside the pipeline, after generation and before the answer is shown or written to a document. This is what I mean by a grounding filter. The generator produces a draft with citations; the judge runs over the draft's claims; then a policy decides what to do with each label. Contradicted claims are removed and logged. Unsupported claims are either dropped, rewritten with a constrained "using only these sources" prompt, or, if too much of the answer is unsupported, the whole answer is replaced by an explicit statement that the sources do not cover the question. Supported claims keep their evidence span, which becomes the citation the reader sees.

The cost concern is real and manageable. The filter runs once on the final draft, not on every intermediate step; extraction and verification are batched per answer; and verdicts are cached by (claim, source set) so a regenerated answer with mostly identical claims does not pay twice. The effect on the evaluation harness is the interesting part: the same metric that is measured offline is enforced online, so the offline score is not a proxy for anything, it is the production behavior measured on a fixed sample. When I design a pipeline under a RAG development engagement, the filter is part of the architecture from the start rather than a patch after the first hallucination complaint.

Golden-file tests as acceptance criteria

Everything above only changes behavior if it can fail a build. The pattern is a golden-file test: one parametrized test per golden case, each case carrying its own thresholds, and the judge's offending claims used as the assertion message so a red test tells you what to read. OpenAI's evals guide frames the same structure as a data source plus testing criteria, with graders that compare outputs against reference values (working with evals); whether you use a hosted evals product or pytest, the shape is identical.

import json
from pathlib import Path
import pytest

CASES = json.loads(Path("evals/golden.json").read_text())

@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_answer_is_grounded(case, pipeline):
    out = pipeline.run(case["question"])

    # retrieval layer: deterministic, no LLM
    retrieved_ids = {c.id for c in out.chunks}
    assert set(case["reference_chunk_ids"]) <= retrieved_ids

    # answer layer: claim-level judge, thresholds live in the case
    score, verdicts = faithfulness(out.answer, [c.text for c in out.chunks])
    bad = [v for v in verdicts if v.label != "supported"]
    assert score >= case["min_faithfulness"], bad

    if case.get("expect_refusal"):
        assert out.refused, out.answer

Per-case thresholds matter because a multi-hop question and a lookup question should not be held to the same bar on day one; a global threshold gets set to the weakest case and stops catching regressions everywhere else. Thresholds only move up, and they move in the same commit as the change that earned the improvement. For LLM-graded assertions I run the judge a small fixed number of times and take the majority label, which is cheaper than the variance it removes. In client work these tests are the acceptance criteria written into the proposal: the sprint is done when the golden-file suite passes at the agreed thresholds, not when someone feels the answers look better.

The model bake-off: why model choice was not the bottleneck

A bake-off is a controlled comparison: the same golden set, the same retriever, the same prompts, the same judge and the same thresholds, with exactly one variable swapped per run. Swap the generator model and hold everything else; then swap the retriever configuration and hold the model. Log every run with its commit hash, the model identifiers, the judge identifier and the per-row scores, so two runs can be diffed row by row rather than compared by their averages. Without that discipline a bake-off measures prompt drift and judge variance as much as it measures models.

The result I keep seeing, and the one the Jozveh-AI harness showed clearly, is that model choice was not the bottleneck. Swapping generators moved faithfulness less than improving retrieval quality, adding the grounding filter, and closing the evaluation loop so that failures were visible and fixable. This is not an argument that models are interchangeable. It is an argument about where the marginal hour goes: once retrieval delivers the right chunks and the filter removes what the sources do not support, the difference between capable models on a grounded task is smaller than the difference between a pipeline with an evaluation loop and one without. The bake-off is how you find that out for your own corpus rather than taking my word for it, and it doubles as the evidence you need when a stakeholder asks for the most expensive model by default.

What fails first, in my experience

• Chunking that cuts a definition, a table or a numbered procedure in half, so the answer-bearing text exists in the corpus but never appears whole in any chunk. Context recall looks fine on chunk IDs and faithfulness collapses anyway.
• Near-duplicate chunks filling the top K, so precision is low while recall is high and the generator sees three copies of the same paragraph instead of the second source it needed.
• Answers from parametric memory when retrieval returns nothing useful. The answer reads well, the judge marks every claim unsupported, and without a no-answer case in the golden set nobody notices.
• Citations that point at the right document and the wrong passage. Only an evidence-span requirement in the judge catches this; a document-level citation check passes.
• A judge that agrees with confident prose. This shows up as a widening gap between the judge's labels and the human-labeled calibration subset, which is why that gap is tracked.
• Golden set contamination: someone regenerates the reference answers with the pipeline to save time, and every subsequent score is a measure of self-agreement.
• Non-English corpora, Persian in my case, where embedding quality, tokenization of right-to-left text and claim extraction all behave differently from the English demos the tooling was built on. Every metric needs its own sanity check in the target language.

Most of these are visible in an afternoon with the harness described here, which is why an AI technical audit starts by building or reading the golden set before touching any prompt.

What I observed in Jozveh-AI

RAG evaluation checklist

• A versioned golden set in the repository, reviewed like code, with questions drawn from real usage.
• Reference answers written by a person against the sources, never generated by the pipeline.
• Reference chunk IDs for every case so retrieval can be scored without an LLM.
• No-answer cases whose correct behavior is an explicit refusal.
• Multi-hop cases that need two or more chunks, and cases in the corpus's awkward formats (tables, lists, split definitions).
• Recall@k, precision@k and a rank-aware metric on every change to chunking, embeddings, index, reranker or query rewriting.
• RAGAS faithfulness, answer relevancy, context precision and context recall with a pinned evaluator model and a measured run-to-run variance.
• A claim-level judge with three-way labels, a mandatory evidence span, and a default of "unsupported".
• A judge model different from the generator model.
• A human-labeled calibration subset and a tracked judge-versus-human agreement number.
• The same judge reused as a grounding filter at generation time, with a policy for contradicted and unsupported claims.
• Golden-file tests with per-case thresholds that only move up, and offending claims as the failure message.
• Every evaluation run logged with commit hash, model identifiers, judge identifier and per-row scores.
• A bake-off procedure that changes one variable per run before anyone argues about which model to buy.

Further reading

• RAGAS: Faithfulness – the claim-decomposition definition and the Faithfulness class.
• RAGAS: Answer relevancy – generated-question cosine similarity, and why a score can fall outside 0 to 1.
• RAGAS: Context precision and RAGAS: Context recall – LLM, non-LLM and ID-based variants.
• RAGAS: Evaluate a simple RAG system – the EvaluationDataset schema and the evaluate() call.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena – the biases every judge design has to answer for.
• Anthropic: Define success criteria and build evaluations – task-specific, automated, volume-first eval design and LLM-graded rubrics.
• OpenAI: Working with evals – data source plus testing criteria as the structure of an eval.
• Related on this site: LangGraph production architecture, where the evaluation stage sits inside the graph, and LLM evaluation services for building this harness on your system.