Jozveh-AI: an eight-agent LangGraph GraphRAG pipeline with evaluation as code

Tests

673

Test files

77

Role

Designed and built the pipeline and its evaluation stack (solo engineer)

What problem did Jozveh-AI have to solve?

Jozveh-AI turns heterogeneous source material, such as lecture audio and reference text, into cited, structured Persian study documents. I built it as a LangGraph supervisor over eight specialized agents, with Neo4j GraphRAG and reranking for retrieval, source-cited generation, and an evaluation stack made of RAGAS, a claim-level LLM judge and post-generation grounding filters. The pipeline is held to 673 tests across 77 files, and those tests are the acceptance criteria I work against when anything in the chain changes.

The hard part was never producing text. Any capable model will write a plausible study document from a transcript. The problem was producing a document a student can trust: every claim traceable to a passage in the source, the structure predictable enough to study from, the output rendered as a right-to-left Persian document, and the whole run priced so that a long lecture does not become an unbounded bill. Faithfulness and predictable cost were the two requirements that shaped every decision below.

Context and constraints

Jozveh-AI is a Persian-market product, which adds constraints a typical English RAG demo never meets. The source material is inconsistent: transcribed lecture audio and written reference material in more than one format, arriving together in one job. Persian output needs RTL layout, correct handling of mixed-direction fragments (formulas, code, Latin terms), and typographic conventions that generic document generators get wrong. Retrieval also has to cope with a language that tokenizes and embeds less predictably than English.

On the engineering side I was the solo engineer on the pipeline and its evaluation, so the system had to be operable by one person. That ruled out anything that needed a human in the loop to keep quality up. Quality had to be enforced by code: evaluators, filters and tests that fail loudly rather than dashboards someone has to watch. This is the same discipline I describe in the guide on LangGraph production architecture, applied to a document pipeline instead of a chat product.

How is the pipeline structured?

The flow is linear at the top level and agentic inside each stage. Source material enters a LangGraph supervisor, which routes work across eight agents. Retrieval runs against a Neo4j knowledge graph combined with vector search, and the candidates are reranked before generation. Generation is source-cited by construction. The draft then passes through RAGAS metrics, the claim-level judge and the grounding filters before the RTL document is assembled.

The supervisor owns control flow, not content. It decides which agent runs next from the state of the job, enforces step budgets, and terminates the graph when the acceptance checks pass or the budget is spent. Each agent owns one responsibility and one slice of the shared state. That split is what made the system testable: an agent can be exercised in isolation with a fixed state and a fixed expectation, and the supervisor can be tested with stub agents. In LangGraph the routing is expressed with conditional edges or a Command returned from a node, which lets a node update state and choose the next node in one step. A sketch of the shape:

from typing import Literal
from langgraph.graph import StateGraph, END
from langgraph.types import Command

def supervisor(state: JobState) -> Command[Literal["retrieve", "write", "check", "__end__"]]:
    if state["steps"] >= state["step_budget"]:
        return Command(update={"halt_reason": "budget"}, goto=END)
    if not state["evidence"]:
        return Command(goto="retrieve")
    if state["draft"] is None or state["verdict"] == "rewrite":
        return Command(update={"steps": state["steps"] + 1}, goto="write")
    if state["verdict"] == "pass":
        return Command(goto=END)
    return Command(goto="check")

graph = StateGraph(JobState)
graph.add_node("supervisor", supervisor)
# ... retrieve / write / check nodes each return to "supervisor"
app = graph.compile()

The real graph has eight agents rather than three nodes, but the principle is identical: a deterministic routing function reads state, every agent returns to the supervisor, and the budget check runs before any model call is made.

Key design decisions

A supervisor over eight specialized agents instead of one large prompt

The first version of any pipeline like this is one prompt that takes the whole transcript and asks for a document. It works on short inputs and degrades on everything else: structure drifts, citations get invented, and there is no place to attach a check. Splitting the work into eight agents with a supervisor gave each step a narrow prompt, a narrow input and a narrow output contract. The cost is an extra routing call per hop, which the LangChain multi-agent docs acknowledge for coordinator-style designs where all routing passes through the main agent. I accepted that overhead because it bought observability and a place to enforce budgets. This is the core of what I offer as LangGraph development and AI agent development.

GraphRAG on Neo4j instead of pure vector retrieval

Study material is relational. A definition in minute twelve of a lecture is used in an example in minute forty, and a good study document has to connect them. Pure vector retrieval returns the chunks that look like the question, not the chunks that are linked to it. With Neo4j I could store entities and their relations alongside chunk embeddings and retrieve by similarity first, then expand through the graph. Neo4j's own GraphRAG package describes exactly this pattern as a vector search followed by a retrieval query over the graph. The trade-offs between the two approaches are the subject of my guide on GraphRAG versus vector RAG; the short version is that the graph earned its place here because the questions are about structure, not lookup. The service page for GraphRAG development covers when I would recommend it and when plain RAG development is enough.

Reranking before generation

Graph expansion widens the candidate set, which is the point, but a wide set is a bad prompt. A reranking stage scores candidates against the specific section being written and keeps the smallest set that still covers it. This is one of the retrieval-side changes that moved faithfulness more than prompt or model changes did, because the generator no longer had to choose among loosely related passages.

Source-cited generation by construction

The generator is not asked to "add citations". It is given evidence items with stable identifiers and is required to attach an identifier to every claim it makes. A claim without a citation is malformed output and is rejected before evaluation runs. This turns citation from a stylistic request into a structural contract that the downstream filters can verify mechanically.

Post-generation grounding filters

Even with cited generation, models will occasionally cite a passage that does not say what the claim says. The grounding filters re-open each cited passage and check that it actually supports the claim attached to it. Unsupported claims are dropped or sent back for a rewrite. The filters are deterministic where they can be (identifier exists, passage non-empty, claim not empty) and model-assisted where they must be (does this passage entail this claim).

Evaluation as code

RAGAS metrics, the judge and the filters are all invoked from the test suite, not only from the runtime. A change to a prompt, a retriever parameter or a model has to pass the same suite as a change to the parser. That is what I mean by evaluation as code, and it is the reason the model bake-off below could be run at all.

What alternatives did I consider?

• One long-context call per document. Simplest to build, cheapest to reason about, and impossible to evaluate at claim level, because there is no place between input and output to attach a check.
• Handoff-style agents without a supervisor. Each agent decides who runs next. The LangChain docs describe this as a good fit for sequential, conversational flows. For a batch document job it hid the control flow inside prompts and made budgets hard to enforce, so I centralized routing.
• Vector-only retrieval with a larger top-k. Cheaper to operate than a graph, but similarity alone does not connect a definition to the passages that use it, and those connections are what a study document is made of.
• A single "verify this document" judge call. One call per document is cheap, but a document-level verdict does not tell you which claim failed. A claim-level judge costs more tokens and produces actionable output.
• Human review before publishing. Not viable for a solo-operated product. The filters and tests replaced it.

Failure modes and how the pipeline contains them

I will describe classes of failure rather than incidents, because the useful lesson is the containment, not the anecdote.

Unfaithful claims

The generator states something the sources do not say, usually a confident generalization of a specific example. Containment: cited generation makes the claim carry an identifier, the grounding filter checks entailment against that passage, and RAGAS faithfulness is computed on the final document in the test suite so a regression shows up as a failing test rather than a complaint.

Citations pointing to the wrong passage

The claim is true and present in the sources, but the attached identifier points elsewhere. This is more common than fabrication and is invisible to a document-level judge. Containment: the claim-level judge scores each claim-citation pair separately, and the filter rejects pairs where the passage does not support the claim even when the claim is correct elsewhere in the corpus.

Agents looping

A rewrite fails a check, is rewritten, fails again. Without a budget this runs until the recursion limit, and LangGraph's default limit is generous. Containment: the supervisor keeps a step counter in state and terminates with an explicit halt reason before any model call once the budget is spent. The graph-level recursion limit stays as a backstop, not as the primary control.

Cost drift

Longer inputs, wider graph expansion or a chattier model quietly raise the per-document cost. Containment: per-call token accounting is recorded in the job state, rate limits bound concurrency against the providers, and the test suite includes budget assertions on fixed fixtures so a change that doubles token use fails a test before it reaches production.

Evaluation: RAGAS, a claim-level LLM judge and grounding filters

Three layers, each catching what the others miss. I go deeper on the methodology in the guide on RAG evaluation with RAGAS and an LLM judge; here is how the layers fit this system.

RAGAS

RAGAS provides the standard retrieval and generation metrics: context precision and recall for the retriever, response relevancy and faithfulness for the generator. The RAGAS metric list is the reference, and its faithfulness metric is computed by decomposing a response into claims and dividing the number supported by the context by the total. Those metrics gave me a common yardstick across retriever changes and model changes. They are coarse on purpose; they answer "did this get worse" rather than "which sentence is wrong".

The claim-level LLM judge

The custom judge does what RAGAS faithfulness does, but at the granularity the product needs: one verdict per claim-citation pair, with a reason, in a fixed schema. The output is parsed, not read. The prompt shape is deliberately boring:

JUDGE_PROMPT = """You are checking one claim against one source passage.
Answer only with JSON: {"supported": true|false, "reason": "<one sentence>"}.
A claim is supported only if the passage states it or it follows directly.
Do not use outside knowledge.

Claim: {claim}
Passage [{source_id}]: {passage}
"""

def judge_claim(llm, claim: str, source_id: str, passage: str) -> dict:
    raw = llm.invoke(JUDGE_PROMPT.format(claim=claim, source_id=source_id, passage=passage))
    verdict = json.loads(raw)            # schema failure is a test failure, not a warning
    assert set(verdict) == {"supported", "reason"}
    return verdict

The judge is itself under test: fixtures with known-supported and known-unsupported pairs, in Persian and in mixed Persian-English, assert that the judge's verdicts match. A judge that drifts is worse than no judge, so its own accuracy on the fixtures is part of the suite.

Grounding filters

The filters run inside the pipeline at request time, while RAGAS and most judge runs happen in the suite. They are the last gate before the document is assembled, and they are conservative: a claim that fails is removed or returned for a rewrite. A shorter document with every claim grounded beats a fuller one with a single fabricated line.

Tests as acceptance criteria

The 673 tests across 77 files cover the parsers, each agent in isolation, the supervisor's routing and budgets, the retrievers and reranker against fixed fixtures, the judge's own accuracy, the grounding filters, the RTL document assembly, and cost assertions. They are the definition of done. When I take on LLM evaluation work for a client, this is the structure I aim for: an evaluation layer that lives in the test suite and gates every change.

The model bake-off: why model choice was not the bottleneck

Because the evaluation stack was already in the suite, I could compare several models under one harness: same fixtures, same retriever, same reranker, same judge, same filters, only the generation model swapped. I am not going to publish the scores here, but the shape of the result is worth stating because it contradicts the instinct most teams have.

Model choice was not the bottleneck. Swapping the generation model moved faithfulness less than improving retrieval quality, tightening grounding, or closing the evaluation loop did. Once the generator was given a small, well-reranked evidence set and forced to cite, the differences between capable models narrowed to style and cost. The differences that mattered came from upstream: which passages reached the prompt and whether unsupported claims were caught afterward.

The practical consequence is a different spending order. Before paying for a larger model, spend on retrieval, reranking and the judge. Those investments compound with every model you try later, while a model upgrade is a one-time gain that the next model release erases. The same lesson applies in reverse to routing: in OmidGPT I route by complexity precisely because many requests do not need the most capable model, and the harness is what tells you where the line is.

How is cost kept predictable?

Predictable cost was a requirement, not a nice-to-have, because a study document is a long-running, multi-call job. Four mechanisms keep it bounded. Token accounting records every call's usage in the job state, so the cost of a document is known before it is delivered and can be attributed to the agent that spent it. Step budgets in the supervisor cap how many rewrite loops a document can consume. Rate limiting bounds concurrency against the model providers so a burst of jobs cannot exceed a known spend rate. And the reranker keeps prompts small: the largest cost driver in a RAG generator is the context it is handed, and a tight evidence set is the cheapest optimization available.

The bake-off harness also produces cost alongside quality for every model, which is what turns "which model is best" into "which model is best per unit of cost at the quality bar we require". That is a decision a CTO can make with numbers rather than with a vendor's benchmark table.

Results

The one number I can state is the test count: 673 tests across 77 files, covering the whole chain from parsing to RTL assembly. Qualitatively, the pipeline produces cited Persian study documents from mixed source material with every retained claim traceable to a passage, it terminates within a known budget, and every change to it is gated by the same evaluation layer that runs in production. The bake-off gave a defensible answer to the model question and a clear priority order for future work.

Lessons

1. Make citation a structural contract, not a prompt instruction. If a claim cannot exist without an identifier, most downstream checks become mechanical.
2. Score at the granularity you will act on. Document-level metrics detect regressions; claim-level verdicts tell you what to fix.
3. Put the judge under test. An evaluator with unknown accuracy is a liability, and its own fixtures belong in the suite.
4. Budget before the recursion limit. A step counter in state, checked before each model call, is cheaper and more explicit than the framework's backstop.
5. Spend on retrieval and evaluation before spending on a larger model. The bake-off made this concrete, and the gains carried over to every model tried afterward.
6. Treat RTL and mixed-direction text as a first-class requirement. Persian output that reads correctly is a product feature, and it needs its own tests.

Technologies

Limitations

The grounding filters are conservative, which means they sometimes drop a claim that a human would accept as a fair paraphrase. I chose recall of errors over recall of content, and that is a trade-off a product owner may want to tune. The claim-level judge is itself a model call, so the evaluation layer has its own cost and its own failure modes; the fixtures reduce that risk but do not remove it. The graph adds operational surface compared with a vector store, and the knowledge-graph construction step is sensitive to the quality of entity extraction on Persian text. And the bake-off results are specific to this corpus and this harness; the lesson about where to spend transfers, the ranking of models does not.

If you are building a pipeline with the same shape, cited generation from heterogeneous sources with a hard quality bar, the parts that transfer directly are the supervisor pattern, citation as a contract, and evaluation as code. Those three are where I would start an engagement, and they are the ones I audit first when a client brings me an existing RAG system.