Service
LLM evaluation engineering
LLM evaluation is the harness that tells you whether a RAG pipeline or an agent got better after a change, instead of relying on a demo and a feeling. I am Matin Labkhandagh, and I build evaluation systems from golden sets, RAGAS, claim-level LLM-as-a-Judge scoring and agent trajectory checks, wired into CI, starting from a 5-working-day AI / LLM Technical Audit of what you already run.
When you need this
- Every prompt or model change ships on instinct because nobody can say, with a number, whether it helped.
- The agent passes your ten hand-picked examples and fails in ways that only show up in production logs.
- You are choosing between models on price and a public leaderboard, with no evidence from your own workload.
- Hallucinations are reported by customers before they are reported by your monitoring.
- The team wrote an LLM-as-a-Judge prompt, and now nobody trusts the judge any more than the model it grades.
What you get
- Evaluation strategy document: what to measure at each stage of your pipeline, with which metric, which dataset and which threshold.
- Hallucination analysis over a sample of real outputs, classified by cause: retrieval miss, ranking failure, prompt drift or model behavior.
- Retrieval / RAG review and tool-calling review with concrete findings and the tests that would catch each one.
- Golden set specification, and the initial set itself when the sources are available during the audit.
- RAGAS harness design for retrieval and faithfulness scores, and the harness itself where scope allows.
- Claim-level LLM-as-a-Judge design with a calibration plan against human-labeled samples.
- Agent trajectory evaluation: checks on tool selection, argument correctness, retries and termination, not only the final answer.
- Latency analysis and token-cost analysis per request class, so evaluation covers economics as well as quality.
- Prioritized implementation roadmap ordering the fixes by expected effect on the metrics.
Outcomes
- A golden set that reflects your real traffic, built with your domain experts and versioned with the code.
- Metrics your team agrees on, for retrieval, faithfulness, tool use and task completion, run automatically on every change.
- A judge you can trust because it is calibrated against human labels and audited for its own failure modes.
- Evidence for model choice from a controlled bake-off on your workload, so the decision rests on your questions and your costs.
- A prioritized roadmap that names the layer to fix first, with the metric that will show it worked.
How it works
- 1
20-minute technical call
You describe the system, the failures you have seen and how you judge quality today. I ask for read access to code, prompts and a slice of logs, and I say on the call whether the audit is the right first step or whether you already know enough to go straight to building the harness.
- 2
Written diagnosis
During the 5 working days of the audit I read the architecture, replay real requests through the pipeline and classify the failures I find. The diagnosis says where quality is lost, what should be measured, and what a first golden set should contain.
- 3
Fixed-scope proposal
The audit ends with a prioritized implementation roadmap. If you want the harness built rather than specified, I quote a fixed scope for that as an AI Agent Engineering Sprint, with the evaluation tests themselves as the acceptance criteria.
- 4
Implementation and handover
I build the golden set, the RAGAS harness, the judge and the agent trajectory checks, wire them into CI and hand over code, documentation and a runbook, plus a session with your engineers on adding cases and reading the reports.
Proof: what I built
In Jozveh-AI, an eight-agent GraphRAG pipeline that turns lecture audio and source material into cited study documents, I built the evaluation layer: RAGAS, a custom claim-level LLM judge that scores each generated statement against its sources, and post-generation grounding filters that act on the judge's verdicts. The system is covered by 673 tests across 77 files.
Read the Jozveh-AI case studyI ran a controlled model bake-off there, several models compared under the same evaluation harness. Model choice was not the bottleneck; retrieval quality, grounding and the evaluation loop moved faithfulness more than swapping models did. Without the harness that conclusion would have been a guess, and the budget would have gone to a larger model.
Read the Jozveh-AI case studyIn OmidGPT I built per-token cost accounting, a tool-call ledger and complexity-based model routing for a multi-provider agentic platform. The ledger is what makes agent evaluation possible in production: every tool turn is recorded, so trajectories can be replayed and scored rather than reconstructed from chat logs.
Read the OmidGPT case study
Questions about llm evaluation engineering
- Can you evaluate an AI agent we already built?
- Yes. That is the AI / LLM Technical Audit: 5 working days, starting at $400. I review architecture, retrieval, tool calling, prompts, hallucination, latency and token cost, define the evaluation strategy and deliver a prioritized implementation roadmap. For agents the emphasis is on trajectories, meaning whether the right tools were called with the right arguments and whether the agent stopped when it should, not only whether the final message looked right.
- What does an evaluation harness include?
- A golden set of inputs with reference outputs and, for RAG, the supporting source passages; RAGAS faithfulness and relevance scores; a claim-level LLM-as-a-Judge calibrated against a sample of human labels; agent trajectory checks for tool selection, arguments and termination; and a CI job that runs everything and fails on regression. Reports are per run, so a prompt change and its effect on the scores land in the same commit.
- Can you improve an existing RAG system, or only measure it?
- Both. Measurement comes first because it is what makes improvement provable. Once the harness exists, I fix the layer it points to, usually chunking, retrieval, reranking or grounding, under the AI Agent Engineering Sprint. The RAG development page describes that work in detail.
- When is GraphRAG worth it over vector RAG, and can evaluation tell us?
- Evaluation is exactly how to decide. Build a golden set that includes the multi-hop questions you care about, run the vector baseline, then run a GraphRAG configuration on the same set. If the graph only helps on questions nobody asks, you have your answer cheaply. GraphRAG is worth it when the failures are about connected context that similarity search cannot reach, which the GraphRAG development page and the GraphRAG versus vector RAG guide cover.
- How do you keep an LLM judge honest?
- By treating the judge as a model under evaluation too. I calibrate it against human-labeled samples and check that it agrees with them before trusting it, score claims individually rather than whole answers so its errors stay visible, and re-check it whenever the underlying model changes. A judge that is never audited drifts as quietly as the system it grades.