AI / LLM technical audit

When you need this

• The demo works. Real users keep finding the failure modes, and nobody can say which layer is producing them.
• Token spend grows faster than usage, and the provider invoice cannot be traced back to a feature, a user or a prompt.
• There is no evaluation, so every prompt change is a gamble and regressions are discovered by customers instead of tests.
• Latency is unpredictable: the same question is fast one time and slow the next, and the team blames retrieval, the model or the tool loop without evidence.
• You are about to commit engineering weeks to a rebuild and want an outside engineer to check the plan before the budget is spent.

What you get

• Architecture review: how requests move through models, retrieval, tools, state and caches, where the design itself creates failure modes, and which components are doing work the architecture does not need.
• Retrieval / RAG review: chunking, embeddings, indexes, reranking and the retrieval-to-prompt path, checked against real queries to find where the right context is missing or the wrong context is winning.
• Tool-calling review: tool contracts, argument validation, error handling, retries, idempotency and consent for sensitive actions, so the agent loop fails safely instead of silently.
• Prompt review: system and task prompts read against the model's actual outputs, to find instructions that are ignored, contradictory, too long, or doing work that belongs in code or retrieval.
• Hallucination analysis: a sample of real outputs checked claim by claim against their sources, classifying each failure as a retrieval gap, a grounding gap or a model behavior, because each needs a different fix.
• Latency analysis: where the time goes across retrieval, model calls, tool execution and orchestration, and which of those are serial by design rather than by necessity.
• Token-cost analysis: which features, prompts and users drive spend, what a semantic cache and complexity-based model routing would change, and where the accounting is missing entirely.
• Evaluation strategy: the metrics, datasets and judges that fit your system, RAGAS and LLM-as-a-Judge where they apply, and how to wire them into tests the team runs on every change.
• Prioritized implementation roadmap: findings ordered by impact and effort, with what can be fixed in days, what needs a sprint, and what should be left alone.

Outcomes

• A written map of your AI system as it actually runs: model calls, retrieval, tools, state and caches, and where each request spends its time and tokens.
• Named root causes for hallucination, latency and cost, each tied to the layer that produces it, so fixes go to the right place the first time.
• An evaluation strategy your team can run on every change, with acceptance criteria written as tests wherever practical.
• A prioritized implementation roadmap that separates cheap, immediate wins from work that needs a sprint.
• A decision basis for the next step: fix in place, run an agent engineering sprint, or take the prototype to production.

How it works

1. 1

   Technical call

   A 20-minute technical call. You describe what you built, what is failing and what good looks like for your users. I ask about model calls, retrieval, tools, traffic and what evidence already exists. If an audit is the wrong instrument for your problem, I say so on the call.

2. 2

   Written diagnosis of scope

   After the call I send a short written note: what I will review, what questions the audit will answer, and what I need from you, typically read access to the repository, sample traces or logs, and a cost export from your providers. Missing traces are not a blocker; they become a finding.

3. 3

   Fixed-scope proposal

   Scope, timeline and price in writing. The audit is five working days, starting at $400; the price moves only if the system is materially larger than what we discussed, and you know that before we start.

4. 4

   Audit and handover

   Five working days of reading code, replaying real requests, measuring latency and tokens, and checking outputs against sources. You receive the written report covering all nine areas, a walkthrough call with your engineers, and the roadmap. Where practical, findings are expressed as tests your team can keep.

Proof: what I built

• In OmidGPT, a multi-provider agentic AI platform with 209 API endpoints, 50 database models and about 94K lines of Python and TypeScript, I built per-token cost accounting, semantic caching, complexity-based model routing and production billing and quota systems. The token-cost and latency analysis in this audit applies the same discipline: every call attributed, every cache hit counted, every routing decision explainable.

  Read the OmidGPT case study

• In Jozveh-AI, an eight-agent LangGraph pipeline over Neo4j GraphRAG, I wired RAGAS, a custom claim-level LLM judge and post-generation grounding filters into 673 tests across 77 files. In a controlled model bake-off under that harness, retrieval quality, grounding and the evaluation loop moved faithfulness more than swapping models did. The hallucination analysis and evaluation strategy in this audit come from that experience.

  Read the Jozveh-AI case study