OmidGPT: a multi-provider agentic AI platform in production

API endpoints

209

Database models

50

Lines of Python + TypeScript

~94K

Commits

391

Role

Designed, built and operate the platform (solo engineer)

OmidGPT is a multi-provider agentic AI assistant platform for the Persian market, live at omidgpt.ir. I designed it, built it and operate it as a solo engineer. The platform puts frontier models from several providers behind one database-driven catalog, runs an agentic tool-calling runtime with resumable turns and a tool-call ledger, connects to external MCP servers through an OAuth relay, and keeps serving economics under control with per-token cost accounting, semantic caching and complexity-based model routing. Its verified size today is 209 API endpoints, 50 database models, roughly 94K lines of Python and TypeScript, and 391 commits.

What problem did OmidGPT have to solve?

The product requirement was simple to state and hard to build: one assistant that gives users access to the best available models from more than one provider, streams answers in real time, can call tools and act on the user's behalf, handles voice, moderates content, and does all of that at a cost per conversation that a paid consumer product can sustain.

Each requirement is a known quantity on its own. Together they interact. Multi-provider means every provider's tool-calling format, streaming format and failure behavior has to be normalized behind one interface. Tool calling means a turn is no longer one request and one response; it becomes a loop that can pause, wait for a user decision, and resume. Sustainable economics means every token has to be attributed to a user, a model and a price, and the platform has to be able to avoid calling a frontier model when a cheaper one would answer just as well. Real-time streaming means all of this has to happen while text is already flowing to the client.

This case study describes how I built that, which decisions I would defend today, and which ones I would revisit. It is written for CTOs and engineering leads who are deciding how to take an assistant from a working prototype to something they can bill for. That transition is the core of my prototype-to-production work, and OmidGPT is the system where most of those patterns were first proven.

Context and constraints

Constraints shaped the architecture more than preferences did. The ones that mattered most:

• One engineer. I own the schema, the runtime, the front end, billing and operations. That pushed me toward a small number of well-understood components (Django, PostgreSQL, Redis) rather than a fleet of services, and toward making state explicit in the database so that I can reason about a stuck conversation by reading rows, not logs.
• A Persian-market product. Right-to-left interface, Persian text in prompts and tool arguments, and users who compare the assistant directly against the original provider apps. Streaming latency and interruption handling are visible to every user.
• Providers that change under you.Models are added, renamed, deprecated and repriced on the provider's schedule, not mine. A catalog change cannot require a deploy.
• Tool calls in the middle of a stream.The user sees partial text, then the model decides to call a tool, and the platform may need the user's consent before running it. The turn has to survive that pause and any disconnect that happens during it.
• A paid product. Quotas, billing and cost accounting have to be correct on the first attempt, because errors here are either lost revenue or lost trust.

Architecture

The request path runs through six layers. A Next.js, React and TypeScript client talks to a Django application (DRF for the HTTP API, Channels for WebSockets). Django reads and writes state in PostgreSQL and uses Redis for the channel layer and short-lived coordination. A router with a semantic cache in front of it decides whether a request is answered from cache, and if not, which model tier serves it. Provider adapters then call the selected model, and tool calls fan out to MCP connectors and to internal FastAPI services.

Two properties of this layout matter more than the boxes. First, every piece of long-lived state (the model catalog, the conversation and its turns, the tool-call ledger, quota and billing records) lives in PostgreSQL; Redis holds only what can be lost. Second, the router sits between the application and the providers, so caching, routing and cost attribution are applied once, in one place, for every provider. Nothing in the client or in the tool layer knows which provider answered.

Key engineering decisions

A database-driven model catalog

Every model the platform can call is a row: provider, provider-side identifier, capabilities (tools, vision, streaming), context window, input and output price per token, availability, and the routing tier it belongs to. The runtime reads the catalog; it never reads a constant. Adding a model, retiring one, or changing a price is a data change with an audit trail, not a release. The same rows feed the cost accounting, so the price used to bill a call is the price that was in force when the call was made.

Resumable tool turns and a tool-call ledger

Both major provider APIs express tool use the same way at the protocol level: the model returns a structured call carrying an identifier, the application executes it, and the result goes back referencing that identifier so the model can continue. Anthropic documents this as a tool_use block answered by a tool_result with the matching tool_use_id (tool use overview); OpenAI documents it as a function call with a call_id answered by a function_call_output (function calling guide). That identifier is the anchor I build on. Each proposed call becomes a ledger row before anything executes, and the row moves through explicit states.

from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class ToolCallRecord:
    call_id: str            # provider-issued id (tool_use id / call_id)
    turn_id: str            # the assistant turn this call belongs to
    tool_name: str
    arguments: dict
    status: str             # proposed | awaiting_consent | running | done | failed | denied
    attempts: int = 0
    result: dict | None = None
    error: str | None = None
    created_at: datetime = field(default_factory=datetime.utcnow)
    finished_at: datetime | None = None

Because the ledger is the source of truth, a turn is resumable. If the process handling the turn dies, or the user closes the tab while a tool is awaiting consent, the turn can be picked up later by reading the ledger: calls in done are replayed to the model as results, calls in running are checked or retried, calls in awaiting_consent wait. The provider conversation is reconstructed from rows rather than from an in-memory list that no longer exists.

Per-tool user consent

Tools with side effects (anything that writes, sends or spends) do not run automatically. The turn suspends in awaiting_consent, the client shows the proposed call and its arguments, and the user approves or denies it. Since turns are already resumable, consent is not a special code path; it is a suspended turn with a reason. Denied calls are returned to the model as a denial result so it can explain or propose something else rather than silently failing.

MCP connectors and the OAuth / MCP relay

External capabilities arrive as MCP servers rather than as bespoke integrations. The MCP authorization specification makes the server an OAuth 2.1 resource server: clients discover the authorization server through protected resource metadata, register dynamically where supported, use PKCE, and send a resourceindicator so tokens are bound to one server. The specification is explicit that an MCP server must validate token audience and must not pass a client's token through to upstream APIs (MCP authorization specification). I built a relay that owns that dance on behalf of the platform: it keeps per-user, per-server credentials, refreshes them, and presents the tools to the runtime as ordinary ledger-tracked calls. The runtime never sees a third-party token. I describe the design in the MCP OAuth architecture guide, and it is the pattern I reuse in MCP development work.

Semantic caching

A large share of assistant traffic is near-duplicate: the same question phrased slightly differently, by many users. The cache embeds a normalized form of the request, searches for close neighbors, and serves a stored answer when similarity clears a threshold and the request is cache-eligible (same model tier, no tools involved, no user-specific context in play). A hit skips the provider entirely and is recorded in the accounting layer as a cache hit, so cost reports separate real provider spend from traffic served from cache.

Complexity-based model routing

Not every message deserves a frontier model. The router estimates complexity from cheap signals (length, presence of code, whether tools are requested, whether the conversation is already multi-step) and picks a tier from the catalog. Users can pin a model, in which case the router only enforces availability and quota. The sketch below is the shape of the decision, not the production code:

def choose_model(request, catalog):
    if request.pinned_model:
        return catalog.available(request.pinned_model)
    score = estimate_complexity(request)   # length, code, tools, multi-step
    tier = "small" if score < LOW else "mid" if score < HIGH else "frontier"
    return catalog.cheapest_available(tier, needs_tools=bool(request.tools))

Caching and routing are the two levers that made the economics work, and they are worth a guide of their own: semantic caching and model routing.

Real-time streaming

Streaming runs over WebSockets through Django Channels with Redis as the channel layer. Provider streams are normalized into one event vocabulary (text delta, tool call proposed, tool result, turn finished, error) so the client renders one protocol regardless of provider. Because the turn's state lives in PostgreSQL and the ledger, a client that reconnects mid-turn re-fetches the turn instead of depending on the socket it lost.

Billing and quota

Quota is checked before a provider call is made, and a cost row is written when the call completes, using the catalog price for that model and the token counts the provider reports. Tool calls, cache hits and failed attempts are all attributable. This is the layer that turns the platform from a demo into a product, and it is also the layer that most prototypes are missing when they reach me for an AI technical audit.

Alternatives I considered

None of the decisions above were free. The trade-offs I weighed, at the pattern level:

• Single provider versus multi-provider. One provider means one SDK, one streaming format, one failure model, and a much smaller adapter layer. I chose multi-provider because the product promise is model choice and because provider-level outages and policy changes are a single point of failure for the business. The price is a normalization layer that has to be kept current for every provider.
• Hard-coded model list versus database catalog. A constant in code is simpler to read and impossible to corrupt at runtime. It also means every provider change is a deploy, and prices in the billing code drift from prices in the marketing copy. The catalog costs a few admin screens and a migration; it buys the ability to change models without touching the runtime.
• Fire-and-forget tool calls versus a ledger. Executing tool calls inline and keeping the transcript in memory is how every tutorial does it, and it works until a process restarts mid-turn or a user closes the tab during a consent prompt. The ledger adds a table and a state machine, and it makes every call inspectable, resumable and billable.
• Exact-match cache versus semantic cache. An exact-match cache is trivial and never wrong, but its hit rate on natural language is low. A semantic cache raises the hit rate and introduces the possibility of serving a subtly wrong answer, which is why eligibility rules and a conservative threshold matter more than the embedding model.
• Always use the best model versus route by complexity. Always-frontier is the simplest quality guarantee and the most expensive. Routing trades some quality on the margin for a large cost reduction on the bulk of simple traffic; a user override keeps the guarantee available to anyone who wants it.
• Bespoke integrations versus MCP. Direct API integrations are faster to build for the first two tools. MCP standardizes discovery, schemas and authorization for the tenth, at the cost of implementing the OAuth flow properly once.

Failure modes seen in production

A multi-provider agentic system fails in a small number of recurring ways. I will not attach counts or dates to these; the point is which mechanism catches each class.

• Provider errors mid-stream. A stream can stop after partial text, with a rate-limit, a server error or a silent close. The normalized event protocol emits an explicit error event, the partial text is kept with the turn, and the accounting row records what was actually consumed. The client shows a resumable state rather than a frozen cursor.
• Tool calls that never complete. A connector hangs, an MCP server goes away, or a consent prompt is abandoned. Because each call is a ledger row with a status and a timestamp, a stuck call is a query, not a mystery. Timeouts move calls to failed with a reason, and the turn resumes with that failure reported to the model.
• Cost blow-ups from retries. Retrying a provider call is safe; retrying it without bounds, or retrying a tool call that already had a side effect, is not. The ledger carries an attempt counter, retries are capped, and a call that reached running is checked before it is re-executed. Quota is enforced before the call, so a retry storm cannot exceed what the user is allowed to spend.
• Stale model identifiers. Providers rename and retire models. Because the runtime resolves models through the catalog, a retired model is flipped to unavailable and the router selects the next model in the tier; conversations pinned to it fall back with a visible notice rather than a hard error.
• Token misuse across servers. The relay holds per-server credentials and sends the resource indicator the MCP specification requires, so a token issued for one MCP server is never presented to another and the platform token is never forwarded.

Evaluation and reliability

OmidGPT is an assistant product rather than a document pipeline, so its evaluation looks different from the claim-level judging I use in Jozveh-AI. The properties I test are mostly about the runtime rather than the prose:

• Tool-loop correctness. Recorded provider responses are replayed against the runtime to check that every proposed call produces exactly one ledger row, that state transitions are legal, and that a resumed turn reconstructs the same provider conversation as an uninterrupted one.
• Consent boundaries. Tools flagged as side-effecting must never execute without a consent row, and denied calls must reach the model as denials.
• Cache safety. A set of paired requests that should and should not share an answer is used to keep the similarity threshold and eligibility rules honest when either is changed.
• Routing sanity. Fixed examples at each complexity level are checked against the tier the router picks, so a change to the scoring signals cannot silently push simple traffic to the frontier tier or complex traffic to the small one.
• Accounting invariants. Every completed provider call has a cost row; the sum of cost rows for a user never exceeds the quota that was granted.

Reliability comes from the same source as debuggability: state in the database, explicit statuses, and idempotent resumption. When something goes wrong at three in the morning, the fix is usually a state transition on a row, not a redeploy.

Cost: token-level accounting, caching and routing

Cost control in OmidGPT is three layers that depend on each other. Accounting comes first, because you cannot reduce what you cannot attribute: every provider call writes a row with the user, the model, input and output token counts as reported by the provider, and the catalog price in force at the time. Tool calls and failed attempts are included, so the true cost of an agentic turn is visible, not just the final answer.

Caching comes second and removes provider calls entirely for eligible near-duplicate traffic. Routing comes third and reduces the price of the calls that remain by sending simple traffic to smaller models. The order matters: without accounting, neither the cache hit rate nor the routing split can be measured, and a threshold or a scoring rule cannot be tuned against real spend. This is also why the first deliverable in my agent engineering work is usually the cost table, before any optimization.

Results

The verified numbers for the platform as it runs today: 209 API endpoints, 50 database models, roughly 94K lines of Python and TypeScript, and 391 commits. I deliberately do not publish traffic, revenue or latency figures for OmidGPT.

Qualitatively, the platform does what the requirement asked: multiple providers behind one catalog, streaming with tool calls and consent in the middle of a turn, MCP connectors behind a relay that keeps third-party tokens out of the runtime, and a cost layer that makes every turn attributable. It is operated by one person, and the design choices above are the reason that is possible.

Lessons learned

1. Make the tool loop a state machine on day one. Retrofitting a ledger onto a loop that keeps its transcript in memory is far more painful than starting with rows.
2. Put the catalog in the database and bill from it. The moment prices and model identifiers exist in two places, they disagree.
3. Accounting before optimization. Caching and routing are only tunable when you can see, per user and per model, what a turn cost.
4. Normalize providers at the event level, not the SDK level. A single event vocabulary for text, tool calls, results and errors is what lets the client, the ledger and the accounting stay provider-agnostic.
5. Treat consent as a suspended turn. Once turns are resumable, human-in-the-loop is a status, not a subsystem.
6. Implement MCP authorization as the specification describes, once, in a relay. Doing it per connector guarantees at least one of them forwards a token it should not.

Technologies

Limitations and what I would change

An honest list, because a case study that only defends its decisions is marketing.

• Single-operator risk. The system is designed to be run by one person, and it is. The documentation and runbooks exist, but the bus factor is real, and a client adopting these patterns should plan for a second pair of hands from the start.
• Heuristic routing. Complexity scoring is rule-based. It is transparent and cheap, and it is also blunt; a learned router trained on the accounting data would probably make better tier decisions on the margin. I have not built that yet.
• Cache eligibility is conservative by design. Excluding tool turns and user-specific context keeps the cache safe and leaves hits on the table. Widening eligibility safely needs a stronger evaluation set than the one I maintain today.
• The MCP authorization specification is still moving. The relay implements the flow as specified today. Each revision of the specification is maintenance work, and a connector ecosystem that changes its authorization requirements can break silently for one server while the others keep working.
• A monolith with a growing surface. Django with 209 endpoints is still manageable for one engineer, but the serving layer (router, cache, adapters) is the part I would extract first if a team took the platform over, because it changes at a different pace than the product features.
• Observability is adequate, not deep. The ledger and accounting tables carry most of the diagnostic load. Distributed tracing across the client, the runtime, the relay and the providers would shorten the rare investigations that those tables do not resolve.

If you are carrying an assistant prototype toward a paid product and any of the above sounds familiar, the shortest path is usually a technical audit to find which of these mechanisms is missing, followed by a fixed-scope prototype-to-production sprint to put it in place.