Semantic caching, model routing and token-level cost accounting in a multi-provider AI platform

LLM costs explode in production for four reasons that rarely show up in a prototype: retries multiply calls, conversation contexts grow with every turn, one frontier model ends up serving every request regardless of difficulty, and nobody accounts per request, so the bill arrives as a single number that cannot be traced back to a feature or a user. The controls are, in the order I build them: per-token cost accounting on every request and user, a semantic cache with a similarity threshold, complexity-based routing to cheaper models, and quotas that stop runaway usage before it reaches the provider. This guide describes that layer as I built it for OmidGPT, a multi-provider platform, and as I add it during a prototype-to-production sprint.

Why do LLM costs explode in production?

A prototype makes one call per user action with a short prompt. A product makes several: a classifier, a retrieval step, the main generation, maybe a formatting pass, plus whatever the retry wrapper adds when a provider times out. Each of those calls carries the full context, and in a chat product the context is the whole conversation, so the input side of the bill grows roughly with the square of the conversation length if nothing trims it. Add tool calling, where each tool result is fed back into the model, and a single user message can produce five or six provider calls with overlapping inputs.

The second driver is uniform model choice. Teams pick the strongest model during development because it hides prompt weaknesses, and then ship it for everything, including requests that a much cheaper model answers identically. The third driver is blind retries: a naive retry loop on a 429 or a timeout re-sends the full prompt, and if the failure is systemic (the provider is overloaded), the retries multiply cost without producing output. The fourth driver is the absence of accounting. Without a per-request record you cannot tell which of the first three is happening, so the fix becomes guesswork.

The control layer I describe below sits between the application and the providers. Every request passes through it in the same order, and every outcome, including cache hits and failed attempts, is written to the accounting store.

The four levers, what each one saves, and what it costs you if you get it wrong:

Token-level cost accounting: the first thing to build

Accounting comes first because every other lever is a claim about savings, and a claim without a baseline is not engineering. The unit of accounting is one provider call, not one user message. A user message that triggers a classifier, two tool rounds and a final generation produces four records, each linked to the same request id so they can be summed later.

What to store per call

• request id, user id, and the feature or endpoint that initiated the call
• provider, model id, and the model catalog version in effect at the time
• input tokens, output tokens, and cached input tokens as separate columns
• attempt number and outcome (success, provider error, timeout, cache hit, cache miss)
• computed cost in your billing currency, computed at write time from the price table
• latency to first token and total latency, because routing decisions later depend on both

The cached-token column matters more than it looks. Both major providers now report cached input tokens separately in the usage object and bill them at a different rate from ordinary input: the Anthropic usage object splits input into input_tokens, cache_creation_input_tokens and cache_read_input_tokens, with cache writes priced above base input and cache reads priced well below it (Anthropic prompt caching documentation), and OpenAI reports cached_tokens inside the usage details with a discounted cached-input rate (OpenAI prompt caching documentation). If your accounting treats all input tokens as one number, it will overstate cost on cached traffic and understate it on cache-write traffic, and you will misjudge whether caching is paying off.

Prices live in the database, not in code

In OmidGPT the model catalog is database-driven: each model row carries the provider, the provider model id, the per-token prices for input, output and cached input, the context limit, and flags for capabilities such as tool calling and streaming. Price changes are a data migration with an effective date, not a deploy. That is what lets the accounting stay honest when a provider re-prices, and it is what lets the router choose between models by reading the same rows.

from dataclasses import dataclass
from decimal import Decimal

@dataclass(frozen=True)
class ModelPrice:
    model_id: str
    input_per_token: Decimal
    output_per_token: Decimal
    cached_input_per_token: Decimal

@dataclass
class UsageRecord:
    request_id: str
    user_id: str
    model_id: str
    attempt: int
    outcome: str            # "ok" | "cache_hit" | "provider_error" | "timeout"
    input_tokens: int = 0
    output_tokens: int = 0
    cached_input_tokens: int = 0
    cost: Decimal = Decimal(0)

def price_for(model_id: str, catalog: dict[str, ModelPrice]) -> ModelPrice:
    # catalog is loaded from the database on a short TTL; a missing row is a bug, not a default
    return catalog[model_id]

def settle(record: UsageRecord, catalog: dict[str, ModelPrice]) -> UsageRecord:
    p = price_for(record.model_id, catalog)
    uncached = record.input_tokens - record.cached_input_tokens
    record.cost = (
        uncached * p.input_per_token
        + record.cached_input_tokens * p.cached_input_per_token
        + record.output_tokens * p.output_per_token
    )
    return record

Billing and quotas hang off this table. A user's balance is a sum over their records; a daily quota is a count or a cost sum over a window; a per-feature budget is the same query grouped by endpoint. Because failed attempts are recorded with their own outcome, you can decide as a product matter whether the user pays for a provider timeout (usually not) while still seeing that cost in the operator view (always).

How does semantic caching work, and when does it lie?

There are two caches in the diagram and they answer different questions. The exact-match cache asks whether this precise request (same model, same system prompt, same messages, same parameters) has been answered before. The semantic cache asks whether a request that means the same thing has been answered before. Exact match runs first because it is cheap, deterministic and never wrong for deterministic parameters. Semantic match runs second because it requires an embedding call and a vector search, and because it can be wrong.

Note that provider-side prompt caching is a third, separate mechanism. It caches a prefix of the prompt on the provider's infrastructure so repeated system prompts and long documents are not re-processed at full price; it does not return a stored answer. Both Anthropic and OpenAI require a minimum prompt length before a prefix becomes cacheable and match on identical prefixes only, so a timestamp at the top of a system prompt defeats it entirely. It complements the application-level caches described here; it does not replace them.

The mechanics

The semantic cache stores, for each answered request, the embedding of the normalized user query, the answer, the model that produced it, and a scope key. Lookup embeds the incoming query and runs a vector range query: return entries whose distance to the query vector is within a radius. Redis supports exactly this shape with VECTOR_RANGE queries over a FLAT or HNSW index using cosine, L2 or inner-product distance, and its documentation is explicit that the radius must be tuned per use case because the bounds differ per metric (Redis vector search documentation). The threshold is the whole design: too loose and you serve wrong answers, too tight and the cache never hits. I set it from an offline sample of real query pairs labeled as same or different, and I re-check it whenever the embedding model changes, because distances are not comparable across embedding models.

def cache_lookup(query: str, scope: str, *, threshold: float) -> str | None:
    key = exact_key(scope, query)
    hit = kv.get(key)
    if hit is not None:
        account(outcome="cache_hit", kind="exact")
        return hit

    vec = embed(normalize(query))
    # vector range query: entries within `threshold` distance, filtered to this scope
    candidates = vectors.range(scope=scope, vector=vec, radius=threshold, limit=3)
    if not candidates:
        return None
    best = min(candidates, key=lambda c: c.distance)
    if best.expired or best.model_id != current_model(scope):
        return None
    account(outcome="cache_hit", kind="semantic", distance=best.distance)
    return best.answer

Scope and invalidation

The scope key is what keeps the cache from lying. It includes the model id, a hash of the system prompt, and any retrieval corpus version, so a prompt change or a knowledge-base update invalidates every entry from the previous world at once without a scan. Entries also carry a time-to-live that is short for anything derived from changing data and long for stable reference answers. When a model is retired from the catalog, its entries are unreachable by construction because the scope no longer matches.

When a near-duplicate must not be served

A semantic cache is only safe for requests whose correct answer depends on the query text alone. Three classes of request fail that test and must bypass the semantic layer, and I gate them before the lookup rather than trusting the threshold to catch them:

• personalized prompts, where the same words from two users have different correct answers because the system prompt or retrieved context is user-specific; the scope must then include the user, which in practice reduces the cache to exact match
• time-sensitive prompts (anything asking about now, today, the latest, current status); a near-duplicate from yesterday is confidently wrong
• multi-turn requests where the query only makes sense with the prior turns; embedding the last message alone matches unrelated conversations, so either embed a windowed transcript or skip the cache after the first turn

Negation and small entity changes are the classic false positives: two queries that differ by a single product name or a single "not" sit close in embedding space. A cheap guard is a lexical check on the top candidate (do the named entities and negation tokens match) before returning it. It costs microseconds and removes most of the embarrassing hits.

Complexity-based model routing

Routing means deciding, per request, which model in the catalog should answer it. The principle is cheap by default, escalate on evidence. The evidence comes from three sources: a request classifier, the outcome of the first attempt, and the user's plan or the feature's configuration.

Classify the request

The classifier does not need to be a model. In most products a handful of deterministic features separate easy from hard well enough: input length, presence of code or attachments, whether tools are required, the feature that originated the request, and whether the conversation has already escalated once. Where a learned classifier is worth it, a very small model reading the last user message is enough; anything larger eats the savings. The output is a tier, not a model name, so the catalog can change without touching the router.

Escalate on failure or low confidence

The cheap model gets the first attempt. Escalation triggers are explicit and logged: the model refused or returned an empty answer, a tool call failed to parse, an output validator rejected the result, or the model's own self-assessment (when you ask for one) reports low confidence. Each trigger moves the request one tier up and writes an accounting record with the reason, so the escalation rate per trigger is a metric you can watch. If escalation exceeds a threshold for a feature, the classifier is wrong for that feature and the fix is upstream.

from dataclasses import dataclass

@dataclass(frozen=True)
class Route:
    model_id: str
    provider: str

TIERS = {
    "cheap":    [Route("small-a", "provider_a"), Route("small-b", "provider_b")],
    "standard": [Route("mid-a", "provider_a"),   Route("mid-b", "provider_b")],
    "frontier": [Route("large-a", "provider_a"), Route("large-b", "provider_b")],
}
ORDER = ["cheap", "standard", "frontier"]

def choose_route(tier: str, *, attempt: int, unhealthy: set[str]) -> Route | None:
    # same tier, next healthy provider first; escalate tier only when the tier is exhausted
    candidates = [r for r in TIERS[tier] if r.provider not in unhealthy]
    if attempt < len(candidates):
        return candidates[attempt]
    nxt = ORDER.index(tier) + 1
    if nxt >= len(ORDER):
        return None
    return choose_route(ORDER[nxt], attempt=0, unhealthy=unhealthy)

def run(request, tier: str, budget: int = 3):
    unhealthy = health.unhealthy_providers()
    for attempt in range(budget):
        route = choose_route(tier, attempt=attempt, unhealthy=unhealthy)
        if route is None:
            break
        result = call(route, request)
        account(route=route, attempt=attempt, outcome=result.outcome)
        if result.ok and validate(result):
            return result
        if result.provider_error:
            unhealthy.add(route.provider)
        elif result.low_quality:
            tier = ORDER[min(ORDER.index(tier) + 1, len(ORDER) - 1)]
    return fail_closed(request)

Keep a fallback provider in every tier

Every tier lists at least two providers. A provider failure moves the request sideways to the other provider in the same tier before it moves up, because a lateral move preserves cost and (roughly) quality while an escalation changes both. The mapping of tiers to concrete models is data in the catalog, and the health set is maintained by the reliability layer described next. If you are building agents on top of this, the routing decision belongs inside the agent graph as a node with its own state, which I cover in the LangGraph production architecture guide; it is also the piece I most often add during an agent engineering sprint when a team already has a working agent and a surprising bill.

Provider reliability: retries, fallbacks and streaming

Provider rate limits are enforced per minute on requests and on tokens, and both providers answer an exceeded limit with HTTP 429 and a Retry-After style header telling you how long to wait. OpenAI documents per-model requests-per-minute and tokens-per-minute limits that grow with usage tier, exposes remaining capacity in x-ratelimit-remaining-* response headers, and recommends exponential backoff with jitter (OpenAI rate limits documentation). Anthropic separates input and output token limits, uses a token-bucket so capacity refills continuously, exposes the same information in anthropic-ratelimit-* headers, and notes that for most models cached input tokens do not count toward the input-token limit, which means good prefix caching raises your effective throughput as well as lowering cost (Anthropic rate limits documentation). Read those headers into the health set; do not wait for the 429.

Retry budgets that do not multiply cost

A retry is a full re-send of the input, so an unbounded retry loop is a cost multiplier with no upper limit. I give each request a budget of attempts across all providers and tiers, count every attempt in accounting, and distinguish retryable failures (429 with a retry-after, transient network errors, overload) from non-retryable ones (validation errors, content refusals, a spend cap). Anthropic's documentation is explicit that a spend-cap 429 carries no retry-after header and that retrying it fails until access resumes; that is the case a naive loop burns through fastest. When the retry-after value exceeds what the user would tolerate, the right move is a lateral fallback, not a wait.

Mid-stream failures

Streaming complicates accounting because the usage numbers arrive at the end. In the Anthropic stream the message_start event carries the input token count, the final message_delta event carries cumulative output usage, and an error event can arrive mid-stream, for example an overload error that would have been an HTTP 529 in a non-streaming call (Anthropic streaming documentation). The accounting layer therefore opens a provisional record when the stream starts, estimates output tokens from the streamed text if the stream dies, and settles the record from the final usage event when it arrives. The user-facing decision for a dead stream is a product one: either resume with the partial text as context on a fallback route, or discard and retry from scratch. Both are legitimate; silently returning a truncated answer is not.

Quotas and rate limits as product decisions

Your own quotas are not a defensive afterthought; they are the pricing model made executable. The accounting table gives you the primitives: cost per user per window, calls per feature per window, tokens per conversation. The product decisions are which of those to cap, at what level per plan, and what happens at the cap. I have found three patterns that hold up.

• Cap on cost, not on message count, for paid plans. Message counts invite users to write one very long message; cost caps align the user's incentive with yours.
• Degrade before you block. At a soft limit, route the user to the cheap tier and disable the expensive features; hard-block only at the hard limit. Users tolerate a slower model far better than an error page.
• Make the cap visible. A remaining-budget indicator in the client removes most of the support tickets that quotas otherwise generate.

Per-user rate limits (requests per minute) exist for a different reason: they protect your provider rate limits from a single client, whether a script or a bug. They should be set well below the provider limits divided by expected concurrency, and they should be enforced at the edge, before the request reaches the cost layer at all.

What to measure

Every metric below is a query over the accounting table, which is the reason the table exists. I keep them on one dashboard so that a change in one is read against the others.

• cost per request and per user, broken down by model and by feature, as a distribution not a mean
• cache hit rate, split into exact and semantic, and the distribution of semantic-hit distances
• semantic cache complaint rate: user regenerate or thumbs-down immediately after a cached answer
• routing tier mix per feature, and escalation rate by trigger
• attempts per request, and the share of cost spent on attempts that did not produce output
• provider health: 429 rate, retry-after values seen, mid-stream error rate, per provider
• provider-side cached-token share of input, to confirm prefix caching is actually engaging
• quota events: soft-limit degradations and hard-limit blocks per plan

The pairing that matters most is cost per request against a quality signal from the same period. A routing change that lowers cost and raises regenerate rate is not a saving. If you do not yet have an evaluation loop, the cost layer will tell you what you spend but not what you lost, which is why an AI technical audit looks at token cost and evaluation strategy in the same pass.

What fails first, in my experience

The price table drifts. A provider changes prices, the catalog row is not updated, and the accounting quietly diverges from the invoice for weeks. The fix is procedural: the monthly invoice is reconciled against the accounting sum, and any gap above a small tolerance is an incident.

The semantic threshold is set once and never revisited. The embedding model gets upgraded, distances shift, and the cache either goes silent or starts serving wrong answers. Tie the threshold to the embedding model id in the scope key so an upgrade empties the cache and forces a re-calibration.

Routing is evaluated by error rate. The cheap tier produces no errors and worse answers, and nobody notices until users leave. Routing changes need an evaluation set run before and after, not a dashboard of exceptions.

Retries are implemented in three places. The HTTP client retries, the provider SDK retries, and the application retries, and a single transient failure becomes many full-price attempts. Pick one layer, disable the others, and count.

Streaming accounting is skipped. Records are written only on completed responses, so a period of provider instability shows as lower cost and lower traffic when in fact it was higher cost and lost traffic. Provisional records on stream start fix this.

What I observed in OmidGPT

LLM cost control checklist

• Every provider call writes one accounting record, including failed attempts and cache hits.
• Input, output and cached input tokens are separate columns with separate prices.
• Model prices and capabilities live in a database catalog with effective dates, not in code.
• Exact-match cache keys include model, system prompt hash and generation parameters.
• The semantic cache uses a range query with a threshold calibrated on labeled query pairs.
• Personalized, time-sensitive and deep multi-turn requests bypass the semantic cache by rule.
• Routing outputs a tier; the tier-to-model mapping is catalog data.
• Escalation triggers are explicit, logged with a reason, and monitored as a rate per feature.
• Each tier has at least two providers; lateral fallback runs before escalation.
• Retries have one owner in the stack and a per-request attempt budget.
• Rate limit headers feed a provider health set; spend-cap errors are never retried.
• Streams open a provisional accounting record and settle from the final usage event.
• Quotas degrade to the cheap tier before they block, and the remaining budget is visible to the user.
• Every cost change is read against a quality signal from the same period.

Further reading

• Anthropic: prompt caching (breakpoints, minimum cacheable length, TTL options and the three usage fields)
• OpenAI: prompt caching (prefix matching rules, cache retention and cached-token reporting)
• Anthropic: rate limits (token-bucket enforcement, cache-aware input limits, response headers, spend caps)
• OpenAI: rate limits (RPM and TPM, usage tiers, headers, backoff guidance)
• Anthropic: streaming messages (event sequence, cumulative usage in message_delta, mid-stream error events)
• Redis: vector search (FLAT and HNSW indexes, distance metrics, KNN and vector range queries)
• MCP OAuth architecture, for the connector and relay layer that sits beside the router in the same platform