How do you design an LLM gateway?

In this article

1. Before you start
2. Why interviewers ask this
3. LLM gateway architecture from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is part of the LLM inference optimization guide, and the models behind the gateway are the ones described in how transformers work.

Why interviewers ask this

The question separates candidates who know models from candidates who can run them for many teams. A gateway is where cost, reliability and quality trade against each other, so a loose answer ("add a cache and a load balancer") exposes itself quickly. Versions of this prompt are reported by candidates in interviews for platform and generative-AI engineering roles, and one reported version sets targets such as 100 requests per second, a p95 under 2 seconds and a cache hit rate over 40%; those targets are illustrative, not a public rubric.

A weak answer lists features. A strong answer orders them on the request path, says what each one costs in latency, and sizes the system with arithmetic. A Staff-level answer adds failure behavior: what happens when one provider is slow, when the cache returns a wrong answer, and when you migrate to a cheaper model.

LLM gateway architecture from first principles

LLM gateway

An LLM gateway is a service that receives every model request from your applications, applies shared policies such as access control, rate limits, caching and routing, calls a provider on their behalf, and records the result, so no application talks to a provider directly.

Fallback

A fallback is a second model or provider that the gateway calls when the first choice fails or times out, so the caller still gets an answer, possibly from a different model with different output quality, which is why a fallback must be tested before it takes traffic.

In plain words

Think of a hotel front desk. Guests (your applications) do not phone the restaurants, taxis and theaters themselves. They ask the desk. The desk checks who they are and remembers earlier answers. It picks the right place, tries another if the first is full, and keeps a ledger. The analogy breaks in one place. A desk clerk can say "no vacancy" and wait, while a gateway must return an answer within a latency budget, so waiting is rarely free.

In one precise sentence, a gateway is a policy layer on the request path to model providers. It is not a payment gateway, which moves money; searches for "design a payment gateway" are a different problem.

Step 1: The front door checks who is calling and how much they may use

The front door authenticates each caller and applies a rate limit. A rate limit is a cap on how much one caller may use in a time window, counted in requests per minute (RPM) and tokens per minute (TPM). Providers apply both kinds to you, so the gateway applies them again per team or per key, which stops one noisy application from using the whole provider allowance. The same keys carry a token budget and a spend cap per team, enforced from the cost records of Step 5, so a team that reaches its cap is refused before the provider call.

Input: an application request with a key. Output: an accepted request, or an immediate refusal. A refusal here costs microseconds and no provider money. Counting tokens before the call means estimating the prompt length and adding the requested output maximum, because providers commonly count that maximum against the per-minute budget, then correcting once the response reports real usage. You now have a bounded stream of identified requests, which is what the cache needs next.

Step 2: A two-level cache answers repeats before any provider is called

An exact cache returns a stored answer when the normalized request is byte-for-byte the same; a semantic cache returns one when an embedding of the new request is close enough to a stored one. Both live in the gateway, and the page on semantic cache design in caching LLM calls covers their design in depth. Check the exact cache first because it is cheap and cannot match a different question. It can still serve a stale answer, so key it on tenant, model and parameters and give entries a time to live. Never cache provider errors: Request for Comments (RFC) 6585 says 429 responses must not be stored by a cache.

Here is the cost of getting the arithmetic wrong. With a 40% hit rate and a hit costing 50 milliseconds, a miss 1,500 milliseconds (all illustrative), the mean latency is 920 milliseconds. But the 95th percentile of all requests is the 91.7th percentile of the misses, because hits fill the fast end.¹ The cache lowers the average and the load, not the tail. The tail is still decided by the provider call in the next steps.

Step 3: The router picks a model and a deployment

The router decides which model and which deployment serves a miss. Two choices happen here. First, model choice by task: a short classification can go to a small model and a hard reasoning request to a large one, as described in model routing and cascades. Second, deployment choice among copies of the same model across regions or accounts, by remaining quota, current latency or a simple rotation.

Input: a cache miss and its metadata. Output: one chosen deployment. The router reads a health table that the next step updates, so it skips deployments whose circuit breaker is open. Behind the router sits a provider adapter layer. The gateway exposes one unified application programming interface (API), and each adapter translates between it and one provider's request and response schema, tool-call format, streaming chunks and error codes.

Without adapters, routing and fallback across providers cannot work, because a fallback must receive the same request in a different dialect. Keep routing rules in configuration, not code, so an operator can shift traffic during an incident without a release. You now have a target to call; the next step makes the call survive trouble.

Step 4: The provider call retries, breaks and falls back

The call itself needs a timeout, a bounded retry policy and a fallback. A provider returns status 429 when you exceed a limit, as defined in the Request for Comments (RFC) document RFC 6585 (2012), and the Retry-After header is defined in RFC 9110, section 10.2.3 (2022); one rate-limit guide separates a recoverable rate-limit error, where backoff and retry works, from a quota error that needs an account change. It says to honor a Retry-After header when present, and lists exponential backoff with jitter as the first retry strategy. Jitter means a random share of each delay, so clients do not retry in step.

If retries fail, the circuit breaker opens and the router sends traffic to the fallback. After the cooldown the breaker goes half-open: one trial request goes to the sick deployment, and traffic returns only if it succeeds. A fallback is not free: models differ in format and quality, so it must be tested on your own evaluation set first. Input: a chosen deployment. Output: a response or a clear error.

Step 5: Every call leaves a trace

The gateway logs one record per request: caller, model, deployment, input and output tokens, cost, cache outcome, retries, fallback used and total latency, all under one trace id that follows the request through the gateway and into the provider call. A cache hit skips straight to the log record, so tracing spans every stage. Because every request passes the gateway, this is the one place where cost per team and failure rate per provider can be computed without asking each application.

Log metadata by default and prompt text only under a retention policy, since prompts often hold personal data. Alert on the rate of 429 responses, the open-circuit count, the hit rate and the p95. You now have the whole path: front door, cache, router, provider call, trace.

1 · Front door2 · Cache3 · Router4 · Provider call5 · TraceReplay

Step-by-step description

1. Check the caller and apply limits.
2. Answer repeats without a provider call.
3. Pick a model and a deployment.
4. Retry, break and fall back.
5. Log tokens, cost and latency.

• The front door checks the caller and applies rate limits.
• A cache answers repeats before any provider is called.
• The router picks a deployment and the provider call retries or falls back.
• Every call leaves a trace with tokens, cost and latency.

Worked example

Sizing starts from the target: 100 requests per second (illustrative), 1,500 tokens per request and 1,500 milliseconds per provider call (both illustrative). Little's law is the rule that the number of requests in flight equals the arrival rate times the average time each spends in the system. So the provider side holds 100 × 1.5 = 150 requests in flight without a cache, and 90 with a 40% hit rate, because only 60 requests per second reach a provider.¹

The token budget gives the same story. 100 requests per second at 1,500 tokens is 9,000,000 tokens per minute without a cache and 5,400,000 with one.¹ If one deployment allows 1,000,000 tokens per minute (illustrative) and you plan to use 80% of it, you need 12 deployments without the cache and 7 with it.¹ Cost follows the miss rate. Relative cost per request is 1.00 with no cache; counting a lookup as 0.02 of a model call, it is 0.62 at a 40% hit rate, and a cache that never hits would cost 1.02.¹

Text description of the diagram

• Relative cost per request is 1.00 with no cache and 0.62 at a 40% hit rate.
• A lookup is counted as 0.02 of a model call.

Retries and failover have their own arithmetic. With backoff starting at 500 milliseconds and doubling, the delay ceilings are 500, 1,000, 2,000, 4,000 and 8,000 milliseconds. If 1,000 clients fail together and retry with no jitter, 1,000 requests land in one instant; full jitter spreads them over the first 500 milliseconds, about 2 per millisecond.¹ Two providers that each work 99% of the time fail together 0.01% of the time only if their failures are independent, giving 99.99%; if failures are fully correlated, you stay at 99%.¹

Trade-offs and edge cases

The gateway is a single point of failure and an extra hop. Run it stateless in several replicas behind a load balancer, keep cache and counters in a shared store, and measure the added latency; the hop should cost little next to the model call.

Edge cases matter more than the table. A semantic cache with a loose similarity threshold can return the answer to a different question, so tune the threshold on labeled pairs and never cache requests that depend on user-specific data. Streaming responses complicate caching and retries, since a stream cannot be retried after the first tokens reach the user. An exact cache matches identical requests and a semantic cache matches similar meaning, both in the gateway, and only the semantic one can return a wrong answer.

Provider-side prompt caching, which reuses work for a repeated prompt start, is a different mechanism from a gateway cache; the table below compares all three. Cheaper model routing and compressed models (LLM quantization) change quality, so route by measured results. Self-hosted models behind the gateway add queueing that continuous batching manages, and serving a 70B model at a tight p95 shows that side.

Text description of the diagram

• An exact cache matches identical requests in the gateway.
• A semantic cache matches similar meaning in the gateway and can return a wrong answer.
• Provider prompt caching runs at the provider and reuses work for a repeated prompt start.

How to answer it in an interview

The spoken structure: name the path in order, give one number that shows you can size it, then state the main trade-off.

Common mistakes

• "I will retry every failure right away." (wrong) Immediate retries add load to an overloaded provider and arrive together. (right) "I retry rate-limit errors with exponential backoff and jitter, and honor Retry-After when present."
• "A cache hit rate of 40% cuts latency by 40%." (wrong) The tail is set by misses. (right) "It cuts load and mean latency; p95 still depends on the provider call."
• "The fallback model is a drop-in replacement." (wrong) Output format and quality differ. (right) "I evaluate the fallback on my own test set before it takes traffic."
• "Semantic caching is just prompt caching." (wrong) One runs in the gateway on meaning, the other at the provider on the prompt start. (right) "I use both and tune the semantic threshold."
• "Log everything, including prompts." (wrong) Prompts can hold personal data. (right) "I log metadata by default and prompt text under a retention policy."

Follow-up questions

How would you manage rate limits in OpenAI/Azure APIs?

Count tokens and requests per key at the gateway, spread load across deployments by remaining quota, and retry 429 responses with backoff and jitter. A rate-limit guide notes that a quota error, unlike a rate-limit error, needs an account change rather than a retry.

How would you cache LLM responses to reduce cost?

Use an exact cache first, then a semantic cache with a tuned threshold, and skip user-specific requests. Set a time to live (TTL), the time after which a stored answer expires, for content that goes stale. See caching LLM calls.

How would you architect a multi-model AI gateway?

Expose one unified API and put an adapter per provider behind it, then run the same request path: access control and budgets, cache, router, provider call with fallback, trace. The adapters translate schemas, tool calls, streaming and errors, so the router can swap models without the applications noticing.

When do you route to a small distilled model vs. a large LLM?

Route to the small model when a measured evaluation shows it meets the quality bar for that task class, such as classification or extraction, and escalate to the large model on low confidence. Model routing and cascades walks through the design.

You need to migrate a production feature from an expensive model to a cheaper one. How do you do it safely?

Replay logged traffic through both models and compare on an evaluation set, then send a small share of live traffic to the cheaper model while the gateway logs quality and cost for each arm. Keep the old model as the fallback until the metrics hold.

How do you log and monitor LLM API failures in production?

Log one record per request with a trace id, and alert on the 429 rate, open circuits, error rate per provider, hit rate and p95. Separate provider errors from your own timeouts so you know whom to fix.

Sources

1. OpenAI Rate Limits: Complete Guide to TPM, RPM & Tier Limits (2026) — inference.net (secondary guide; accessed 2026-10-26), 26 Oct 2026
2. RFC 6585: Additional HTTP Status Codes (429 Too Many Requests) — IETF, 1 Apr 2012
3. RFC 9110: HTTP Semantics (Retry-After) — IETF, 1 Jun 2022