In this article
Gateway basics
What is an LLM gateway, and why put one in front of model providers?
An LLM gateway is a single service that every product calls instead of calling model providers directly, so keys, limits, retries, caching and logging live in one place. Cloudflare's AI Gateway documentation names analytics, logging, caching, rate limiting, and request retry and model fallback as its core features (Cloudflare, updated April 2026). The payoff is that a provider outage or a price change is fixed once.
Why it is asked: it checks that you see the platform problem, not only the model call. Read the full explanation: GenAI system design for the platform context, and the LLM gateway section of LLM inference for the gateway itself.
How would you architect a multi-model AI gateway?
Put a stateless gateway tier behind a load balancer. Each request passes authentication, a rate-limit check, a cache lookup, a router that picks a provider and model, and a logger. Provider adapters translate one internal request format into each vendor's application programming interface (API). Keep configuration such as routes, budgets and keys in a store the gateway reads, so changing a route needs no deploy, and keep rate-limit counters and budgets in a shared fast store such as Redis so every replica sees the same totals.
Why it is asked: it tests whether you can name components and draw the request path. Read the full explanation: capacity and gateways in LLM inference.
Text description of the diagram
- Clients call the gateway instead of the model providers.
- The gateway applies keys, limits, cache and routing in one place.
How do you handle streaming responses through a gateway?
Pass the stream through chunk by chunk rather than buffering it, so the first token reaches the client as soon as the provider emits it. Count tokens as chunks arrive, or read the final usage record, to charge the right team. If the client disconnects, cancel the upstream request so you stop paying for tokens nobody reads. Failover is only safe before the first chunk has been sent.
Why it is asked: streaming breaks the simple request-response model most candidates assume. Read the full explanation: LLM inference: serving and latency.
Routing and tiering
What is model tiering, and when do you send a query to a small model?
Model tiering routes easy queries to a cheaper, smaller model and hard ones to a stronger one. FrugalGPT describes an LLM cascade that tries the cheap model first and escalates when a scorer judges the answer unreliable, and reports matching OpenAI's generative pre-trained transformer (GPT) model GPT-4 on its tasks with up to 98% lower cost, or 4% better accuracy at the same cost (Chen et al., 2023). Those figures are from the paper's own tasks, not a promise for yours.
Why it is asked: cost-quality trade-offs are the core of platform work. Read the full explanation: caching and routing LLM calls.
How does a router decide between a strong and a weak model?
A router scores each query and sends it to the strong model only when the weak one is likely to fail. RouteLLM trains such routers on human preference data and reports cost reductions of over 2x in some cases without a loss in response quality (Ong et al., 2024). A simpler start is rules by task type, with a learned router added once you have evaluation data.
Why it is asked: it separates rule-based routing from learned routing and tests whether you ask for an evaluation set first. Read the full explanation: caching and routing LLM calls.
Which load-balancing strategies do gateways offer across deployments?
Common strategies pick a deployment at random (weighted by capacity), by least ongoing calls, or by lowest latency. LiteLLM's router documents simple-shuffle as the default it recommends for production, plus least-busy and latency-based, and marks usage-based routing as not recommended for production because of performance impact (LiteLLM documentation, read October 2026). Pick by what you can measure cheaply on the hot path.
Why it is asked: it checks you know routing has a cost of its own. Read the full explanation: capacity and gateways in LLM inference.
How do you migrate a feature to a cheaper model safely?
Evaluate first, then shift traffic gradually. Run the cheaper model in shadow on real requests and compare outputs against the current model on a golden set, then send a small canary share, watch quality and guardrail metrics, widen in steps, and keep a one-step rollback. A 5% canary on 10,000 requests is 500 requests, enough to catch gross failures but not subtle ones.1
Why it is asked: it tests release discipline for a probabilistic component. Read the full explanation: deployment strategies.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Reliability: failover and retries
How do you fail over between providers?
Order the deployments, retry within a level, then fall to the next, and temporarily remove a failing deployment. LiteLLM's router tries order 1 deployments first, then order 2 and so on, and cools a deployment down after repeated failures in a minute, with defaults of three allowed failures and a five-second cooldown (LiteLLM documentation, read October 2026). If two providers were independently available 99.5% of the time, the pair would reach 99.9975%, but shared dependencies make real figures lower.1
Why it is asked: failover is the reason gateways exist. Read the full explanation: 429 replies and retry hints in rate limiter and API gateway design.
How should the gateway retry a failed call?
Retry only errors that can succeed later, with exponential backoff and random jitter. OpenAI's guide describes exponential backoff as waiting briefly after a failure and lengthening the delay each time, and recommends random jitter so retries do not all land together (OpenAI, read October 2026). Cap the attempts and the delay, and never retry a request the client has abandoned.
Why it is asked: it separates naive loops from production retry logic. Read the full explanation: retry hints and backing off in rate limiter and API gateway design.
What does a 429 mean, and what do you do with Retry-After?
A 429 means the client sent too many requests in a given amount of time, and the server may include a Retry-After header saying how long to wait, as the Request for Comments (RFC) 6585 standard puts it (RFC 6585, April 2012). OpenAI says to treat that value as a minimum (OpenAI, read October 2026). Wait at least that long, and shift traffic to another deployment meanwhile.
Why it is asked: it checks you know the standard signal. Read the full explanation: rate limiter and API gateway design.
How do you stop a retry storm?
Add jitter, a retry budget and a cooldown. Without jitter, 1,000 clients that all fail at the same moment retry together, so the third retry puts all 1,000 into one 100-millisecond window; with full jitter in the illustrative simulation the peak was 35.1 A retry budget caps retries as a share of traffic, and a cooldown removes a failing deployment, as LiteLLM's router does (LiteLLM documentation, read October 2026).
Why it is asked: it tests whether you have seen an outage made worse by retries. Read the full explanation: why immediate retries add load, in rate limiter and API gateway design.
Rate limits and cost control
How would you manage provider rate limits across many teams?
Treat the provider limit as a shared pool and divide it. The gateway tracks usage per team and enforces a quota below the provider's, with burst allowance and priority tiers, and queues or rejects the rest with a 429 of its own. Anthropic lets an organisation set lower limits per workspace so one workspace cannot use up the others' share (Anthropic, read October 2026).
Why it is asked: fairness under a shared quota is the heart of a platform role. Read the full explanation: rate limiter and API gateway design.
Why limit by tokens and not only by requests?
Because cost and capacity depend on tokens, and request size varies enormously. OpenAI measures limits in requests per minute (RPM) and tokens per minute (TPM), among others, and the limit hit first applies (OpenAI, read October 2026). Anthropic measures requests per minute, input tokens per minute and output tokens per minute separately (Anthropic, read October 2026). A gateway needs a token estimate before the call and the real count after.
Why it is asked: it tests LLM-specific limiting beyond the classic rate limiter. Read the full explanation: rate limiter and API gateway design.
How does a token bucket work?
A token bucket holds up to a capacity of tokens, refills at a steady rate, and spends one token per request (for a tokens-per-minute limit, each request spends its token count instead). A bucket of 10 tokens refilled at 5 a second admits 10 of a burst of 20 requests, then 5 more one second later.1 Anthropic states that its API uses the token bucket algorithm, so capacity is continuously replenished up to the maximum rather than reset at fixed intervals (Anthropic, read October 2026).
Why it is asked: it is the standard rate-limiting algorithm and the usual follow-up. Read the full explanation: rate limiter and API gateway design.
How do you enforce per-team budgets?
Meter every call from the provider's usage fields, attribute it to a team, and compare it with a budget the gateway can read quickly. Decide what happens at the cap: reject, downgrade to a cheaper model, or alert. Note that provider spend caps behave differently from rate limits: Anthropic says its spend-cap 429 carries no retry-after header, and retries fail until access resumes (Anthropic, read October 2026), so the gateway should not retry it.
Why it is asked: cost attribution is a day-one platform requirement. Read the full explanation: capacity and gateways in LLM inference.
Caching
How would you cache LLM responses to reduce cost?
Start with an exact-match cache keyed on the normalised prompt, model and parameters, and add a semantic cache only where near-duplicates are common. Cloudflare lists caching as serving requests directly from its cache instead of the provider (Cloudflare, updated April 2026). At a 40% hit rate, 600 of every 1,000 requests still reach the provider.1
Why it is asked: caching is the cheapest cost lever and invites trade-off questions. Read the full explanation: caching and routing LLM calls.
Text description of the diagram
- At a 40% hit rate, 600 of every 1,000 requests still reach the provider.
- At a 60% hit rate, 400 of every 1,000 requests still reach the provider.
What are the risks of a semantic cache?
A semantic cache returns a stored answer for a similar prompt, so a wrong match returns a confident wrong answer. If 40% of 1,000 requests hit and 2% of hits are wrong, that is 8 wrong answers (illustrative).1 Other risks are stale answers, leaking one tenant's answer to another unless keys include the tenant, and personal data in cached text. Set a strict similarity threshold and evaluate it on real traffic.
Why it is asked: it separates enthusiasm for caching from judgement. Read the full explanation: caching and routing LLM calls.
How do provider prompt caches change rate limits?
Anthropic states that for most Claude models only uncached input tokens count toward the input-token limit, and gives an example: with a 2,000,000 input-tokens-per-minute limit and an 80% cache hit rate, about 10,000,000 total input tokens per minute can be processed (Anthropic, read October 2026). The arithmetic is 2,000,000 divided by 0.2.1 Stable prefixes in prompts therefore raise effective throughput.
Why it is asked: it tests knowledge of how limits and caching interact. Read the full explanation: LLM inference.
Operations and security
How do you log and monitor LLM API failures in production?
Log one structured record per call with a request id, team, model, provider, token counts, latency, status and error class, with sensitive text redacted or sampled. Alert on error rate by provider, rising latency, rate-limit responses and cost per team. Trace a request across gateway and provider so a slow call can be attributed. Cloudflare lists logging and analytics, including requests, tokens and cost, as gateway features (Cloudflare, updated April 2026).
Why it is asked: you cannot run what you cannot see. Read the full explanation: LLM observability.
How do you keep keys and tenant data safe in a gateway?
Hold provider keys only in the gateway, give each team its own gateway credential that can be revoked, and scope every cache key, log and quota by tenant. Redact personal data before logging, and decide residency per tenant, since a request may cross regions. A gateway that sees all prompts is a high-value target, so limit who can read its logs.
Why it is asked: central control creates central risk. Read the full explanation: access control and multi-tenant design in GenAI system design.
- A gateway centralises keys, limits, retries, caching, routing and logs so provider changes are fixed once.
- Route by evaluated quality and cost, and migrate with shadow, canary and rollback.
- Retry with backoff and jitter, honour Retry-After, and cool down failing deployments.
- Limit by tokens as well as requests, and meter spend per team.
- Cache exact matches first; semantic caches need a strict threshold and tenant-scoped keys.
Sources
- Cloudflare AI Gateway overview — Cloudflare developer documentation (page updated 2026-04-20), 20 Apr 2026
- Router - Load Balancing, Fallbacks and Retries (LiteLLM routing) — LiteLLM documentation (date = read date), 24 Oct 2026
- Rate limits (OpenAI API guide) — OpenAI developer documentation (date = read date), 24 Oct 2026
- Rate limits (Claude API documentation) — Anthropic documentation (date = read date), 24 Oct 2026
- RFC 6585: Additional HTTP Status Codes — IETF (Nottingham and Fielding), 1 Apr 2012
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — arXiv (Chen, Zaharia and Zou), v1, 9 May 2023
- RouteLLM: Learning to Route LLMs with Preference Data — arXiv (Ong et al.), v4 revised 2025-02-23, 26 Jun 2024
