How would you make a project production-ready for a large user base?

In this article

1. Before you start
2. Why interviewers ask this
3. Production readiness from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one answer in the generative AI system design interview guide, and the same checklist applies to any service.

Why interviewers ask this

The question is open on purpose. A project that works for one tester can fail at a million users in ways a demo never shows, so the interviewer wants to hear how you think about failure, load and change.

A weak answer is a list of tools: "Kubernetes, a cache, a load balancer, monitoring." A strong answer starts with what "ready" means in numbers, then walks the request path and names what breaks at each point. A Staff-level answer adds the trade-offs: how much reliability is worth paying for, what happens when a dependency is slow instead of down, and how a bad release is stopped. For a generative AI product, cost and abuse count as failure modes too.

Production readiness from first principles

Stateless service

A stateless service is a program that keeps no user-specific data in its own memory between requests, so any replica can handle any request and copies can be added, removed or replaced freely without losing anyone's work.

In plain words

Think of a restaurant that is about to open to the whole city. The owner does not hope for the best. She decides how long a table may wait, buys enough stoves for the busiest hour, keeps a spare supplier for the key ingredient, tries new dishes on a few tables first, and watches the dining room for angry guests.

The analogy breaks in one place. A restaurant can close its door. A service cannot, so it must shed load and degrade gracefully instead.

In one precise sentence, a production-ready service meets stated targets for availability and speed under peak load and partial failure, can be changed without breaking those targets, and tells its owners when users are hurting.

Step 1: Turn ready into a target and an error budget

Production-ready starts with a measured service level indicator (SLI), such as the share of requests that succeed, and a service-level objective (SLO) on top of it. The Google Site Reliability Engineering (SRE) book chapter on service level objectives says plainly that "100% availability is impossible", so the target must be below it. It recommends an error budget, "a rate at which the SLOs can be missed", and tracking it over time.

The budget is simple arithmetic. A 30-day month has 43,200 minutes, so a 99.9% target allows 43.2 minutes of failure and a 99.99% target allows 4.32.¹ A full 10-minute outage then spends 23.15% of the 99.9% budget.¹

Pick the number from what users need and what the team can afford. A higher target costs more and slows releases. Also set a latency SLO on a high percentile, not the average. The same chapter plots the 50th to 99th percentile latencies of one system and shows how an average can look flat while the tail moves. You now have the yardstick the next steps are measured against.

Text description of the diagram

• A 99% target allows 432 minutes of failure in 30 days.
• A 99.9% target allows 43.2 minutes.
• A 99.99% target allows 4.32 minutes.

Step 2: Scale out with stateless services

To serve a large user base, run many identical replicas behind a load balancer and add more as load grows. A stateless service is a service that keeps no per-user data in its own memory between requests, so any replica can answer any request. Sessions, files and queues live in shared stores instead.

Capacity is arithmetic. Take an illustrative product with 1,000,000 daily users making 8 requests each. That is 8,000,000 requests a day, about 92.59 per second on average. If the busiest moment is four times the average, the peak is about 370.37 per second. If one replica handles 25 requests per second and you run it at 70% of that for headroom, you need 22 replicas.¹

Then ask what happens when a whole availability zone fails. With three zones, the remaining two must carry the peak, which takes 33 replicas in total, 11 per zone.¹ The Kubernetes documentation describes a horizontal autoscaler that adds or removes replicas as measured load changes. Autoscaling lags a sudden spike because new replicas take time to start, so keep a floor.

Step 3: Protect every dependency

Many failures at scale start in a dependency: a database, a queue or a model provider slows down. Protect each call with a timeout, a capped number of retries with growing waits (backoff), and a fallback.

Retries need care. If half of all calls fail and each is tried up to three times, the dependency receives 1.75 calls per request instead of one; when 90% fail it is 2.71, and in a full outage it is 3.¹ That extra load can turn a brownout (a partial outage where the service is slow or failing for some requests) into a full outage, so cap retries and add jitter (a random delay so retries do not arrive together).

Dependencies also multiply. Three services in a chain, each meeting 99.9% on its own, give about 99.7% together if their failures are independent (99.7003%), so the chain allows about 129.47 minutes of failure a month instead of 43.2.¹ The chain is never more available than its weakest link.

Cut the exposure, protect the data and secure access

Cache repeated answers, move slow work to a queue, rate-limit each user, and degrade features instead of failing. For language model (LLM) features, route calls through a gateway that holds the limits, as in the LLM gateway interview questions. Assume an illustrative 2,000 tokens per request and cached requests that cost no tokens: a 30% cache hit rate cuts 16,000,000,000 daily tokens to 11,200,000,000.¹

The data tier needs the same care. Add read replicas (extra database copies that serve reads) and connection pooling first, partition the data (sharding) when one primary cannot keep up, and keep backups with a restore you have tested. For security, require authentication on every endpoint, keep secrets out of code, and isolate each tenant's data; a tenant is one customer account. See permission-aware retrieval-augmented generation (RAG).

Step 4: Release safely and roll back fast

Many outages follow a change: the Google SRE book introduction says roughly "70% of outages are due to changes in a live system". A canary release is a rollout that sends a small share of traffic to the new version first, compares its error rate and latency with the old version, and widens only if the numbers hold. If they do not, the system rolls back automatically.

Make rollback cheap. Keep the previous version deployable, ship risky features behind feature flags (switches that turn a feature on or off without a new release), and make database changes in compatible stages so old and new code can run together.

The error budget from Step 1 makes this a rule instead of an argument: while budget remains, ship; when it is spent, pause feature releases and fix reliability. For retrieval-augmented generation (RAG) products, treat a new index or prompt as a release too, and test it on held-out questions first; see RAG over real documents.

Step 5: Watch user pain and plan for incidents

You cannot keep a target you cannot see. Monitor the four golden signals of the Google SRE book chapter on monitoring: latency, traffic, errors and saturation, which is how full the scarcest resource is. Alerts follow user-facing symptoms, such as error rate or slow responses, not every processor spike.

Load test before launch. Replay realistic traffic, ramp it past the expected peak, and note which component breaks first and whether the system recovers when load drops. Then write a runbook (step-by-step instructions for handling an alert) for each alert, rehearse a failover, and hold a blameless review after each incident.

For generative AI, add signals for tokens spent per user, refusals, and answer quality on a sampled set, and cap spend per tenant so one abusive account cannot drain the budget. A pipeline with several model calls, such as agentic RAG, needs these limits on every step, not only the first.

1 · Set targets2 · Scale out3 · Protect4 · Release safely5 · ObserveReplay

Step-by-step description

1. Define ready as an objective and an error budget.
2. Run stateless replicas sized for peak load.
3. Guard every dependency against slowness.
4. Canary first, roll back fast.
5. Alert on what users feel.

• A target and an error budget come first.
• Stateless replicas scale out with load.
• Every dependency gets timeouts, capped retries and caching.
• Canary releases roll back fast.
• Alerts follow user-facing symptoms.

The Staff-level answer

What makes it Staff-level: it starts from a measurable target, names what breaks (retry storms, chained dependencies, bad releases) and treats cost as part of reliability. Say it in this order: target, scale, protect, release, observe.

The rubric

Three variants of this question

What does production-ready mean for an LLM application?

It means the same targets as any service, plus guards for cost and abuse: token caps per user, rate limits, a fallback model or cached answer, and a quality check on sampled outputs. The model provider is a dependency that can be slow, so wrap it in timeouts and limits.

How do you roll out a risky change to a large user base?

Use a canary release, compare error rate and latency with the old version, widen in stages, and keep rollback one command away. Gate the rollout on the error budget.

What if traffic grows ten times in a week?

Check which part saturates first with a load test, scale the stateless tier and raise autoscaling floors, then look at the shared stores, which scale less easily. Shed low-priority load and queue slow work while you add capacity. For a streaming product, see how the voice agent design handles latency under load.

Common mistakes

• "Production-ready means 100% uptime." (wrong) The Google SRE chapter says 100% availability is impossible. (right) "I set a target below 100% and manage the error budget."
• "I would monitor average latency." (wrong) Averages hide the slowest users. (right) "I set the objective on a high percentile and alert on it."
• "Just add servers." (wrong) Shared state and downstream limits stay the bottleneck. (right) "I make the tier stateless, then check the database and the model provider."
• "Retry on every failure." (wrong) Retries multiply load during an outage. (right) "I cap retries, add backoff with jitter and use a fallback."
• "Cost is a finance problem." (wrong) An unbounded token bill is an outage of the budget. (right) "I cap spend per user and cache repeated answers."

Follow-up questions

How many minutes of downtime does a 99.9% target allow?

In a 30-day month it allows 43.2 minutes, and a 99.99% target allows 4.32.¹ Say the period you measure over, because the same percentage over a year allows a different number.

Why use a high percentile instead of average latency?

The average can stay steady while the slowest users wait much longer, as the percentile plot in the SRE book chapter shows. Users feel the slow requests, and multi-step requests hit them more often: with 30 sequential calls, each with a 1% chance of being slower than its 99th percentile, independently, 26.03% of requests contain at least one slow call.¹

How do you stop retries from making an outage worse?

Cap the attempts, wait longer between them with random jitter, and use a circuit breaker, which stops calling a dependency that keeps failing. With half the calls failing and three attempts, load rises to 1.75 times.¹

How do you keep a language model bill under control at scale?

Cache repeated prompts, cap tokens and requests per user, route simple requests to smaller models, and track spend per tenant in a shared gateway. The Perplexity design walkthrough shows where those controls sit in a search product.

How would you load test before launch?

Replay realistic traffic, ramp past the expected peak, find the first component to saturate, and confirm recovery when load drops. Compare the result with your latency and error targets, not with a feeling.

Sources

1. Service Level Objectives (Site Reliability Engineering, chapter 4) — Google (Site Reliability Engineering book, 2016; read 2026-10-26), 1 Apr 2016
2. Introduction (Site Reliability Engineering) — Google (Site Reliability Engineering book, 2016; read 2026-10-26), 1 Apr 2016
3. Monitoring Distributed Systems (Site Reliability Engineering, chapter 6) — Google (Site Reliability Engineering book, 2016; read 2026-10-26), 1 Apr 2016
4. Horizontal Pod Autoscaling — Kubernetes documentation (page read 2026-10-26), 26 Oct 2026