In this article
  1. Before you start
  2. Why interviewers ask this
  3. Regulated LLM deployment from first principles
  4. Worked example
  5. Trade-offs and edge cases
  6. How to answer it in an interview
  7. Common mistakes
  8. Follow-up questions
  9. Sources

Before you start

This page is one step of the generative AI system design learning path.

Why interviewers ask this

A design that works but cannot be certified does not ship in a bank, a hospital or a defence contractor. That is why this question separates candidates who build demos from candidates who can put a model into a regulated customer's hands. Candidates report versions of it for a regulated-industry customer, for a government contractor and for a multi-tenant deployment, at AI labs and large tech companies.

A weak answer says "use a secure application programming interface (API) and add a filter." A strong answer says where the model runs and why that location carries the compliance, what the prompts travel over, what gets logged, how a user's access is enforced before the model sees any document, and how you would know a new release hurt one customer. A Staff-level answer adds the measurement: a test set per tenant, because an average hides a single tenant's regression. The rest of this page builds that answer from zero.

Regulated LLM deployment from first principles

Inherited compliance
Inherited compliance is a property of a deployment in which the customer's own authorization reuses the controls the hosting cloud platform has already had certified, so the customer is assessed only on the layer it builds (configuration, access control, logging) and never certifies the model itself.
Multi-tenant deployment
A multi-tenant deployment is a single running system that serves several customers, called tenants, while keeping each tenant's data, settings and results separate, so one tenant's prompts, documents and logs are never visible to another. It shares compute and code, and separates everything that carries customer data.
Invocation logging
Invocation logging is a setting in a hosted model service, such as Amazon Bedrock, that saves a copy of each request and response to storage the customer owns, so the customer has an audit trail. The saved copies can contain sensitive information, so that storage needs its own access and encryption controls.

In plain words

Think of a bonded warehouse. The warehouse holds the licence. You store goods inside and your goods are covered by that licence. You do not need your own licence for the building. The analogy breaks in one place. The licence does not cover how you arrange your own shelves or who holds your keys. In the same way, the cloud host carries the certification, and you still own your settings, your access rules and your logs.

In one precise sentence, a regulated deployment picks a host whose authorization covers the customer's data class, then adds private connectivity, access control, logging policy and evaluation on top.

Step 1: Choose the hosting platform that carries the authorization

A model is software, so the hosting platform carries the compliance authorization. FedRAMP and the Defense Department's Impact Level 5 (IL5) apply to cloud services, and customers inherit compliance from the host. Anthropic's public sector FAQ (March 2026) puts it this way: "AI models are software components, not cloud services."

The same host-first pattern applies to any model family; Claude is the worked case here because its availability is publicly documented. For Claude the routes are the direct Anthropic application programming interface (API), Amazon Bedrock, Google Vertex AI, and the Enterprise or Claude for Government plans. Pick the route by the customer's strictest data class: first list the data the customer will send, then choose the host whose authorization covers it. The output is one named hosting route and a statement of what the customer inherits. You now know the first decision. The table in the trade-offs section compares the routes, and the next step covers how prompts reach the host.

Step 2: Keep prompts on a private path and decide what to log

A private endpoint keeps prompts off the public internet. The Amazon Web Services (AWS) security reference architecture says to use AWS PrivateLink for private connectivity from your virtual private cloud (VPC) to the Bedrock endpoint, so traffic is not exposed to the internet.

On Bedrock, AWS documents that the service does not store or log prompts and completions, does not use them to train AWS models, and that model providers have no access to the deployment accounts. The exception is abuse detection: for certain models Bedrock keeps flagged traffic, or all traffic, for up to 30 days (AWS re:Post), so check each model's retention mode.

Invocation logging is off by default; the AWS security blog warns the logs might contain sensitive information. Turn it on only when you need an audit trail, then lock the storage with identity and access management (IAM) policies and key management service (KMS) encryption. Input: a prompt. Output: a response and, only if you chose it, a protected copy.

Step 3: Authorize data before retrieval, per tenant

The access check runs before retrieval, so the model never sees data the user may not read. If a retrieval step fetches a document the user cannot access, the disclosure has already happened inside the model's context, and a filter on the answer cannot undo what the model has seen.

Sensitive information disclosure sits on the Open Worldwide Application Security Project (OWASP) Top 10 for LLM Applications, which has ten entries, as the second entry of the 2025 list. It was sixth in the 2023 list, so it moved up four places, as a secondary summary also notes. In a multi-tenant deployment the check uses the user's tenant and role to choose which index or filter the query may touch. Enforce that in the service, never in the prompt. The permission-aware RAG page shows how, and the regulated bank RAG design applies it end to end.

Step 4: Gate every release with per-tenant evaluation

An evaluation harness is a fixed set of test cases per tenant that every new model, prompt or retrieval change must pass before release. Run the same cases against the current system and the candidate, compare the pass rate per tenant, and add probes that try to pull one tenant's data out of another's session.

A release is blocked if any one tenant's results drop beyond the limit, and that is the rule that protects the customer who would otherwise be hurt silently. The U.S. National Institute of Standards and Technology (NIST) AI 600-1 Generative AI Profile (July 2024) names pre-deployment testing as one of its four main considerations, next to governance, content provenance and incident disclosure. The stage-by-stage RAG evaluation page covers what to measure at each stage. You now have the whole system: an authorized host, a private path, a pre-retrieval check and a per-tenant gate.

Four choices wrap the model host: platform, path, access and testsA customer app reaches a model host through a private endpoint; the host can write to a locked-down log store; an access check guards tenant documents, which feed the host as retrieved context; an evaluation gate blocks a bad release.Customer apptenant usersModel hostinherits theauthorizationPrivateendpointno publicinternetLog storelocked downAccess checkbefore retrievalTenantdocumentsseparate pertenantreleaseEvaluationgateper-tenant testsDiagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/design-deploy-llms-for-a-regulated-enterprise.html

Four choices wrap the model host: where it runs, how prompts reach it, who may retrieve what, and what must pass before release.CC BY 4.0 — source: How do you deploy LLMs for a regulated enterprise?
Step-by-step description
  1. The host carries the authorization.
  2. Prompts use a private endpoint; logs are locked down.
  3. Access is checked before retrieval.
  4. Each tenant's tests gate a release.
  • A model is software, so the hosting platform carries the compliance authorization.
  • A private endpoint keeps prompts off the public internet.
  • The access check runs before retrieval, so the model never sees data the user may not read.
  • A release is blocked if any one tenant's results drop beyond the limit.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.

Start Free Trial →

Worked example

Take an illustrative deployment with three tenants, each with 200 test cases, comparing the current model with a candidate. Tenant A passes 184 cases now and 188 with the candidate, tenant B passes 176 and then 169, and tenant C passes 190 and then 191.1 These counts are illustrative, not measurements of any real system.

With a release limit of a 2.0-point drop for any single tenant (illustrative), tenant A gains 2.0 points, tenant B loses 3.5 points and tenant C gains 0.5 points.1 The mean across tenants moves from 91.67 to 91.33, a fall of only 0.33 points, so an average-only check would pass this release while tenant B gets worse.1 In short, the mean across tenants falls 0.33 points while tenant B falls 3.5 points.1

Separately, 300 illustrative leakage probes, 50 aimed from each tenant at each of the other two, found 0 leaks.1 Zero leaks in 300 probes still allows a true leak rate of up to about 1.0% at 95% confidence (the rule of three, 3 divided by 300).1 If invocation logging is on at 50,000 requests a day with 6,000 bytes per record kept for 90 days (illustrative), the log store holds 27.0 GB that must be protected like the prompts it copies.1

Tenant B's 3.5-point drop blocks the release even though the mean across tenants moves by only 0.33 points.
QuantityValue (illustrative, computed)
Tenant A pass rate, current to candidate92.0% to 94.0% (+2.0 points)
Tenant B pass rate, current to candidate88.0% to 84.5% (-3.5 points)
Tenant C pass rate, current to candidate95.0% to 95.5% (+0.5 points)
Mean across the three tenants91.67% to 91.33% (-0.33 points)
Release decision with a 2.0-point drop limitBlock, because of tenant B
Cross-tenant leakage probes300 sent, 0 leaked
Log storage, 50,000 requests a day, 90 days27.0 GB
The mean across tenants hides one tenant's regressionGrouped bars of illustrative pass rates for the current and candidate model: tenant B falls from 88.0 to 84.5 percent while the mean across tenants moves only from 91.67 to 91.33 percent.Pass rate (%)050100Tenant ATenant BTenant CMeanTenantCurrent modelCandidate model92889591.679484.595.591.33Diagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/design-deploy-llms-for-a-regulated-enterprise.html
An average across tenants can hide one tenant's regression, so the release gate checks each tenant on its own.CC BY 4.0 — source: How do you deploy LLMs for a regulated enterprise?
Text description of the diagram
  • The mean across tenants falls 0.33 points while tenant B falls 3.5 points.
  • The mean across the three tenants moves from 91.67 to 91.33 percent.

Trade-offs and edge cases

The hosting route decides which authorizations you can claim. Per Anthropic's availability table, Claude in Amazon Bedrock is FedRAMP High and includes IL5, and Claude on Google Vertex AI is FedRAMP High with IL5 not included. Claude for Government is FedRAMP High with IL5 not included. The same table lists Claude Enterprise as not FedRAMP High with IL5 not included, and the FAQ says the direct Anthropic API is not FedRAMP authorized today. Customers requiring FedRAMP must use Claude for Government or a FedRAMP-authorized host.

Only Bedrock reaches IL5; Enterprise and the API lack FedRAMP HighTable of five Claude hosting routes against FedRAMP High and IL5: Bedrock in GovCloud holds both, Vertex and Claude for Government hold FedRAMP High only, Claude Enterprise and the direct API are not FedRAMP High.RouteFedRAMPHighIL5Bedrock in GovCloudYesYesVertex, AssuredWorkloadsYesNot includedClaude for GovernmentYesNot includedClaude EnterpriseNot includedNot includedDirect Anthropic APINot authorizedtodayNot statedDiagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/design-deploy-llms-for-a-regulated-enterprise.html
Compliance follows the hosting route, not the model, so the route is the first design decision.CC BY 4.0 — source: How do you deploy LLMs for a regulated enterprise?
Text description of the diagram
  • Claude in Amazon Bedrock is FedRAMP High and includes IL5.
  • Claude on Google Vertex AI is FedRAMP High with IL5 not included.
  • Claude for Government is FedRAMP High with IL5 not included.
  • Claude Enterprise is not FedRAMP High, and the direct Anthropic API is not FedRAMP authorized today.

Other edge cases:

  • Health data. The same FAQ says a business associate agreement (BAA) under the Health Insurance Portability and Accountability Act (HIPAA) is for Enterprise and direct API customers, requires a zero data retention (ZDR) agreement, and excludes some features such as web search.
  • Provider retention. Abuse detection can keep traffic for some models (see Step 2), so read each model's data-handling page.
  • Data residency. Pin the Region where the model runs, then check whether cross-Region inference can send prompts to a Region outside the chosen geography; see the AWS Bedrock cross-Region inference page (read 2026-10-26).
  • Agents. A model that can call tools adds the excessive-agency risk; see agentic RAG.
  • Governance. The NIST AI 600-1 profile groups actions under four functions: Govern, Map, Measure and Manage, and it names 12 generative-AI risks; a secondary summary points to the same document.
  • Model governance and audit. A regulated customer, such as a bank, expects a documented model inventory, human review of high-impact outputs and change records for every model, prompt or retrieval change; the NIST profile above is the primary reference for this kind of governance.

How to answer it in an interview

Interviewer asks

A regulated-industry customer wants to use an LLM API. How would you design the deployment?

A strong answer

I'd start from the customer's strictest data class and pick a host that already carries that authorization, because a model is software and compliance comes from the cloud platform. For government work that means Bedrock in GovCloud or Vertex with Assured Workloads. Prompts travel over a private endpoint, and I'd keep invocation logging off or locked down. Each tenant's data sits behind an access check that runs before retrieval, never after generation. Before any release, a per-tenant evaluation set plus leakage probes must pass, because an average can hide one tenant's regression.

The spoken structure: name the host and why it carries the compliance, then the private path and logging, then the pre-retrieval check, then the per-tenant release gate.

Common mistakes

  • "The model is FedRAMP authorized." (wrong) FedRAMP applies to cloud services, not model software. (right) "The hosting service carries the authorization and we inherit it." The FAQ says exactly this.
  • "Using Claude through Bedrock sends our data to Anthropic." (wrong) AWS says model providers have no access to the deployment accounts, to Bedrock logs or to prompts and completions. (right) "Prompts are processed inside AWS, in Bedrock deployment accounts the model provider cannot reach, and I confirm the model's retention mode on its data page."
  • "Bedrock keeps nothing, so there is no audit trail." (wrong) Logging is off by default, but turning it on creates a sensitive copy. (right) "I enable invocation logging when needed and protect the store with IAM and KMS."
  • "We filter the answers to stop leaks." (wrong) By then the model has already seen the data. (right) "Authorize before retrieval."
  • "The average score held, so the release is safe." (wrong) In the worked example the mean fell 0.33 points while tenant B fell 3.5.1 (right) "I gate each tenant separately."

Follow-up questions

What changes in a government contractor environment?

The host and its impact level decide everything. The FAQ says International Traffic in Arms Regulations (ITAR) data can only be processed in Claude via AWS Bedrock, which is IL5 accredited, and that Claude is available in the AWS Secret region, Impact Level 6 (IL6), via Bedrock. Check that each model is in the host's authorization scope.

A user invokes the right to be forgotten, but their data is in your model weights. How do you comply?

If the data is already in the weights, there are three options: retrain or re-tune from a cleaned dataset, which is exact but costly; machine unlearning, which is approximate and still research-stage; or output-suppression filters, which are a stopgap and not erasure. The erasure right is in Article 17 of the General Data Protection Regulation (GDPR), and the European Data Protection Board (EDPB) opinion says whether a model is anonymous under the GDPR is assessed case by case, so a model trained on personal data is neither automatically anonymous nor automatically unlawful. Going forward, keep personal data out of the weights and in stores you can delete from.

How do you prevent sensitive enterprise data from leaking into public models?

Close the three routes. Training: use a host whose terms say prompts are not used to train models. Logs: keep invocation logging off or locked down. Retrieval: authorize before the model sees a document.

Is Claude Enterprise or the direct API FedRAMP High?

No. The availability table lists Claude Enterprise as not including FedRAMP High, and the FAQ says the direct API is not FedRAMP authorized today. Use Claude for Government or a FedRAMP-authorized host.

What is different about a multi-tenant LLM application?

A single-tenant system needs one access policy. A multi-tenant system must also separate indexes, settings, logs, cost and evaluation per tenant, and it must prove the separation with probes. The permission-aware RAG page covers the retrieval side.

Where does an LLM gateway fit in this design?

The gateway is the one service every request passes through, so it is the natural place to resolve the tenant, apply the access policy and route to the approved host. The gateway interview questions go deeper.

Key takeaways
  • Compliance is inherited from the hosting platform, because a model is software, not a cloud service.
  • Pick the host by the customer's strictest data class, then add a private endpoint.
  • Invocation logging is off by default; turning it on creates a sensitive copy to protect.
  • Authorize before retrieval; a filter after generation comes too late.
  • Gate each release per tenant, because an average can hide one tenant's regression.

Sources

  1. Public Sector FAQs (Claude for Government) — Anthropic Claude Help Center, 25 Mar 2026
  2. Data protection - Amazon Bedrock User Guide — Amazon Web Services, 26 Oct 2026
  3. Does Amazon Bedrock share input and output data with model providers? — AWS re:Post Knowledge Center, 23 Sep 2026
  4. Generative AI model inference - Security Reference Architecture — AWS Prescriptive Guidance, 26 Oct 2026
  5. Implementing least privilege access for Amazon Bedrock — AWS Security Blog, 26 Oct 2026
  6. Claude for Government — Anthropic, 26 Oct 2026
  7. NIST AI 600-1, Generative AI Profile: a guide — Anaconda (summary of NIST AI 600-1, July 2024), 26 Oct 2026
  8. OWASP Top 10 for LLM Applications: a summary — Promptfoo (summary of the 2025 list), 26 Oct 2026
  9. OWASP Top 10 risks for LLM security 2025 — Invicti, 26 Oct 2026
  10. EDPB Opinion 28/2024: key takeaways — CMS (summary of the EDPB opinion of December 2024), 26 Oct 2026
  11. Opinion 28/2024 on AI models and personal data — European Data Protection Board, 17 Dec 2024
  12. Regulation (EU) 2016/679 (General Data Protection Regulation), Article 17 — EUR-Lex (European Union), 27 Apr 2016
  13. OWASP Top 10 for LLM Applications — OWASP GenAI Security Project, 18 Nov 2024
  14. NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative AI Profile — National Institute of Standards and Technology, 26 Jul 2024
  15. Increase throughput with cross-Region inference - Amazon Bedrock User Guide — Amazon Web Services, 26 Oct 2026