What is LLM data residency compliance, and how do you design for it?

In this article

1. Before you start
2. Why interviewers ask this
3. LLM data residency compliance from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the generative AI system design interview guide.

Why interviewers ask this

A design that is fast and accurate can still be unlaunchable if a bank, hospital or public body cannot say where its prompts go. That is why residency questions come up when a candidate designs an assistant for a regulated customer, as in a retrieval-augmented generation (RAG) design for a regulated bank.

A weak answer says "we deploy in the EU region" and stops. A strong answer separates where data is stored from where the model runs, names the routing setting that controls the second, and says what evidence proves it. A Staff-level answer adds the failure modes: a new model with no compliant deployment option, a gateway that quietly reroutes traffic, and logs that live in a third place. These questions are reported by candidates in design rounds at companies that sell to regulated industries; no public source ties them to a named company.

LLM data residency compliance from first principles

Data residency

Data residency is a rule or design choice that fixes the geographic location where an organization's data is stored, and sometimes where it is processed, so the data stays under the legal regime of that place. For an LLM application it covers prompts, responses, documents, logs and any stored configuration.

Inference residency

Inference residency is a setting that keeps the model's computation on a request, the inference step run on graphics processing unit (GPU) hardware, inside a named region or zone. It is separate from where data is stored at rest, so a deployment can offer one without the other.

Cross-Region inference

Cross-Region inference is a cloud feature that lets the service process a request in any Region of an inference profile, not only the Region where it arrived. In Amazon Bedrock the Region is picked within a geography, or among all supported commercial Regions for a global profile.

In plain words

Think of a bank vault and a counting room. The vault holds your cash. The counting room is where staff handle it. A rule that says "the vault must be in Germany" says nothing about the counting room. You need a second rule for that room.

An LLM application has the same split. Storage is the vault. Processing is the counting room. The analogy breaks in one place: a cloud service can move the counting room between sites unless you pin it.

In one precise sentence, residency compliance means choosing a storage location and a processing scope that match your rules, and then showing evidence that both were applied.

Step 1: A data map lists every place prompt data is stored or processed

A data map lists every place prompt data is stored or processed. Start from the user's request and follow it. The application keeps session state. The gateway may keep request logs. A vector database holds document chunks that get pasted into prompts. The model endpoint receives the full prompt. A log store keeps traces.

Input: the architecture diagram. Output: a table with one row per hop, marking where data rests and where it is processed. A gateway for LLM traffic is the usual place where the scope and the logs are enforced. Teams often miss the logs and the retrieval store, because they are not "the model". A permission-aware retrieval design shows how document chunks reach prompts at all. You now have the list of hops that Step 2 treats one by one.

Step 2: Processing location is a separate setting from storage

Storage at rest and processing location are separate settings, and providers document them separately. Microsoft states that for all Foundry deployment types, data stored at rest remains in the designated Azure geography, while Global types may be processed in any Azure region. OpenAI treats the two as different features for ChatGPT: data residency stores content at rest in a region, and inference residency is a separate setting for eligible Enterprise and Edu customers.

Input: one hop from your data map. Output: two answers, "where does it rest?" and "where does it run?". Many designs answer only the first. A common error is to call the system compliant because the vault is in the right country, while the counting room is somewhere else.

Step 3: The routing scope should be the narrowest one the provider offers

The routing scope should be the narrowest one the provider offers. Azure offers three: Global, Data Zone and Regional. A Data Zone is a Microsoft-specified area, the US, the EU or Asia Pacific (APAC), inside which the service processes data. Microsoft says Data Zone types are processed anywhere within the Microsoft-specified data zone (US, EU or APAC).

Amazon Web Services (AWS) has the same idea. For Bedrock, choose geographic cross-Region inference when you have data residency requirements, and choose global inference only when you want cost savings without geographic limits. OpenAI lets you set a data residency region per project and pick regional processing per request through a regional domain prefix, if the endpoint and model support it. The routing scope should be the narrowest one the provider offers, and the fallback should fail closed rather than reroute.

Step 4: Evidence shows the settings were applied and still hold

Evidence shows the settings were applied and still hold. A setting you cannot show is a claim, not a control. Keep exported deployment configuration, the region fields in your request logs, and a record of which model and deployment type served each request.

Check the edges of each setting. OpenAI states that residency covers the endpoints and models that support it. For Amazon Bedrock, the user guide says that with a geographic inference profile the service selects a commercial Region within that geography to process your request, while a global profile may use any supported commercial Region. Processing can therefore happen in a Region other than the one you called, so confirm where logs and stored configuration live in the AWS documentation for your setup. OpenAI adds that data residency does not apply to system data, which may be processed and stored outside the selected region. You now have the full loop: map, separate, scope, prove.

1 · Map2 · Separate3 · Scope4 · ProveReplay

Step-by-step description

1. List every hop that touches prompts.
2. Answer storage and processing separately.
3. Pick the narrowest routing scope.
4. Keep evidence that settings hold.

• A data map lists every place prompt data is stored or processed.
• Storage at rest and processing location are separate settings.
• The routing scope should be the narrowest one the provider offers.
• Evidence shows the settings were applied and still hold.

Worked example

An illustrative design for a European bank assistant shows the hop-by-hop table. The system has five hops, and the design uses an EU Data Zone deployment on Azure.¹

Now the cost. Take 200 million tokens a day and a base rate of 1 cost unit per million tokens (both illustrative). A Global deployment then costs 200 units a day, or 6,000 over 30 days.¹ With illustrative uplifts of 10% for Data Zone and 25% for Regional, the 30-day cost is 6,600 and 7,500 units.¹ The uplifts are assumptions for the exercise, not published prices; check your provider's price list.

The Azure deployment-types page lists Global Batch with a 50% discount and a 24-hour turnaround. Applied to the illustrative 6,000 units, Global batch costs 3,000 units, which is 50% below Global.¹ Global Batch is a Global scope, so it processes anywhere: it shows what the cheapest option gives up, not a residency option. Check the provider's price list for any batch type inside a zone. Batch fits only work that can wait, such as nightly document processing; see making RAG fast and cheap for the interactive side.

Text description of the diagram

• Illustrative 30-day cost is 6,000 units for Global and 7,500 units for Regional.
• Global batch costs 3,000 units in the same illustrative model.

Trade-offs and edge cases

Narrower routing costs money and can limit model choice. Check region availability per model: the region-availability page shows models that are missing in some regions. A model may not be offered in the zone or region your rule needs, and the design must then either wait or use another model.

A regional endpoint is not one data center. A cloud region is a geographic area that contains several data centers. Microsoft says Standard and Regional types are processed in the region associated with your deployment. If your rule names a single country, check which country that region sits in. Data Zones cover only the US, the EU and APAC. An Azure geography can also be wider than the EU: Microsoft names the storage geographies as Americas, Europe, Asia Pacific, and Middle East and Africa, and Europe is wider than the EU.

Text description of the diagram

• Global types may be processed in any Azure region.
• Data Zone types process data only within the zone.
• Regional types process data in the region of the deployment.

Global types may be processed in any Azure region. Data Zone types process data only within the zone. Regional types process data in the region of the deployment. Storage at rest is the same for all three.

On the legal side, the General Data Protection Regulation (GDPR) does not say data must stay in the EU. Article 44 allows a transfer of personal data to a third country only if the conditions of its chapter are met. Those conditions include adequacy decisions, safeguards such as standard contractual clauses and binding corporate rules, and narrow derogations. Keeping processing in the EU is a design choice that can remove the transfer question for the model call, but it does not replace the data processing agreement, access control or logging.

GDPR's area is the European Economic Area (EEA), not only the EU, and remote access from a third country, for example by a provider's support staff, can itself be a transfer. Other regimes, such as national localization laws, have their own rules; ask counsel which apply. A chat product design shows how these constraints meet the rest of the system.

How to answer it in an interview

The spoken structure: say the two controls, walk the hops, name the routing scope and its trade-off, then close with evidence.

Common mistakes

• "We deploy in the EU region, so we are compliant." (wrong) That covers storage only. (right) "Storage is in the EU, and processing is pinned to an EU zone or region."
• "A regional endpoint means one data center." (wrong) A cloud region is a geographic area with several data centers. (right) "Regional means processed in the deployment's region; I check which country that region is in."
• "Cross-Region inference keeps everything in my own Region." (wrong) The service may pick another Region in the profile to process the request. (right) "I choose a geographic profile for residency and treat a global profile as unsuitable."
• "Switching residency on covers every endpoint." (wrong) Support depends on endpoint, model and account eligibility. (right) "I check the supported-endpoint list before relying on it."
• "Residency equals compliance." (wrong) It is one control among several. (right) "Residency sits next to the data processing agreement, access control and logging."

Follow-up questions

Does data residency mean the model runs in the same region?

No. Storage at rest and processing are separate settings, so data can rest in one region while a global deployment processes it elsewhere. Ask for the processing scope in writing.

Is a regional endpoint the same as one data center?

No. A cloud region is a geographic area with several data centers, and Microsoft says Regional types are processed in the region associated with the deployment. A single-country rule needs a check of which country that region sits in.

Does cross-Region inference keep processing in my Region?

Not necessarily. Bedrock selects a Region to process the request: inside the geography for a geographic profile, any supported commercial Region for a global profile. The guide recommends geographic profiles for data residency requirements, and you should confirm where logs and stored configuration live.

Does GDPR require data to stay in the EU?

Not literally. It restricts transfers to third countries unless a Chapter V mechanism applies. Residency is one way to avoid the question, and the data processing agreement and other controls still apply.

What if a new model has no residency-compliant deployment?

Wait, or use an older model with a compliant scope. Do not accept a global deployment for regulated data just to get the newer model. A stage-by-stage RAG evaluation can show whether the older model is good enough.

Sources

1. Deployment types for Microsoft Foundry Models — Microsoft Learn (undated page, read 2026-10-26), 26 Oct 2026
2. Region availability for Foundry Models sold by Azure — Microsoft Learn (undated page, read 2026-10-26), 26 Oct 2026
3. Route model inference requests across AWS Regions with cross-Region inference — Amazon Bedrock User Guide (undated page, read 2026-10-26), 26 Oct 2026
4. Data controls in the OpenAI platform — OpenAI (undated page, read 2026-10-26), 26 Oct 2026
5. Data residency and inference residency for ChatGPT — OpenAI Help Center (undated page, read 2026-10-26), 26 Oct 2026
6. Art. 44 GDPR: General principle for transfers — gdpr-info.eu (regulation text, undated page, read 2026-10-26), 26 Oct 2026