How do you design multi-tenant code sandboxes?

In this article

1. Before you start
2. Why interviewers ask this
3. Design multi-tenant code sandboxes from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the distributed systems interview guide, and its arithmetic uses the habits of back-of-the-envelope estimation.

Why interviewers ask this

Any product that runs code written by its customers needs this design: online code editors, serverless functions, notebook services, automated graders and plugin platforms. If one tenant escapes, it reads another tenant's data or takes down the shared host, so the question tests whether you treat isolation as the main requirement and not as an afterthought.

Candidates report related prompts, such as a multi-tenant database provisioning system with fast spin-up and per-project isolation, or a secure way for customers to reach rented graphics processing unit (GPU) machines without exposing them to the public internet. Nothing here says a particular company asks a particular question.

A weak answer says "run each job in a Docker container." A strong answer names the shared kernel as the risk, picks a stronger boundary, adds limits, restricts the network and tears everything down. A Staff-level answer also sizes the fleet and says which number is the bottleneck.

Design multi-tenant code sandboxes from first principles

Multi-tenant code sandbox

A multi-tenant code sandbox is an isolated execution environment that runs untrusted code from many customers on the same physical hosts, built so that no tenant can read, starve or escape into another tenant or into the host.

System call

A system call is a request from a running program to the operating system kernel to do something the program cannot do alone, such as open a file, allocate memory or send network data.

MicroVM

A microVM is a virtual machine stripped to the minimum devices and features needed to run one workload, so it starts quickly and uses little memory while still giving the workload its own guest kernel.

Control group

A control group (cgroup) is a Linux kernel feature that limits and accounts for the resources, such as processor time and memory, that a set of processes may use.

In plain words

Think of an apartment block. Each tenant gets a locked flat. Each flat has a fuse box that trips if the tenant draws too much power. A doorman decides who may visit. A container is a flat built from thin partitions inside one shared room. The walls are painted on. Everyone breathes the same air, which is the host kernel. The analogy breaks in one place. A real flat's wall is not a program. A sandbox boundary is software, so it must itself be free of bugs.

In one precise sentence, you design multi-tenant code sandboxes by layering a per-sandbox kernel boundary, resource limits, an egress policy and a lifecycle that destroys every sandbox after use.

Step 1: Choose the isolation boundary, because containers share the host kernel

The isolation boundary is the layer that separates untrusted code from the host, and it is the first decision. The gVisor documentation names two common families: virtual machines, and Linux kernel features such as seccomp-bpf, namespaces, AppArmor and Landlock. With kernel features alone, it says, the workload is only one system call away from host compromise, because the code still talks to the single host kernel.

Filters can shrink that surface, but they must be tailored to each workload. The same page notes that broad calls such as ioctl or io_uring can make a safe filter very hard to write, and arbitrary user code needs broad calls.

Two stronger options exist. gVisor is an application kernel that implements a Linux-like interface in a memory-safe language (Go) in user space, and it moves the system interfaces into a separate kernel for each sandbox. A microVM such as Firecracker gives each workload a guest kernel on the Kernel-based Virtual Machine (KVM). The virtual machine monitor that runs each microVM is itself attack surface, so run it with the fewest privileges it needs. You now know the first rule: untrusted code needs its own kernel layer.

Step 2: Cap resources with control groups

A control group limit stops one tenant from starving the rest. The Linux kernel's cgroup v2 documentation describes cpu.max, a bandwidth limit written as a maximum and a period; the default, "max 100000", means no limit. Writing "50000 100000" lets the group use 0.5 of a processor in each period.¹

For memory, memory.max is the hard limit: if usage reaches it and cannot be reduced, the out-of-memory (OOM) killer runs inside that group. memory.high is only a throttle. The same document says going over it never invokes the OOM killer, and under extreme conditions the limit may be breached. So use memory.max as the cap and, optionally, memory.high as an early warning.

Also limit wall-clock time and the number of processes, and cap output size. A cap on processor and memory does not stop a program that simply runs forever. Step 3 covers what the sandbox may reach.

Step 3: Control egress and tenant access

Egress is the network traffic that leaves a sandbox. Deny all egress by default and open only what the workload needs, through a proxy you control. Otherwise a sandbox can scan your internal network, call the cloud metadata service or attack other customers.

Egress is denied by default and the tenant quota is checked before a start. Identity and quota sit in front of the sandbox. Every request carries a tenant identity, and a control plane checks that tenant's quota, such as a cap on concurrent sandboxes, before anything starts. A distributed rate limiter at the gateway is the usual place to enforce request rates, and a concurrency counter held in a shared store, of the kind covered in the distributed cache design, enforces the sandbox cap.

On Kubernetes, namespaces, role-based access control (RBAC) and quotas are logical isolation. The Amazon Elastic Kubernetes Service (EKS) best practices guide says the cluster is the only construct that gives a strong security boundary, and its quotas and limit ranges control how many resources each tenant uses. Network rules shape what can be reached; they do not replace the kernel boundary of step 1.

Step 4: Place, start and tear down

A scheduler picks a host with spare capacity and asks it to start a sandbox from a prepared image. After the run, the sandbox is destroyed: processes killed, filesystem discarded, network rules removed. A sandbox that is reused between tenants carries leftover files and memory from the first tenant, so reuse only within one tenant and only if the risk is accepted.

Dedicated node pools reduce sharing only when tenants have no direct access to the Kubernetes control plane, because the platform, not the tenant, schedules the sandboxes. Google's documentation warns that taints and tolerations can be circumvented by malicious tenants, so taints alone do not stop one. For untrusted code the boundary is Google Kubernetes Engine (GKE) Sandbox (gVisor), a microVM, or a separate cluster, which the EKS guide calls the only construct that provides a strong security boundary. Give each sandbox a read-only root filesystem and its own scratch disk, wiped at teardown.

Startup time: measure it on your own images

Starting must be fast and must be measured. The Firecracker project page (read 2026-10-26) states a startup time under 125 milliseconds, up to 150 microVMs created per second per host, and memory overhead under 5 MiB per microVM. The Networked Systems Design and Implementation (NSDI) 2020 paper describes the design. Treat these as figures the project states for its minimal device model and streamlined kernel loading; measure them on your own hardware and images. The worked example next sizes the fleet.

1 · Boundary2 · Limits3 · Egress4 · LifecycleReplay

Step-by-step description

1. Give each run its own kernel layer.
2. Cap processor, memory and time.
3. Deny network traffic by default.
4. Place, run and destroy each sandbox.

• The isolation boundary is a microVM or gVisor, because containers share the host kernel.
• Control group limits cap processor and memory use.
• Egress is denied by default and the tenant quota is checked before a start.
• A scheduler places each sandbox and every sandbox is destroyed after the run.

Worked example

Suppose the platform must start 10,000 sandboxes per second at peak (an illustrative target). Using Firecracker's best-case 150 creations per second per host, the minimum is 67 hosts, because 10,000 divided by 150 is 66.7 and hosts come whole.¹ Halving the published figure to 75 per second doubles the need to 134 hosts.¹

Now memory and processor time. If each run lasts two seconds on average (illustrative), Little's law says the number running at once equals the arrival rate times the average time in the system, so 20,000.¹ With 128 MiB of guest memory (illustrative) plus the cited 5 MiB of overhead, each sandbox costs 133 MiB, so the fleet holds about 2,597.66 GiB. On hosts with 256 GiB usable (illustrative), that is 11 hosts.¹ For processor time, give each sandbox a cpu.max of 0.5 processor (illustrative): 20,000 sandboxes need 10,000 processors. At 64 usable cores per host (illustrative), that is 157 hosts.¹

At a cap of 50 concurrent sandboxes per tenant (illustrative), 400 tenants running at their cap would produce the 20,000 concurrent sandboxes. The row checks the quota against the load; it is not a capacity limit.

The lesson is the habit: compute each resource separately and size to the largest. Here processor time binds, so a lower per-sandbox processor limit or shorter runs help most. A snapshot or a pool of pre-started sandboxes attacks the start rate, which matters only when that is the largest number. The estimation guide covers the technique.

Text description of the diagram

• Ten thousand starts per second need 67 hosts at 150 creations per second per host.
• At 75 creations per second per host the need is 134 hosts.
• By memory the same load needs only 11 hosts.
• By processor time the same load needs 157 hosts.

Trade-offs and edge cases

Stronger isolation costs more per sandbox, and the table summarises the choice.

Text description of the diagram

• Containers share the host kernel.
• gVisor is an application kernel written in Go that runs in user space.
• Firecracker's project page states a startup time under 125 milliseconds.

Edge cases to name:

• Containers for trusted code. If every tenant's code is reviewed by you, containers may be enough. For arbitrary customer code they are not, per the one-system-call point in step 1.
• Compatibility. An application kernel such as gVisor implements a Linux-like interface, so rarely used system calls may behave differently. Test your languages and libraries.
• Pooled sandboxes. Pre-started sandboxes cut latency but hold idle memory and must never carry data between tenants.
• Noisy neighbours. Limits on processor and memory do not remove every shared resource, such as caches or disk bandwidth, so add disk and network limits and watch tail latency. Hardware side channels can also leak information between tenants that share a core. The Firecracker production host setup guide (read 2026-10-26) covers mitigations such as disabling simultaneous multithreading and applying the kernel's speculative-execution mitigations. Keep dedicated hosts for sensitive tenants.
• Kubernetes is not a boundary. Google's documentation says operators of multi-tenant clusters must isolate tenants, and it offers Google Kubernetes Engine (GKE) Sandbox, built on gVisor, for untrusted workloads. Related deployment strategies apply when you roll out new sandbox images.

How to answer it in an interview

The spoken structure: say the threat, choose the boundary, add limits, restrict the network, then manage the lifecycle and the numbers.

Common mistakes

• "Each job runs in a Docker container, so tenants are isolated." (wrong) Containers share the host kernel. (right) "Containers are my packaging; the security boundary is a microVM or a gVisor sandbox."
• "Kubernetes namespaces isolate my tenants." (wrong) They are logical isolation, and the cluster is the strong boundary. (right) "Namespaces organise tenants; for untrusted code I add a sandbox runtime such as gVisor, a microVM or a separate cluster, because taints alone do not stop a malicious tenant."
• "I set memory.high as the memory cap." (wrong) It throttles and can be breached. (right) "memory.max is the hard cap, and the OOM killer acts inside that group."
• "A microVM is too slow to start per request." (wrong) Firecracker's project page states a startup time under 125 milliseconds, for its minimal setup. (right) "Boot time is measured on my image, and a pool or snapshot hides the rest."

Follow-up questions

Are containers a security boundary for untrusted code?

Not on their own. A container shares the host kernel, so the workload is, in the gVisor documentation's words, only one system call away from host compromise. Add a microVM or an application kernel.

Is a microVM too slow to start for every request?

The Firecracker project page states a startup time under 125 milliseconds, with up to 150 microVMs created per second per host, figures the project states for its minimal device model and streamlined kernel loading. Measure it on your own hardware and images, and pool or pre-start sandboxes if the target is tighter.

What is the difference between memory.max and memory.high?

memory.max is a hard limit that invokes the OOM killer in the group when usage cannot be reduced. memory.high is a throttle that never invokes the OOM killer and can be breached under extreme conditions.

Do Kubernetes namespaces give multi-tenant isolation?

Kubernetes namespaces give logical separation, together with RBAC and quotas, but the cluster is the strong boundary. For untrusted code, run a sandbox runtime such as GKE Sandbox, a microVM, or a separate cluster. Dedicated node pools help only when tenants cannot reach the control plane, and Google says taints alone can be circumvented by malicious tenants.

How would customers SSH into their rented GPU instances without exposing them to the public internet?

A strong answer puts a Secure Shell (SSH) proxy in front as the only public entry point, which authenticates the customer, checks which graphics processing unit (GPU) instance they own and forwards the connection over a private network where the instances have no public address. Issue short-lived certificates instead of long-lived keys, and record each session at the proxy for audit. Reported by candidates as a question; this is design reasoning, not a sourced company practice, and the backend interview questions and concurrency questions pages cover neighbouring ground.

Sources

1. Firecracker: Lightweight Virtualization for Serverless Applications — USENIX NSDI '20 (Agache et al.), 25 Feb 2020
2. Firecracker: secure and fast microVMs for serverless computing (project page) — Firecracker project (Amazon Web Services, open source), 26 Oct 2026
3. Firecracker production host setup — Firecracker official repository, 26 Oct 2026
4. Introduction to gVisor security — gVisor (Google, open source), 26 Oct 2026
5. What is gVisor? — gVisor (Google, open source), 26 Oct 2026
6. Control Group v2 — The Linux Kernel documentation, 26 Oct 2026
7. Cluster multi-tenancy — Google Kubernetes Engine documentation, 26 Oct 2026
8. Tenant Isolation — Amazon EKS Best Practices Guide, 26 Oct 2026