Healthcare LLM safety architecture: how to design one

In this article

1. Before you start
2. Why interviewers ask this
3. Healthcare LLM safety architecture from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the generative AI system design learning path. It assumes you know what a model prompt is and have met retrieval.

Why interviewers ask this

In healthcare a wrong answer can change a treatment, so the architecture matters more than the model choice. Kim et al. surveyed 70 clinicians: 91.8% had met medical hallucinations and 84.7% thought they could affect patient health (arXiv 2503.05777v2, November 2025). Design rounds for regulated domains therefore test whether you reach for layers rather than a single clever prompt.

A weak answer says "use a medical model with retrieval and a disclaimer." A strong answer starts with who the user is and what harm a wrong output can do, then places a control for each failure: bad input, leaked data, unsupported claims, missed reasoning errors and over-trusting reviewers. A Staff-level answer also says which controls are required by law, which are design choices, and how each is tested after launch. Questions like this are reported by candidates in GenAI design rounds; no company is named here.

Healthcare LLM safety architecture from first principles

Healthcare LLM safety architecture

A healthcare LLM safety architecture is a set of independent controls placed before, around and after a language model that limits harm from wrong output and data exposure in a clinical setting, by routing each request through checks matched to its risk and ending in human review for the dangerous cases.

Medical hallucination

A medical hallucination is a model output that is factually incorrect, logically inconsistent, or unsupported by authoritative clinical evidence in ways that could alter clinical decisions, a definition taken from Kim et al. (2025).

In plain words

Think of a hospital front desk. A triage nurse sorts patients by urgency. A records clerk checks who may see which file. A doctor works from the chart. A pharmacist double-checks the prescription. A senior clinician signs off on the risky cases.

The analogy breaks in one place. A model's mistakes are fluent and can be shared by every checker built on similar models, so layers are less independent than people are.

In one precise sentence, the model is one component, and safety comes from controls that do not depend on the model being right. The layers are still not fully independent: a verifier built on a similar model can repeat the generator's error, so the combined miss rate can be higher than multiplying each layer's separate miss rate suggests, and the worked example below puts numbers on that gap.

Step 1: Classify the request by risk before any model call

Triage sorts each request by risk tier before any model call, because the tier decides which later layers run. Tiers are a design choice; a common split is administrative help, general information, clinician-facing decision support and urgent or patient-facing advice. Kim et al. use a physician risk scale from no risk through minor, significant, considerable and major to catastrophic.

The intended use also shapes the regulatory picture. Under 21 U.S.C. §360j(o)(1)(E), clinical decision support software falls outside the medical-device definition only if it meets four criteria. It is not intended to acquire, process or analyse a medical image or signal. It is intended to display, analyse or print medical information. It is intended to support or provide recommendations to a health care professional. And it is intended to enable that professional to independently review the basis for the recommendations. Software that misses any one criterion cannot rely on that exclusion, so the Food and Drug Administration (FDA) may regulate it as a device.

Hard-coded red-flag rules that send urgent symptoms to a person are a design pattern, not a sourced standard. The next four steps hang from this router.

Step 2: Protect patient data and authority on the way in

The data guard strips or limits patient information before the prompt is built and checks what the caller is allowed to see. De-identification is the process of removing or transforming identifiers so that data no longer points to a person. The Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule names two methods, Expert Determination and Safe Harbor (guidance of the U.S. Department of Health and Human Services (HHS), dated November 2012). Free-text notes are the hard case; names hide in sentences.

Retrieved text is also untrusted input. The OWASP Top 10 for LLM Applications (2025 list) puts prompt injection first, under the item name Prompt Injection (LLM01), and lists excessive agency as Excessive Agency (LLM06), whose mitigations include human approval for high-impact actions. A summary of that list separates direct prompt injection from indirect injection, where the model accepts input from an external source such as a website or file. Aembit's reading of the 2026 edition (September 2026) stresses identity-based, least-privilege access enforced outside the model. The design for who may see which record is covered in permission-aware RAG.

Step 3: Ground the draft in evidence and make it reason

The grounded draft layer makes the model answer from retrieved clinical evidence and reason before it concludes. Retrieval is not a cure by itself. Kim et al. name a primary failure mode as retrieval of irrelevant or low-quality information. The paper suggests adaptive retrieval, which means searching only when it helps.

Explicit reasoning helped most. In the results of Kim et al. (v2), Gemini 2.5 Pro rose from 87.6% to above 97% with chain of thought. Chain of thought significantly reduced hallucinations in 86.4% of tested comparisons after false-discovery-rate correction (Kim et al., arXiv 2503.05777v2). The paper's explanation is that explicit reasoning enables self-verification.

Domain tuning is no shortcut either. In Kim et al. (v2), medical-specialised models had a median 51.3% hallucination-free rate against 76.6% for general-purpose models, although the paper notes the specialised models are smaller open models.

Text description of the diagram

• Gemini 2.5 Pro rose from 87.6% to above 97% with chain of thought.
• Chain of thought significantly reduced hallucinations in 86.4% of tested comparisons after false-discovery-rate correction.

Step 4: Verify the draft and abstain when evidence is thin

A verifier checks the draft against the evidence or abstains. It must catch errors such as wrong facts, outdated references and fabricated sources or guidelines. Kim et al. list three detection families: factual verification, summary-consistency checks and uncertainty-based methods. Uncertainty options include sequence log-probability, semantic entropy, which samples several answers and clusters them by meaning, and abstention thresholding.

Design the verifier for the failure that remains. Physician audits found that 64 to 72% of residual hallucinations came from causal or temporal reasoning failures rather than knowledge gaps. A verifier that only matches names and doses against a source misses a wrong order of events or a wrong cause.

Scoring rules shape behaviour too. The Med-HALT (Medical Domain Hallucination Test) setup of Pal et al. (2023) gives +1 for a correct answer and −0.25 for a wrong one, so confident guessing has a cost. A system that abstains sends the case to Step 5 instead.

Step 5: Put a clinician in the loop who can really review

Clinician review shows the draft with its evidence, and every decision is logged. Automation bias is the tendency to over-rely on a suggestion from an automated system. The fourth statutory criterion asks that the software enable the health care professional to independently review the basis for its recommendations, so a review screen that hides the basis works against it.

So the review screen should show the retrieved passages, the reasoning and the uncertainty. A log keeps inputs, evidence, outputs and the reviewer's decision, with patient data protected in it.

After launch, Kim et al. call for task-specific validation, continuous monitoring and human-in-the-loop workflows. The National Institute of Standards and Technology (NIST) released its Generative AI Profile, NIST AI 600-1, on July 26, 2024 as a companion to its AI Risk Management Framework.

1 · Triage2 · Data guard3 · Grounded draft4 · Verifier5 · Clinician reviewReplay

Step-by-step description

1. Sort the request by risk tier.
2. De-identify and check permissions.
3. Answer from evidence with reasoning.
4. Check the draft or abstain.
5. Show the basis and log the decision.

• Triage sorts each request by risk tier before any model call.
• A verifier checks the draft against the evidence or abstains.
• Clinician review shows the draft with its evidence, and every decision is logged.

Worked example

Two small calculations show why measurement and layering need care; every input is illustrative except the scoring rule. First the point score. The Med-HALT scoring, which Kim et al. (v2) apply, gives +1 for a correct answer and −0.25 for a wrong one (Pal et al., 2023). Take a test of 350 cases (an illustrative size, such as 50 from each of seven tasks). A model with 280 right and 70 wrong answers (illustrative) scores (280 − 17.5) / 350 = 0.75.¹

Second, layers. Take 10,000 drafts of which 5% are unsafe, which is 500 (illustrative). Three layers catch 60%, 70% and 80% of what reaches them (illustrative): a grounding check, a verifier and clinician review. If their misses are independent, 500 × 0.4 × 0.3 × 0.2 = 12 unsafe drafts get through.¹

Now suppose 30% of unsafe drafts (150) have a shared blind spot that both automated layers miss, such as a wrong sequence of events. Those 150 meet only the clinician and 30 pass; the other 350 contribute 8.4, so the expected number of unsafe drafts that get through is 38.4, 3.2 times as many as in the independent case.¹

In an illustrative run, 500 unsafe drafts out of 10,000 drop to 12 with three independent layers. With a shared blind spot the same layers let 38.4 through. The catch rates are design assumptions, because the sources give no real layer catch rates.

Text description of the diagram

• In an illustrative run, 500 unsafe drafts out of 10,000 drop to 12 with three independent layers.
• With a shared blind spot the same layers let 38.4 through.

Trade-offs and edge cases

Every control buys safety with cost, delay or a new failure mode, so match the control to the tier. Reading the sourced facts above together gives this comparison.

The regulatory edge matters. Under the statute all four criteria must hold for the exclusion, so a review step does not exempt a product by itself. Guidance and enforcement practice change, so confirm the current position with regulatory counsel. This page is not legal advice and does not cover rules outside the United States.

Benchmark scores are not safety. Kim et al. (v2) say their scores do not directly capture patient harm, add physician risk grading, and report that even low-hallucination models made "Significant" or "Considerable" risk errors. Because reviewers can disagree, use several. For stage-level measurement see evaluating a RAG system stage by stage, and for a comparable regulated design see RAG in a regulated bank.

How to answer it in an interview

The spoken structure: name the risk tier first, walk the layers in order, say where layers can fail together, then close on monitoring.

Common mistakes

• "We tuned the model on medical data, so it is safe." (wrong) In one benchmark the specialised models scored lower than general ones. (right) "Domain tuning helps vocabulary; safety comes from layers and testing."
• "Retrieval fixes hallucination." (wrong) Retrieval can bring bad or conflicting text. (right) "I ground the draft, then verify it against the evidence."
• "A human reviews everything, so no further controls." (wrong) Review fails under automation bias and time pressure. (right) "The reviewer sees the evidence and the system abstains on weak cases."
• "A prompt guardrail stops prompt injection." (wrong) A prompt is advice to the model, and an injected document can contradict it. (right) "Authorization is enforced outside the model."

Follow-up questions

How do you reduce hallucination in a medical chatbot?

Combine grounded retrieval, chain-of-thought reasoning, a verifier and abstention, then test each against physician-graded cases. Kim et al. (v2) report that structured prompting and retrieval reduced hallucinations while high-risk reasoning errors remained.

Does retrieval-augmented generation fix medical hallucination?

No. Retrieval can bring irrelevant or low-quality text, which Kim et al. name a primary failure mode. Decide when to retrieve, as covered in agentic RAG.

When does a clinical LLM count as a medical device?

When it fails any of the four statutory criteria for non-device clinical decision support: for example it analyses images or signals, is not meant to support a health care professional, or does not let that professional independently review its basis. Failing a criterion only removes the non-device exclusion; the function must still meet the statutory definition of a device to be regulated as one.

What guardrails make an LLM setup HIPAA-compliant?

No model is HIPAA-compliant on its own; compliance depends on how protected health information is handled around it. The guardrails are controls outside the model: minimise the data, de-identify where possible using Expert Determination or Safe Harbor, restrict access by permission, and treat logs as regulated data too. Confirm contract terms such as a business associate agreement with your compliance team.

How do you stop prompt injection in a clinical assistant?

Treat retrieved documents as untrusted, enforce authorization outside the model, and require human approval for high-impact or hard-to-reverse actions. A gateway layer helps, as in the LLM gateway interview questions.

How do you measure whether a clinical LLM is safe?

Use task-specific tests with physician risk grading, several reviewers and continuous monitoring after launch. Hallucination-rate benchmarks alone do not show patient harm.

Sources

1. 21 U.S.C. §360j: General provisions respecting control of devices intended for human use (subsection (o)) — Cornell Legal Information Institute (United States Code), 13 Dec 2016
2. Medical Hallucination in Foundation Models and Their Impact on Healthcare (arXiv 2503.05777v2) — arXiv (Kim et al.), 2 Nov 2025
3. Med-HALT: Medical Domain Hallucination Test for Large Language Models — ACL Anthology (Pal, Umapathi, Sankarasubbu), 1 Dec 2023
4. Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the HIPAA Privacy Rule — HHS Office for Civil Rights, 26 Nov 2012
5. AI Risk Management Framework and the Generative AI Profile (NIST AI 600-1) — NIST, 26 Jul 2024
6. OWASP Top 10 for LLM Applications (2025 list) — OWASP GenAI Security Project, 1 Jan 2025
7. LLM06:2025 Excessive Agency — OWASP GenAI Security Project, 1 Jan 2025
8. OWASP Top 10 for LLM Applications (2025 list), summary — Promptfoo (summary of the OWASP list), 25 Oct 2026
9. The OWASP Top 10 for LLM Applications 2026: what changed — Aembit, 1 Sep 2026